Package io.github.kirstenali.deepj.layers.transformer

Class TransformerBlock

java.lang.Object

io.github.kirstenali.deepj.layers.transformer.TransformerBlock

All Implemented Interfaces:

Layer, Trainable

public final class TransformerBlock
extends Object
implements Layer

Pre-LN Transformer block: x = x + Attn(LN(x)) x = x + MLP(LN(x))

Constructor Summary

Constructors

Constructor	Description
TransformerBlock(int dModel, int nHeads, int dFF, Supplier<ActivationFunction> ffnActivationFactory, Random rnd)
TransformerBlock(int dModel, int nHeads, int dFF, Random rnd)

Method Summary

Modifier and Type	Method	Description
Tensor	backward(Tensor gradOut)
Tensor	forward(Tensor x)
List<Parameter>	parameters()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

TransformerBlock

public TransformerBlock(int dModel,
 int nHeads,
 int dFF,
 Random rnd)

TransformerBlock

public TransformerBlock(int dModel,
 int nHeads,
 int dFF,
 Supplier<ActivationFunction> ffnActivationFactory,
 Random rnd)

Method Details

forward

public Tensor forward(Tensor x)

Specified by:

forward in interface Layer

backward

public Tensor backward(Tensor gradOut)

Specified by:

backward in interface Layer

parameters

public List<Parameter> parameters()

Specified by:

parameters in interface Trainable