Package io.github.kirstenali.deepj.layers.transformer.blocks

Class GPTTransformerBlock

java.lang.Object

io.github.kirstenali.deepj.layers.transformer.blocks.GPTTransformerBlock

All Implemented Interfaces:

Layer, Trainable

public class GPTTransformerBlock
extends Object

Pre-LN GPT-style Transformer block:

   x = x + Attn( LayerNorm(x) )
   x = x + MLP(  LayerNorm(x) )
 

Composes LayerNorm1D, MultiHeadSelfAttention, and an FNN (feed-forward network with a configurable activation — default: GELU).

Constructor Summary

Constructors

Constructor	Description
GPTTransformerBlock(int dModel, int nHeads, int dFF, Supplier<ActivationFunction> ffnActivationFactory, Random rnd)	Full constructor.
GPTTransformerBlock(int dModel, int nHeads, int dFF, Random rnd)	Convenience constructor using GELU activation.

Method Summary

Modifier and Type	Method	Description
Tensor	backward(Tensor gradOut)
Tensor	forward(Tensor x)
List<Parameter>	parameters()
protected Layer[]	subLayers()	Return the ordered sub-layers that own trainable parameters.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

GPTTransformerBlock

public GPTTransformerBlock(int dModel,
 int nHeads,
 int dFF,
 Random rnd)

Convenience constructor using GELU activation.

Parameters:

dModel - model dimension

nHeads - attention heads (must divide dModel)

dFF - feed-forward inner dimension

rnd - random source for weight initialisation

GPTTransformerBlock

public GPTTransformerBlock(int dModel,
 int nHeads,
 int dFF,
 Supplier<ActivationFunction> ffnActivationFactory,
 Random rnd)

Full constructor.

Parameters:

dModel - model dimension

nHeads - attention heads (must divide dModel)

dFF - feed-forward inner dimension

ffnActivationFactory - factory for the FFN hidden activation (e.g. GELU::new)

rnd - random source for weight initialisation

Method Details

subLayers

protected Layer[] subLayers()

Return the ordered sub-layers that own trainable parameters.

forward

public Tensor forward(Tensor x)

backward

public Tensor backward(Tensor gradOut)

parameters

public List<Parameter> parameters()

Specified by:

parameters in interface Trainable