Package io.github.kirstenali.deepj.layers.transformer

Class MultiHeadSelfAttention

java.lang.Object

io.github.kirstenali.deepj.layers.transformer.MultiHeadSelfAttention

All Implemented Interfaces:

Layer, Trainable

public final class MultiHeadSelfAttention
extends Object
implements Layer

Multi-head causal self-attention for a single sequence (no batch dimension). Input/Output shape: [seqLen x dModel]

Constructor Summary

Constructors

Constructor	Description
MultiHeadSelfAttention(int dModel, int nHeads, boolean causalMask, Random rnd)

Method Summary

Modifier and Type	Method	Description
Tensor	backward(Tensor dOut)
Tensor	forward(Tensor x)
List<Parameter>	parameters()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

MultiHeadSelfAttention

public MultiHeadSelfAttention(int dModel,
 int nHeads,
 boolean causalMask,
 Random rnd)

Method Details

forward

public Tensor forward(Tensor x)

Specified by:

forward in interface Layer

backward

public Tensor backward(Tensor dOut)

Specified by:

backward in interface Layer

parameters

public List<Parameter> parameters()

Specified by:

parameters in interface Trainable