Package io.github.kirstenali.deepj.layers.transformer.attention

Class RoPEMultiHeadSelfAttention

java.lang.Object

io.github.kirstenali.deepj.layers.transformer.attention.MultiHeadSelfAttention

io.github.kirstenali.deepj.layers.transformer.attention.RoPEMultiHeadSelfAttention

All Implemented Interfaces:

Layer, Trainable

public final class RoPEMultiHeadSelfAttention
extends MultiHeadSelfAttention

Field Summary

Fields inherited from class io.github.kirstenali.deepj.layers.transformer.attention.MultiHeadSelfAttention

headDim, nHeads

Constructor Summary

Constructors

Constructor	Description
RoPEMultiHeadSelfAttention(int dModel, int nHeads, boolean causalMask, RotaryEmbedding rope, Random rnd)
RoPEMultiHeadSelfAttention(int dModel, int nHeads, boolean causalMask, RotaryEmbedding rope, Random rnd, ParameterFactory parameters)

Method Summary

Modifier and Type	Method	Description
protected Tensor	transformQueryKey(Tensor heads, int seqLen)
protected Tensor	transformQueryKey(Tensor heads, int length, int offset)
protected Tensor	transformQueryKey(Tensor heads, int length, InferenceContext context)

Methods inherited from class io.github.kirstenali.deepj.layers.transformer.attention.MultiHeadSelfAttention

forward, forward, parameters, supportsIncrementalInference

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

RoPEMultiHeadSelfAttention

public RoPEMultiHeadSelfAttention(int dModel,
 int nHeads,
 boolean causalMask,
 RotaryEmbedding rope,
 Random rnd)

RoPEMultiHeadSelfAttention

public RoPEMultiHeadSelfAttention(int dModel,
 int nHeads,
 boolean causalMask,
 RotaryEmbedding rope,
 Random rnd,
 ParameterFactory parameters)

Method Details

transformQueryKey

protected Tensor transformQueryKey(Tensor heads,
 int seqLen)

Overrides:

transformQueryKey in class MultiHeadSelfAttention

transformQueryKey

protected Tensor transformQueryKey(Tensor heads,
 int length,
 int offset)

Overrides:

transformQueryKey in class MultiHeadSelfAttention

transformQueryKey

protected Tensor transformQueryKey(Tensor heads,
 int length,
 InferenceContext context)

Overrides:

transformQueryKey in class MultiHeadSelfAttention