Package io.github.kirstenali.deepj.layers.transformer.blocks

Class DeepSeekTransformerBlock

java.lang.Object

io.github.kirstenali.deepj.layers.transformer.blocks.DeepSeekTransformerBlock

All Implemented Interfaces:

Layer, Trainable

public final class DeepSeekTransformerBlock
extends Object

Pre-LN transformer block with DeepSeek-style latent attention:

   x = x + MLA( RMSNorm(x) )
   x = x + SwiGLU( RMSNorm(x) )
 

Identical to LlamaTransformerBlock except attention uses MultiHeadLatentAttention instead of RoPE-MHA. The latent representation can support a compressed cache in a future incremental decoder.

Constructor Summary

Constructors

Constructor	Description
DeepSeekTransformerBlock(int dModel, int nHeads, int qRank, int kvRank, int dFF, int maxSeqLen, Random rnd)

Method Summary

Modifier and Type	Method	Description
Tensor	backward(Tensor gradOut)
Tensor	forward(Tensor x)
List<Parameter>	parameters()
protected Layer[]	subLayers()	Return the ordered sub-layers that own trainable parameters.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

DeepSeekTransformerBlock

public DeepSeekTransformerBlock(int dModel,
 int nHeads,
 int qRank,
 int kvRank,
 int dFF,
 int maxSeqLen,
 Random rnd)

Parameters:

dModel - model dimension

nHeads - attention heads (must divide dModel)

qRank - Q latent dimension

kvRank - KV latent dimension

dFF - SwiGLU intermediate dimension

maxSeqLen - maximum sequence length for the RoPE table

rnd - random source for weight initialisation

Method Details

subLayers

protected Layer[] subLayers()

Return the ordered sub-layers that own trainable parameters.

forward

public Tensor forward(Tensor x)

backward

public Tensor backward(Tensor gradOut)

parameters

public List<Parameter> parameters()

Specified by:

parameters in interface Trainable