Package io.github.kirstenali.deepj.models

Class DecoderOnlyModel

java.lang.Object

io.github.kirstenali.deepj.models.DecoderOnlyModel

All Implemented Interfaces:

CausalLM, Persistable, Trainable

Direct Known Subclasses:

DeepSeekModel, GPTModel, LlamaModel

public abstract class DecoderOnlyModel
extends Object
implements CausalLM, Persistable

Shared skeleton for decoder-only transformer models: token embedding → block stack → final norm → LM-head.

Subclasses pass their concrete TransformerStack to the protected constructor and provide CausalLM.gradClipNorm().

Models that add positional embeddings (e.g. GPT) override embed(int[]), backwardEmbeddings(io.github.kirstenali.deepj.tensor.Tensor), and embeddingParameters().

Field Summary

Fields

Modifier and Type	Field	Description
protected final Projection	lmHead
protected final NormLayer	normF
protected final TransformerStack	stack
protected final Embedding	tokEmb

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	DecoderOnlyModel(Embedding tokEmb, TransformerStack stack, NormLayer normF, Projection lmHead)

Method Summary

Modifier and Type	Method	Description
protected final void	applyInitScale(float factor)
void	backward(Tensor dLogits)	Back-propagates gradient of the logits through the model.
protected void	backwardEmbeddings(Tensor g)	Back-propagates gradient into embedding layer(s).
protected Tensor	embed(int[] inputIds)	Maps input ids to the initial hidden state.
protected List<Parameter>	embeddingParameters()	Returns all embedding parameters.
Tensor	forward(int[] inputIds)	Maps token ids to logits [seqLen × vocabSize].
List<Parameter>	parameters()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.models.CausalLM

gradClipNorm

Methods inherited from interface io.github.kirstenali.deepj.persistence.Persistable

load, save

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Field Details

tokEmb

protected final Embedding tokEmb

stack

protected final TransformerStack stack

normF

protected final NormLayer normF

lmHead

protected final Projection lmHead

Constructor Details

DecoderOnlyModel

protected DecoderOnlyModel(Embedding tokEmb,
 TransformerStack stack,
 NormLayer normF,
 Projection lmHead)

Method Details

embed

protected Tensor embed(int[] inputIds)

Maps input ids to the initial hidden state. Override to add positional embeddings.

backwardEmbeddings

protected void backwardEmbeddings(Tensor g)

Back-propagates gradient into embedding layer(s). Override to include positional.

embeddingParameters

protected List<Parameter> embeddingParameters()

Returns all embedding parameters. Override to include positional embedding.

forward

public Tensor forward(int[] inputIds)

Description copied from interface: CausalLM

Maps token ids to logits [seqLen × vocabSize].

Specified by:

forward in interface CausalLM

backward

public void backward(Tensor dLogits)

Description copied from interface: CausalLM

Back-propagates gradient of the logits through the model.

Specified by:

backward in interface CausalLM

parameters

public List<Parameter> parameters()

Specified by:

parameters in interface Persistable

Specified by:

parameters in interface Trainable

applyInitScale

protected final void applyInitScale(float factor)