Package io.github.kirstenali.deepj.models.gpt

Class GPTModel

java.lang.Object

io.github.kirstenali.deepj.models.DecoderOnlyModel

io.github.kirstenali.deepj.models.gpt.GPTModel

All Implemented Interfaces:

CausalLM, Persistable, Trainable

public final class GPTModel
extends DecoderOnlyModel

Minimal GPT-style decoder-only transformer for educational/training use.

Extends DecoderOnlyModel; the only GPT-specific additions are:

• A learned PositionalEmbedding added to the token embedding.
• An optional init-scale applied to random weights.
• LayerNorm1D (instead of RMSNorm) as the final normalisation.

It intentionally omits features such as dropout, tied embeddings, and an incremental KV cache, so it should be described as GPT-style rather than GPT-2/3.

Field Summary

Fields inherited from class io.github.kirstenali.deepj.models.DecoderOnlyModel

lmHead, normF, stack, tokEmb

Constructor Summary

Constructors

Constructor	Description
GPTModel(GPTConfig cfg, long seed)

Method Summary

Modifier and Type	Method	Description
protected void	backwardEmbeddings(Tensor g)	Back-propagates gradient into embedding layer(s).
GPTConfig	config()
protected Tensor	embed(int[] inputIds)	Maps input ids to the initial hidden state.
protected List<Parameter>	embeddingParameters()	Returns all embedding parameters.
float	gradClipNorm()	Global gradient-clipping threshold (e.g.

Methods inherited from class io.github.kirstenali.deepj.models.DecoderOnlyModel

applyInitScale, backward, forward, parameters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.persistence.Persistable

load, save

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

GPTModel

public GPTModel(GPTConfig cfg,
 long seed)

Method Details

embed

protected Tensor embed(int[] inputIds)

Description copied from class: DecoderOnlyModel

Maps input ids to the initial hidden state. Override to add positional embeddings.

Overrides:

embed in class DecoderOnlyModel

backwardEmbeddings

protected void backwardEmbeddings(Tensor g)

Description copied from class: DecoderOnlyModel

Back-propagates gradient into embedding layer(s). Override to include positional.

Overrides:

backwardEmbeddings in class DecoderOnlyModel

embeddingParameters

protected List<Parameter> embeddingParameters()

Description copied from class: DecoderOnlyModel

Returns all embedding parameters. Override to include positional embedding.

Overrides:

embeddingParameters in class DecoderOnlyModel

gradClipNorm

public float gradClipNorm()

Description copied from interface: CausalLM

Global gradient-clipping threshold (e.g. 1.0).

config

public GPTConfig config()