Package io.github.kirstenali.deepj.models.llama

Class LlamaModel

java.lang.Object

io.github.kirstenali.deepj.models.DecoderOnlyModel

io.github.kirstenali.deepj.models.llama.LlamaModel

All Implemented Interfaces:

CausalLM, Persistable, Trainable

public final class LlamaModel
extends DecoderOnlyModel

Llama-style decoder-only transformer.

Architecture differences from GPTModel:

• No learned positional embedding — RoPE is applied inside each attention block.
• RMSNorm instead of LayerNorm for the final pre-head normalisation.
• SwiGLU feed-forward instead of GELU-FFN.

This compact implementation uses full multi-head attention and does not maintain an incremental KV cache; it is not an exact release of Meta's Llama models.

Forward/backward/parameters are provided by DecoderOnlyModel.

Field Summary

Fields inherited from class io.github.kirstenali.deepj.models.DecoderOnlyModel

lmHead, normF, stack, tokEmb

Constructor Summary

Constructors

Constructor	Description
LlamaModel(LlamaConfig cfg, long seed)

Method Summary

Modifier and Type	Method	Description
LlamaConfig	config()
float	gradClipNorm()	Global gradient-clipping threshold (e.g.

Methods inherited from class io.github.kirstenali.deepj.models.DecoderOnlyModel

applyInitScale, backward, backwardEmbeddings, embed, embeddingParameters, forward, parameters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.persistence.Persistable

load, save

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

LlamaModel

public LlamaModel(LlamaConfig cfg,
 long seed)

Method Details

gradClipNorm

public float gradClipNorm()

Description copied from interface: CausalLM

Global gradient-clipping threshold (e.g. 1.0).

config

public LlamaConfig config()