All Classes and Interfaces
Class
Description
AdamW optimizer with per-parameter state.
One language-modeling batch: x are input token ids, y are targets (next-token ids).
Supplies token batches to a causal language-model trainer.
Minimal byte-level tokenizers (0-255).
Common contract for decoder-only causal language models (GPT, Llama, DeepSeek, …).
Evaluation helpers for causal language models.
Wiring helpers for causal language model training.
Collects GPU operations lazily and flushes them as a single command buffer.
Linear warmup followed by cosine decay.
Cross-entropy loss with integer class targets.
Shared skeleton for decoder-only transformer models:
token embedding → block stack → final norm → LM-head.
Exports a DeepJ checkpoint, tokenizer, config, and Hugging Face model card.
Configuration for a DeepSeek-style decoder-only transformer.
DeepSeek-style decoder-only transformer.
Configuration for the bounded-memory TinyStories DeepSeek training example.
Pre-LN transformer block with DeepSeek-style latent attention:
Builder for DeepSeek-style transformer stacks.
Token embedding: ids -> vectors.
Loads and evaluates a trained TinyStories DeepSeek-style bundle.
Aggregate causal language-model evaluation metrics.
Evaluates and exports the trained TinyStories checkpoint for Hugging Face.
Flexible fully-connected neural network (MLP) built from
Linear projections.Gaussian Error Linear Unit (GELU), using the tanh approximation popularized by GPT-2.
Minimal GPT-style decoder-only transformer for educational/training use.
Pre-LN GPT-style Transformer block:
Builder for GPT-style transformer stacks.
Handle to a GPU-resident float buffer managed by a
ComputeGraph.Abstraction over a GPU compute runtime (Metal, CUDA, Vulkan, etc.).
Differentiable module mapping Tensor -> Tensor.
LayerNorm over feature dimension (cols) with trainable gamma/beta exposed as
Parameters.Fully-connected layer: y = xW + b
x: [n x dIn], W: [dIn x dOut], b: [1 x dOut]
Llama-style decoder-only transformer.
Pre-LN transformer block with Llama-style full-head attention:
Builder for Llama-style transformer stacks.
Descriptive metadata used to build a Hugging Face model card.
Mean-Squared-Error loss.
Compact Multi-Head Latent Attention (MLA) inspired by DeepSeek-V2/V3.
Multi-head causal self-attention for a single sequence (no batch dimension).
Marker interface for row-wise normalisation layers (LayerNorm, RMSNorm, …).
Simple mutable parameter holder for optimizers.
Optimizer that updates a set of Parameters once per training step.
Learnable positional embeddings added to token embeddings.
Marker interface for linear projection layers (fully-connected, no activation).
Random, bounded-memory batches sampled directly from a UTF-8 text file.
Root-Mean-Square Layer Normalisation — the normalisation used in Llama, Mistral, Qwen, and DeepSeek.
Multi-head self-attention with Rotary Positional Embedding (RoPE).
Rotary Positional Embedding (RoPE) — used in Llama, Mistral, Qwen, DeepSeek, and GPT-NeoX.
Sigmoid Linear Unit (SiLU / Swish).
Row-wise softmax for 2D tensors: applies softmax independently to each row.
Helpers to train classic Tensor->Tensor supervised models (e.g., FNN)
using the unified Trainer wrapper.
SwiGLU feed-forward layer — the FFN used in Llama, Mistral, Qwen, and DeepSeek.
Utilities for converting between
Tensor and flat GPU float arrays.Dataset that samples random contiguous chunks from token ids.
Autoregressive text generation for any decoder-only transformer model.
Trains a compact DeepSeek-style model directly from the TinyStories corpus.
A small, reusable training loop wrapper.
Example: tiny DeepSeek training on a small text file using byte-level tokens.
Example: tiny GPT training on a small text file using byte-level tokens.
Example: tiny GPT training on a small text file using a BPE tokenizer.
Example: tiny Llama training on a small text file using byte-level tokens.
Common configuration fields shared by all decoder-only transformer models.
A sequential stack of transformer blocks.