Uses of Interface
io.github.kirstenali.deepj.tokenizers.Tokenizer

Packages that use Tokenizer

Package	Description
io.github.kirstenali.deepj.data
io.github.kirstenali.deepj.models
io.github.kirstenali.deepj.tokenizers
io.github.kirstenali.deepj.tokenizers.bpe

Uses of Tokenizer in io.github.kirstenali.deepj.data

Methods in io.github.kirstenali.deepj.data with parameters of type Tokenizer

Modifier and Type	Method	Description
static TextDataset	TextDataset.fromFile(Path path, Tokenizer tok, int seqLen, long seed)	Stream-tokenize a text file and memory-map the result.

Constructors in io.github.kirstenali.deepj.data with parameters of type Tokenizer

Modifier	Constructor	Description
	RandomAccessTextDataset(Path path, Tokenizer tokenizer, int seqLen, long seed)

Uses of Tokenizer in io.github.kirstenali.deepj.models

Methods in io.github.kirstenali.deepj.models with parameters of type Tokenizer

Modifier and Type	Method	Description
static String	TextGenerator.generate(DeepSeekModel model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(GPTModel model, Tokenizer tok, GPTConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(LlamaModel model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(Function<int[],Tensor> forwarder, int maxSeqLen, Tokenizer tok, String prompt, int maxNewTokens, float temperature, int topK, long seed)	Generate text with an explicit maxSeqLen — useful when no config is available.
static String	TextGenerator.generate(Function<int[],Tensor> forwarder, TransformerConfig cfg, Tokenizer tok, String prompt, int maxNewTokens, float temperature, int topK, long seed)	Generate text using any model that maps int[] ids → [seqLen × vocabSize] logits.

Uses of Tokenizer in io.github.kirstenali.deepj.tokenizers

Classes in io.github.kirstenali.deepj.tokenizers that implement Tokenizer

Modifier and Type	Class	Description
final class	ByteTokenizer	Minimal byte-level tokenizers (0-255).

Uses of Tokenizer in io.github.kirstenali.deepj.tokenizers.bpe

Classes in io.github.kirstenali.deepj.tokenizers.bpe that implement Tokenizer

Modifier and Type	Class	Description
final class	BPETokenizer