Uses of Interface
io.github.kirstenali.deepj.tokenizers.Tokenizer

Packages that use Tokenizer

Package	Description
io.github.kirstenali.deepj.data
io.github.kirstenali.deepj.nn.transformer
io.github.kirstenali.deepj.tokenizers
io.github.kirstenali.deepj.tokenizers.bpe
io.github.kirstenali.deepj.training

Uses of Tokenizer in io.github.kirstenali.deepj.data

Methods in io.github.kirstenali.deepj.data with parameters of type Tokenizer

Modifier and Type	Method	Description
static TextDataset	TextDataset.fromFile(Path path, Tokenizer tok, int seqLen, long seed)

Constructors in io.github.kirstenali.deepj.data with parameters of type Tokenizer

Modifier	Constructor	Description
	IndexedResponseTextDataset(List<ResponseOnlyTextDataset.Source> sources, Tokenizer tokenizer, int maxSequenceLength, long seed)
	RandomAccessTextDataset(Path path, Tokenizer tokenizer, int seqLen, long seed)
	RandomAccessTextDataset(Path path, Tokenizer tokenizer, int seqLen, long seed, TextFileRange range)
	ResponseOnlyTextDataset(List<ResponseOnlyTextDataset.Source> sources, Tokenizer tokenizer, int maxSequenceLength, long seed)
	SequentialTextDataset(Path path, Tokenizer tokenizer, int sequenceLength)
	SequentialTextDataset(Path path, Tokenizer tokenizer, int sequenceLength, TextFileRange range)

Uses of Tokenizer in io.github.kirstenali.deepj.nn.transformer

Methods in io.github.kirstenali.deepj.nn.transformer with parameters of type Tokenizer

Modifier and Type	Method	Description
static String	TextGenerator.generate(CausalLM model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(CausalLM model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed, float repetitionPenalty)
static String	TextGenerator.generate(DeepJOrbit model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(DeepJOrigin model, Tokenizer tok, DeepJOriginConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(DeepJPrism model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(Function<int[],Tensor> forwarder, int maxSeqLen, Tokenizer tok, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.generate(Function<int[],Tensor> forwarder, int maxSeqLen, Tokenizer tok, String prompt, int maxNewTokens, float temperature, int topK, long seed, float repetitionPenalty)
static String	TextGenerator.generate(Function<int[],Tensor> forwarder, TransformerConfig cfg, Tokenizer tok, String prompt, int maxNewTokens, float temperature, int topK, long seed)
static String	TextGenerator.stream(CausalLM model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed, float repetitionPenalty, Consumer<String> listener)
static String	TextGenerator.stream(CausalLM model, Tokenizer tok, TransformerConfig cfg, String prompt, int maxNewTokens, float temperature, int topK, long seed, Consumer<String> listener)
static String	TextGenerator.stream(Function<int[],Tensor> forwarder, int maxSeqLen, Tokenizer tok, String prompt, int maxNewTokens, float temperature, int topK, long seed, float repetitionPenalty, Consumer<String> listener)
static String	TextGenerator.stream(Function<int[],Tensor> forwarder, int maxSeqLen, Tokenizer tok, String prompt, int maxNewTokens, float temperature, int topK, long seed, Consumer<String> listener)

Uses of Tokenizer in io.github.kirstenali.deepj.tokenizers

Classes in io.github.kirstenali.deepj.tokenizers that implement Tokenizer

Modifier and Type	Class	Description
final class	ByteTokenizer

Uses of Tokenizer in io.github.kirstenali.deepj.tokenizers.bpe

Classes in io.github.kirstenali.deepj.tokenizers.bpe that implement Tokenizer

Modifier and Type	Class	Description
final class	BPETokenizer

Uses of Tokenizer in io.github.kirstenali.deepj.training

Methods in io.github.kirstenali.deepj.training with parameters of type Tokenizer

Modifier and Type	Method	Description
static EvaluationResult	FixedTextValidation.evaluate(CausalLM model, Path corpus, Tokenizer tokenizer, FixedTextValidation.Settings settings)
static EvaluationResult	FixedTextValidation.evaluate(CausalLM model, Path corpus, Tokenizer tokenizer, FixedTextValidation.Settings settings, TextFileRange range)