Uses of Class
io.github.kirstenali.deepj.tokenizers.bpe.BPETokenizer

Packages that use BPETokenizer

Package	Description
io.github.kirstenali.deepj.tokenizers.bpe

Uses of BPETokenizer in io.github.kirstenali.deepj.tokenizers.bpe

Methods in io.github.kirstenali.deepj.tokenizers.bpe that return BPETokenizer

Modifier and Type	Method	Description
BPETokenizer	BPETrainer.trainTokenizer(String text, int targetVocabSize)
BPETokenizer	BPETrainer.trainTokenizer(String text, int targetVocabSize, List<String> specialTokens)
BPETokenizer	BPETrainer.trainTokenizerFromFile(Path path, int targetVocabSize)
BPETokenizer	BPETrainer.trainTokenizerFromFile(Path path, int targetVocabSize, List<String> specialTokens)
BPETokenizer	BPETrainer.trainTokenizerWithDefaults(String text, int targetVocabSize)
BPETokenizer	BPETrainer.trainTokenizerWithDefaultsFromFile(Path path, int targetVocabSize)