Package io.github.kirstenali.deepj.tokenizers

Class ByteTokenizer

java.lang.Object

io.github.kirstenali.deepj.tokenizers.ByteTokenizer

All Implemented Interfaces:

Tokenizer

public final class ByteTokenizer
extends Object
implements Tokenizer

Minimal byte-level tokenizers (0-255). This is enough to train a GPT-style model end-to-end without external dependencies. For real projects you can swap this for BPE/Unigram.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	VOCAB_SIZE

Constructor Summary

Constructors

Constructor	Description
ByteTokenizer()

Method Summary

Modifier and Type	Method	Description
String	decode(int[] ids)
int[]	encode(String text)
int	vocabSize()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.tokenizers.Tokenizer

isEndOfSequence

Field Details

VOCAB_SIZE

public static final int VOCAB_SIZE

See Also:

• Constant Field Values

Constructor Details

ByteTokenizer

public ByteTokenizer()

Method Details

encode

public int[] encode(String text)

Specified by:

encode in interface Tokenizer

decode

public String decode(int[] ids)

Specified by:

decode in interface Tokenizer

vocabSize

public int vocabSize()

Specified by:

vocabSize in interface Tokenizer