Package io.github.kirstenali.deepj.tokenizers.bpe

Class BPETokenizer

java.lang.Object

io.github.kirstenali.deepj.tokenizers.bpe.BPETokenizer

All Implemented Interfaces:

Tokenizer

public final class BPETokenizer
extends Object
implements Tokenizer

Constructor Summary

Constructors

Constructor	Description
BPETokenizer(BPEModel model)

Method Summary

Modifier and Type	Method	Description
String	decode(int[] ids)
int[]	encode(String text)
boolean	isEndOfSequence(int tokenId)	Returns true when generation should stop before appending this token.
BPEModel	model()
int	vocabSize()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BPETokenizer

public BPETokenizer(BPEModel model)

Method Details

model

public BPEModel model()

encode

public int[] encode(String text)

Specified by:

encode in interface Tokenizer

decode

public String decode(int[] ids)

Specified by:

decode in interface Tokenizer

vocabSize

public int vocabSize()

Specified by:

vocabSize in interface Tokenizer

isEndOfSequence

public boolean isEndOfSequence(int tokenId)

Description copied from interface: Tokenizer

Returns true when generation should stop before appending this token.

Specified by:

isEndOfSequence in interface Tokenizer