Package io.github.kirstenali.deepj.tokenizers

Interface Tokenizer

All Known Implementing Classes:

BPETokenizer, ByteTokenizer

public interface Tokenizer

Method Summary

Modifier and Type	Method	Description
String	decode(int[] ids)
int[]	encode(String text)
int	vocabSize()

Method Details

encode

int[] encode(String text)

decode

String decode(int[] ids)

vocabSize

int vocabSize()