Package io.github.kirstenali.deepj.tokenizers

Interface Tokenizer

All Known Implementing Classes:
BPETokenizer, ByteTokenizer
public interface Tokenizer

  • Method Details

    • encode

      int[] encode(String text)

    • decode

      String decode(int[] ids)

    • vocabSize

      int vocabSize()

    • isEndOfSequence

      default boolean isEndOfSequence(int tokenId)