Package io.github.kirstenali.deepj.tokenizers

Interface Tokenizer

All Known Implementing Classes:
BPETokenizer, ByteTokenizer
public interface Tokenizer

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    decode(int[] ids)
     
    int[]
    encode(String text)
     
    default boolean
    isEndOfSequence(int tokenId)
    Returns true when generation should stop before appending this token.
    int
    vocabSize()
     

  • Method Details

    • encode

      int[] encode(String text)

    • decode

      String decode(int[] ids)

    • vocabSize

      int vocabSize()

    • isEndOfSequence

      default boolean isEndOfSequence(int tokenId)
      Returns true when generation should stop before appending this token.