Package io.github.kirstenali.deepj.tokenizers.bpe

Record Class BPEModel

java.lang.Object
java.lang.Record
io.github.kirstenali.deepj.tokenizers.bpe.BPEModel
public record BPEModel(List<byte[]> idToBytes, Map<String,Integer> tokenKeyToId, List<TokenPair> merges, Map<TokenPair,Integer> mergeToNewId, int endOfWordId, int modelFormatVersion, Map<String,Integer> specialTokenToId) extends Record

  • Field Details

    • CURRENT_FORMAT_VERSION

      public static final int CURRENT_FORMAT_VERSION
      See Also:

  • Constructor Details

    • BPEModel

      public BPEModel(List<byte[]> idToBytes, Map<String,Integer> tokenKeyToId, List<TokenPair> merges, Map<TokenPair,Integer> mergeToNewId, int endOfWordId)

    • BPEModel

      public BPEModel(List<byte[]> idToBytes, Map<String,Integer> tokenKeyToId, List<TokenPair> merges, Map<TokenPair,Integer> mergeToNewId, int endOfWordId, int modelFormatVersion, Map<String,Integer> specialTokenToId)
      Creates an instance of a BPEModel record class.
      Parameters:
      idToBytes - the value for the idToBytes record component
      tokenKeyToId - the value for the tokenKeyToId record component
      merges - the value for the merges record component
      mergeToNewId - the value for the mergeToNewId record component
      endOfWordId - the value for the endOfWordId record component
      modelFormatVersion - the value for the modelFormatVersion record component
      specialTokenToId - the value for the specialTokenToId record component

  • Method Details

    • idToBytes

      public List<byte[]> idToBytes()
      Returns the value of the idToBytes record component.
      Returns:
      the value of the idToBytes record component

    • specialTokenToId

      public Map<String,Integer> specialTokenToId()
      Returns the value of the specialTokenToId record component.
      Returns:
      the value of the specialTokenToId record component

    • vocabSize

      public int vocabSize()

    • hasSpecialTokens

      public boolean hasSpecialTokens()

    • idToSpecialToken

      public Map<Integer,String> idToSpecialToken()

    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object

    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object

    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with '=='.
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.

    • tokenKeyToId

      public Map<String,Integer> tokenKeyToId()
      Returns the value of the tokenKeyToId record component.
      Returns:
      the value of the tokenKeyToId record component

    • merges

      public List<TokenPair> merges()
      Returns the value of the merges record component.
      Returns:
      the value of the merges record component

    • mergeToNewId

      public Map<TokenPair,Integer> mergeToNewId()
      Returns the value of the mergeToNewId record component.
      Returns:
      the value of the mergeToNewId record component

    • endOfWordId

      public int endOfWordId()
      Returns the value of the endOfWordId record component.
      Returns:
      the value of the endOfWordId record component

    • modelFormatVersion

      public int modelFormatVersion()
      Returns the value of the modelFormatVersion record component.
      Returns:
      the value of the modelFormatVersion record component