Record Class BPEModel
java.lang.Object
java.lang.Record
io.github.kirstenali.deepj.tokenizers.bpe.BPEModel
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionintReturns the value of theendOfWordIdrecord component.final booleanIndicates whether some other object is "equal to" this one.final inthashCode()Returns a hash code value for this object.List<byte[]> Returns the value of theidToBytesrecord component.merges()Returns the value of themergesrecord component.Returns the value of themergeToNewIdrecord component.Returns the value of thetokenKeyToIdrecord component.final StringtoString()Returns a string representation of this record class.int
-
Constructor Details
-
BPEModel
public BPEModel(List<byte[]> idToBytes, Map<String, Integer> tokenKeyToId, List<TokenPair> merges, Map<TokenPair, Integer> mergeToNewId, int endOfWordId) Creates an instance of aBPEModelrecord class.- Parameters:
idToBytes- the value for theidToBytesrecord componenttokenKeyToId- the value for thetokenKeyToIdrecord componentmerges- the value for themergesrecord componentmergeToNewId- the value for themergeToNewIdrecord componentendOfWordId- the value for theendOfWordIdrecord component
-
-
Method Details
-
vocabSize
public int vocabSize() -
toString
-
hashCode
-
equals
Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared withObjects::equals(Object,Object); primitive components are compared with '=='. -
idToBytes
Returns the value of theidToBytesrecord component.- Returns:
- the value of the
idToBytesrecord component
-
tokenKeyToId
-
merges
-
mergeToNewId
-
endOfWordId
public int endOfWordId()Returns the value of theendOfWordIdrecord component.- Returns:
- the value of the
endOfWordIdrecord component
-