Package io.github.kirstenali.deepj.tokenizers.bpe

Record Class BPEModel

java.lang.Object

java.lang.Record

io.github.kirstenali.deepj.tokenizers.bpe.BPEModel

public record BPEModel(List<byte[]> idToBytes, Map<String,Integer> tokenKeyToId, List<TokenPair> merges, Map<TokenPair,Integer> mergeToNewId, int endOfWordId)
extends Record

Constructor Summary

Constructors

Constructor	Description
BPEModel(List<byte[]> idToBytes, Map<String,Integer> tokenKeyToId, List<TokenPair> merges, Map<TokenPair,Integer> mergeToNewId, int endOfWordId)	Creates an instance of a BPEModel record class.

Method Summary

Modifier and Type	Method	Description
int	endOfWordId()	Returns the value of the endOfWordId record component.
final boolean	equals(Object o)	Indicates whether some other object is "equal to" this one.
final int	hashCode()	Returns a hash code value for this object.
List<byte[]>	idToBytes()	Returns the value of the idToBytes record component.
List<TokenPair>	merges()	Returns the value of the merges record component.
Map<TokenPair,Integer>	mergeToNewId()	Returns the value of the mergeToNewId record component.
Map<String,Integer>	tokenKeyToId()	Returns the value of the tokenKeyToId record component.
final String	toString()	Returns a string representation of this record class.
int	vocabSize()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Details

BPEModel

public BPEModel(List<byte[]> idToBytes,
 Map<String,Integer> tokenKeyToId,
 List<TokenPair> merges,
 Map<TokenPair,Integer> mergeToNewId,
 int endOfWordId)

Creates an instance of a BPEModel record class.

Parameters:

idToBytes - the value for the idToBytes record component

tokenKeyToId - the value for the tokenKeyToId record component

merges - the value for the merges record component

mergeToNewId - the value for the mergeToNewId record component

endOfWordId - the value for the endOfWordId record component

Method Details

vocabSize

public int vocabSize()

toString

public final String toString()

Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.

Specified by:

toString in class Record

Returns:

a string representation of this object

hashCode

public final int hashCode()

Returns a hash code value for this object. The value is derived from the hash code of each of the record components.

Specified by:

hashCode in class Record

Returns:

a hash code value for this object

equals

public final boolean equals(Object o)

Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with '=='.

Specified by:

equals in class Record

Parameters:

o - the object with which to compare

Returns:

true if this object is the same as the o argument; false otherwise.

idToBytes

public List<byte[]> idToBytes()

Returns the value of the idToBytes record component.

Returns:

the value of the idToBytes record component

tokenKeyToId

public Map<String,Integer> tokenKeyToId()

Returns the value of the tokenKeyToId record component.

Returns:

the value of the tokenKeyToId record component

merges

public List<TokenPair> merges()

Returns the value of the merges record component.

Returns:

the value of the merges record component

mergeToNewId

public Map<TokenPair,Integer> mergeToNewId()

Returns the value of the mergeToNewId record component.

Returns:

the value of the mergeToNewId record component

endOfWordId

public int endOfWordId()

Returns the value of the endOfWordId record component.

Returns:

the value of the endOfWordId record component