Package io.github.kirstenali.deepj.tokenizers.bpe

Record Class BPETokenizer

java.lang.Object

java.lang.Record

io.github.kirstenali.deepj.tokenizers.bpe.BPETokenizer

All Implemented Interfaces:

Tokenizer

public record BPETokenizer(BPEModel model)
extends Record
implements Tokenizer

Constructor Summary

Constructors

Constructor	Description
BPETokenizer(BPEModel model)	Creates an instance of a BPETokenizer record class.

Method Summary

Modifier and Type	Method	Description
String	decode(int[] ids)
int[]	encode(String text)
final boolean	equals(Object o)	Indicates whether some other object is "equal to" this one.
final int	hashCode()	Returns a hash code value for this object.
BPEModel	model()	Returns the value of the model record component.
final String	toString()	Returns a string representation of this record class.
int	vocabSize()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Details

BPETokenizer

public BPETokenizer(BPEModel model)

Creates an instance of a BPETokenizer record class.

Parameters:

model - the value for the model record component

Method Details

encode

public int[] encode(String text)

Specified by:

encode in interface Tokenizer

decode

public String decode(int[] ids)

Specified by:

decode in interface Tokenizer

vocabSize

public int vocabSize()

Specified by:

vocabSize in interface Tokenizer

toString

public final String toString()

Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.

Specified by:

toString in class Record

Returns:

a string representation of this object

hashCode

public final int hashCode()

Returns a hash code value for this object. The value is derived from the hash code of each of the record components.

Specified by:

hashCode in class Record

Returns:

a hash code value for this object

equals

public final boolean equals(Object o)

Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. All components in this record class are compared with Objects::equals(Object,Object).

Specified by:

equals in class Record

Parameters:

o - the object with which to compare

Returns:

true if this object is the same as the o argument; false otherwise.

model

public BPEModel model()

Returns the value of the model record component.

Returns:

the value of the model record component