BPETrainer

java.lang.Object

io.github.kirstenali.deepj.tokenizers.bpe.BPETrainer

public final class BPETrainer extends Object

Constructor Summary

Constructors

Constructor

Description

BPETrainer()
Method Summary

Modifier and Type

Method

Description

BPEModel

train(String text, int targetVocabSize)

BPEModel

trainFromFile(Path path, int targetVocabSize)

BPETokenizer

trainTokenizer(String text, int targetVocabSize)

BPETokenizer

trainTokenizerFromFile(Path path, int targetVocabSize)

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details
- BPETrainer
  
  public BPETrainer()
Method Details
- train
  
  public BPEModel train(String text, int targetVocabSize)
- trainFromFile
  
  public BPEModel trainFromFile(Path path, int targetVocabSize) throws IOException
  
  Throws:
  
  IOException
- trainTokenizer
  
  public BPETokenizer trainTokenizer(String text, int targetVocabSize)
- trainTokenizerFromFile
  
  public BPETokenizer trainTokenizerFromFile(Path path, int targetVocabSize) throws IOException
  
  Throws:
  
  IOException