Package io.github.kirstenali.deepj.data

Class ResponseOnlyTextDataset

java.lang.Object

io.github.kirstenali.deepj.data.ResponseOnlyTextDataset

All Implemented Interfaces:

BatchSource, StatefulBatchSource

public final class ResponseOnlyTextDataset
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	ResponseOnlyTextDataset.Example
static final record	ResponseOnlyTextDataset.Source

Constructor Summary

Constructors

Constructor	Description
ResponseOnlyTextDataset(List<ResponseOnlyTextDataset.Source> sources, Tokenizer tokenizer, int maxSequenceLength, long seed)

Method Summary

Modifier and Type	Method	Description
List<ResponseOnlyTextDataset.Example>	examples()
final Batch	nextBatch(int batchSize)
final long	randomState()
final void	restoreRandomState(long state)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ResponseOnlyTextDataset

public ResponseOnlyTextDataset(List<ResponseOnlyTextDataset.Source> sources,
 Tokenizer tokenizer,
 int maxSequenceLength,
 long seed)
                        throws IOException

Throws:

IOException

Method Details

examples

public List<ResponseOnlyTextDataset.Example> examples()

nextBatch

public final Batch nextBatch(int batchSize)

Specified by:

nextBatch in interface BatchSource

randomState

public final long randomState()

Specified by:

randomState in interface StatefulBatchSource

restoreRandomState

public final void restoreRandomState(long state)

Specified by:

restoreRandomState in interface StatefulBatchSource