Package io.github.kirstenali.deepj.data

Class TextDataset

java.lang.Object

io.github.kirstenali.deepj.data.TextDataset

All Implemented Interfaces:

BatchSource

public final class TextDataset
extends Object
implements BatchSource

Dataset that samples random contiguous chunks from token ids.

When created via fromFile(java.nio.file.Path, io.github.kirstenali.deepj.tokenizers.Tokenizer, int, long), the source text is streamed line-by-line, tokenized one or more complete lines at a time, and written to a temporary binary file that is then memory-mapped. The full file never needs to fit in Java heap, although a single unusually long line must fit while it is tokenized.

Method Summary

Modifier and Type	Method	Description
static TextDataset	fromFile(Path path, Tokenizer tok, int seqLen, long seed)	Stream-tokenize a text file and memory-map the result.
Batch	nextBatch(int batchSize)
int	seqLen()
long	size()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

fromFile

public static TextDataset fromFile(Path path,
 Tokenizer tok,
 int seqLen,
 long seed)
                            throws IOException

Stream-tokenize a text file and memory-map the result. The text is read in bounded chunks (split on line boundaries) so that neither the raw text nor the full token array need to fit in heap.

Throws:

IOException

nextBatch

public Batch nextBatch(int batchSize)

Specified by:

nextBatch in interface BatchSource

seqLen

public int seqLen()

size

public long size()