Package io.github.kirstenali.deepj.data

Class TextDataset

java.lang.Object

io.github.kirstenali.deepj.data.TextDataset

public final class TextDataset
extends Object

Simple in-memory dataset that samples random contiguous chunks from token ids. This is intentionally minimal but correct; for large corpora, replace with a memory-mapped or streaming implementation.

Constructor Summary

Constructors

Constructor	Description
TextDataset(int[] tokens, int seqLen, long seed)

Method Summary

Modifier and Type	Method	Description
static TextDataset	fromFile(Path path, Tokenizer tok, int seqLen, long seed)
Batch	nextBatch(int batchSize)
int	seqLen()
int	size()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TextDataset

public TextDataset(int[] tokens,
 int seqLen,
 long seed)

Method Details

fromFile

public static TextDataset fromFile(Path path,
 Tokenizer tok,
 int seqLen,
 long seed)
                            throws IOException

Throws:

IOException

nextBatch

public Batch nextBatch(int batchSize)

seqLen

public int seqLen()

size

public int size()