Package io.github.kirstenali.deepj.data

Class RandomAccessTextDataset

java.lang.Object

io.github.kirstenali.deepj.data.RandomAccessTextDataset

All Implemented Interfaces:

BatchSource, AutoCloseable

public final class RandomAccessTextDataset
extends Object
implements BatchSource, AutoCloseable

Random, bounded-memory batches sampled directly from a UTF-8 text file.

Constructor Summary

Constructors

Constructor	Description
RandomAccessTextDataset(Path path, Tokenizer tokenizer, int seqLen, long seed)

Method Summary

Modifier and Type	Method	Description
void	close()
Batch	nextBatch(int batchSize)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

RandomAccessTextDataset

public RandomAccessTextDataset(Path path,
 Tokenizer tokenizer,
 int seqLen,
 long seed)
                        throws IOException

Throws:

IOException

Method Details

nextBatch

public Batch nextBatch(int batchSize)

Specified by:

nextBatch in interface BatchSource

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Throws:

IOException