org.deeplearning4j.datasets.datavec

Class RecordReaderDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class RecordReaderDataSetIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Record reader dataset iterator. Takes a DataVec RecordReader as input, and handles the conversion to ND4J DataSet objects as well as producing minibatches from individual records.

Multiple constructors are available, though a RecordReaderDataSetIterator.Builder class is also available.

Example 1: Image classification, batch size 32, 10 classes

  RecordReader rr = new ImageRecordReader(28,28,3); //28x28 RGB images
  rr.initialize(new FileSplit(new File("/path/to/directory")));

  DataSetIterator iter = new RecordReaderDataSetIterator.Builder(rr, 32)
       //Label index (first arg): Always value 1 when using ImageRecordReader. For CSV etc: use index of the column
       //  that contains the label (should contain an integer value, 0 to nClasses-1 inclusive). Column indexes start
       // at 0. Number of classes (second arg): number of label classes (i.e., 10 for MNIST - 10 digits)
       .classification(1, nClasses)
       .preProcessor(new ImagePreProcessingScaler())      //For normalization of image values 0-255 to 0-1
       .build()
 
 

Example 2: Multi-output regression from CSV, batch size 128

  RecordReader rr = new CsvRecordReader(0, ','); //Skip 0 header lines, comma separated
  rr.initialize(new FileSplit(new File("/path/to/myCsv.txt")));

  DataSetIterator iter = new RecordReaderDataSetIterator.Builder(rr, 128)
       //Specify the columns that the regression labels/targets appear in. Note that all other columns will be
       // treated as features. Columns indexes start at 0
       .regression(labelColFrom, labelColTo)
       .build()
 
 

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RecordReaderDataSetIterator.Builder Builder class for RecordReaderDataSetIterator

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchNum
protected int	batchSize
protected org.datavec.api.io.WritableConverter	converter
protected int	labelIndex
protected int	labelIndexTo
protected org.nd4j.linalg.dataset.DataSet	last
protected int	maxNumBatches
protected int	numPossibleLabels
protected org.nd4j.linalg.dataset.api.DataSetPreProcessor	preProcessor
protected org.datavec.api.records.reader.RecordReader	recordReader
protected boolean	regression
protected Iterator<List<org.datavec.api.writable.Writable>>	sequenceIter
protected boolean	useCurrent

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	RecordReaderDataSetIterator(RecordReaderDataSetIterator.Builder b)
	RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize) Constructor for classification, where: (a) the label index is assumed to be the very last Writable/column, and (b) the number of classes is inferred from RecordReader.getLabels() Note that if RecordReader.getLabels() returns null, no output labels will be produced
	RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize, int labelIndex, int numPossibleLabels) Main constructor for classification.
	RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize, int labelIndexFrom, int labelIndexTo, boolean regression) Main constructor for multi-label regression (i.e., regression with multiple outputs)
	RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize, int labelIndex, int numPossibleLabels, int maxNumBatches) Constructor for classification, where the maximum number of returned batches is limited to the specified value
	RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, org.datavec.api.io.WritableConverter converter, int batchSize, int labelIndexFrom, int labelIndexTo, int numPossibleLabels, int maxNumBatches, boolean regression) Main constructor

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported()
int	batch()
int	cursor()
List<String>	getLabels()
boolean	hasNext()
int	inputColumns()
org.nd4j.linalg.dataset.DataSet	loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list) Load a multiple examples to a DataSet, using the provided RecordMetaData instances.
org.nd4j.linalg.dataset.DataSet	loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData) Load a single example to a DataSet, using the provided RecordMetaData.
org.nd4j.linalg.dataset.DataSet	next()
org.nd4j.linalg.dataset.DataSet	next(int num)
int	numExamples()
void	remove()
void	reset()
boolean	resetSupported()
void	setCollectMetaData(boolean collectMetaData) When set to true: metadata for the current examples will be present in the returned DataSet.
void	setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)
int	totalExamples()
int	totalOutcomes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

recordReader

protected org.datavec.api.records.reader.RecordReader recordReader

converter

protected org.datavec.api.io.WritableConverter converter

batchSize

protected int batchSize

maxNumBatches

protected int maxNumBatches

batchNum

protected int batchNum

labelIndex

protected int labelIndex

labelIndexTo

protected int labelIndexTo

numPossibleLabels

protected int numPossibleLabels

sequenceIter

protected Iterator<List<org.datavec.api.writable.Writable>> sequenceIter

last

protected org.nd4j.linalg.dataset.DataSet last

useCurrent

protected boolean useCurrent

regression

protected boolean regression

preProcessor

protected org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor

Constructor Detail

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize)

Constructor for classification, where:
(a) the label index is assumed to be the very last Writable/column, and
(b) the number of classes is inferred from RecordReader.getLabels()
Note that if RecordReader.getLabels() returns null, no output labels will be produced

Parameters:

recordReader - Record reader to use as the source of data

batchSize - Minibatch size, for each call of .next()

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels)

Main constructor for classification. This will convert the input class index (at position labelIndex, with integer values 0 to numPossibleLabels-1 inclusive) to the appropriate one-hot output/labels representation.

Parameters:

recordReader - RecordReader: provides the source of the data

batchSize - Batch size (number of examples) for the output DataSet objects

labelIndex - Index of the label Writable (usually an IntWritable), as obtained by recordReader.next()

numPossibleLabels - Number of classes (possible labels) for classification

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels,
                                   int maxNumBatches)

Constructor for classification, where the maximum number of returned batches is limited to the specified value

Parameters:

recordReader - the recordreader to use

labelIndex - the index/column of the label (for classification)

numPossibleLabels - the number of possible labels for classification. Not used if regression == true

maxNumBatches - The maximum number of batches to return between resets. Set to -1 to return all available data

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize,
                                   int labelIndexFrom,
                                   int labelIndexTo,
                                   boolean regression)

Main constructor for multi-label regression (i.e., regression with multiple outputs)

Parameters:

recordReader - RecordReader to get data from

labelIndexFrom - Index of the first regression target

labelIndexTo - Index of the last regression target, inclusive

batchSize - Minibatch size

regression - Require regression = true. Mainly included to avoid clashing with other constructors previously defined :/

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   org.datavec.api.io.WritableConverter converter,
                                   int batchSize,
                                   int labelIndexFrom,
                                   int labelIndexTo,
                                   int numPossibleLabels,
                                   int maxNumBatches,
                                   boolean regression)

Main constructor

Parameters:

recordReader - the recordreader to use

converter - Converter. May be null.

batchSize - Minibatch size - number of examples returned for each call of .next()

labelIndexFrom - the index of the label (for classification), or the first index of the labels for multi-output regression

labelIndexTo - only used if regression == true. The last index inclusive of the multi-output regression

numPossibleLabels - the number of possible labels for classification. Not used if regression == true

maxNumBatches - Maximum number of batches to return

regression - if true: regression. If false: classification (assume labelIndexFrom is the class it belongs to)

RecordReaderDataSetIterator

protected RecordReaderDataSetIterator(RecordReaderDataSetIterator.Builder b)

Method Detail

setCollectMetaData

public void setCollectMetaData(boolean collectMetaData)

When set to true: metadata for the current examples will be present in the returned DataSet. Disabled by default.

Parameters:

collectMetaData - Whether to collect metadata or not

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

batch

public int batch()

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

cursor

public int cursor()

Specified by:

cursor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

numExamples

public int numExamples()

Specified by:

numExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.DataSet>

next

public org.nd4j.linalg.dataset.DataSet next()

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.DataSet>

remove

public void remove()

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.DataSet>

getLabels

public List<String> getLabels()

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

loadFromMetaData

public org.nd4j.linalg.dataset.DataSet loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData)
                                                 throws IOException

Load a single example to a DataSet, using the provided RecordMetaData. Note that it is more efficient to load multiple instances at once, using loadFromMetaData(List)

Parameters:

recordMetaData - RecordMetaData to load from. Should have been produced by the given record reader

Returns:

DataSet with the specified example

Throws:

IOException - If an error occurs during loading of the data

loadFromMetaData

public org.nd4j.linalg.dataset.DataSet loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list)
                                                 throws IOException

Load a multiple examples to a DataSet, using the provided RecordMetaData instances.

Parameters:

list - List of RecordMetaData instances to load from. Should have been produced by the record reader provided to the RecordReaderDataSetIterator constructor

Returns:

DataSet with the specified examples

Throws:

IOException - If an error occurs during loading of the data