org.deeplearning4j.datasets.datavec

Class SequenceRecordReaderDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class SequenceRecordReaderDataSetIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Sequence record reader data set iterator Given a record reader (and optionally another record reader for the labels) generate time series (sequence) data sets.
Supports padding for one-to-many and many-to-one type data loading (i.e., with different number of inputs vs. labels via the SequenceRecordReaderDataSetIterator.AlignmentMode mode.

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SequenceRecordReaderDataSetIterator.AlignmentMode Alignment mode for dealing with input/labels of differing lengths (for example, one-to-many and many-to-one type situations).

Constructor Summary

Constructors

Constructor and Description
SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader reader, int miniBatchSize, int numPossibleLabels, int labelIndex) Constructor where features and labels come from the SAME RecordReader (i.e., target/label is a column in the same data as the features).
SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader reader, int miniBatchSize, int numPossibleLabels, int labelIndex, boolean regression) Constructor where features and labels come from the SAME RecordReader (i.e., target/label is a column in the same data as the features)
SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader featuresReader, org.datavec.api.records.reader.SequenceRecordReader labels, int miniBatchSize, int numPossibleLabels) Constructor where features and labels come from different RecordReaders (for example, different files), and labels are for classification.
SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader featuresReader, org.datavec.api.records.reader.SequenceRecordReader labels, int miniBatchSize, int numPossibleLabels, boolean regression) Constructor where features and labels come from different RecordReaders (for example, different files)
SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader featuresReader, org.datavec.api.records.reader.SequenceRecordReader labels, int miniBatchSize, int numPossibleLabels, boolean regression, SequenceRecordReaderDataSetIterator.AlignmentMode alignmentMode) Constructor where features and labels come from different RecordReaders (for example, different files)

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported()
int	batch()
int	cursor()
List<String>	getLabels()
boolean	hasNext()
int	inputColumns()
org.nd4j.linalg.dataset.DataSet	loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list) Load a multiple sequence examples to a DataSet, using the provided RecordMetaData instances.
org.nd4j.linalg.dataset.DataSet	loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData) Load a single sequence example to a DataSet, using the provided RecordMetaData.
org.nd4j.linalg.dataset.DataSet	next()
org.nd4j.linalg.dataset.DataSet	next(int num)
int	numExamples()
void	remove()
void	reset()
boolean	resetSupported()
void	setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)
int	totalExamples()
int	totalOutcomes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

SequenceRecordReaderDataSetIterator

public SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader featuresReader,
                                           org.datavec.api.records.reader.SequenceRecordReader labels,
                                           int miniBatchSize,
                                           int numPossibleLabels)

Constructor where features and labels come from different RecordReaders (for example, different files), and labels are for classification.

Parameters:

featuresReader - SequenceRecordReader for the features

labels - Labels: assume single value per time step, where values are integers in the range 0 to numPossibleLables-1

miniBatchSize - Minibatch size for each call of next()

numPossibleLabels - Number of classes for the labels

SequenceRecordReaderDataSetIterator

public SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader featuresReader,
                                           org.datavec.api.records.reader.SequenceRecordReader labels,
                                           int miniBatchSize,
                                           int numPossibleLabels,
                                           boolean regression)

Constructor where features and labels come from different RecordReaders (for example, different files)

SequenceRecordReaderDataSetIterator

public SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader featuresReader,
                                           org.datavec.api.records.reader.SequenceRecordReader labels,
                                           int miniBatchSize,
                                           int numPossibleLabels,
                                           boolean regression,
                                           SequenceRecordReaderDataSetIterator.AlignmentMode alignmentMode)

Constructor where features and labels come from different RecordReaders (for example, different files)

SequenceRecordReaderDataSetIterator

public SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader reader,
                                           int miniBatchSize,
                                           int numPossibleLabels,
                                           int labelIndex)

Constructor where features and labels come from the SAME RecordReader (i.e., target/label is a column in the same data as the features). Defaults to regression = false - i.e., for classification

Parameters:

reader - SequenceRecordReader with data

miniBatchSize - size of each minibatch

numPossibleLabels - number of labels/classes for classification

labelIndex - index in input of the label index. If in regression mode and numPossibleLabels > 1, labelIndex denotes the first index for labels. Everything before that index will be treated as input(s) and everything from that index (inclusive) to the end will be treated as output(s)

SequenceRecordReaderDataSetIterator

public SequenceRecordReaderDataSetIterator(org.datavec.api.records.reader.SequenceRecordReader reader,
                                           int miniBatchSize,
                                           int numPossibleLabels,
                                           int labelIndex,
                                           boolean regression)

Constructor where features and labels come from the SAME RecordReader (i.e., target/label is a column in the same data as the features)

Parameters:

reader - SequenceRecordReader with data

miniBatchSize - size of each minibatch

numPossibleLabels - number of labels/classes for classification

labelIndex - index in input of the label index. If in regression mode and numPossibleLabels > 1, labelIndex denotes the first index for labels. Everything before that index will be treated as input(s) and everything from that index (inclusive) to the end will be treated as output(s)

regression - Whether output is for regression or classification

Method Detail

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.DataSet>

next

public org.nd4j.linalg.dataset.DataSet next()

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.DataSet>

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

batch

public int batch()

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

cursor

public int cursor()

Specified by:

cursor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

numExamples

public int numExamples()

Specified by:

numExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getLabels

public List<String> getLabels()

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

remove

public void remove()

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.DataSet>

loadFromMetaData

public org.nd4j.linalg.dataset.DataSet loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData)
                                                 throws IOException

Load a single sequence example to a DataSet, using the provided RecordMetaData. Note that it is more efficient to load multiple instances at once, using loadFromMetaData(List)

Parameters:

recordMetaData - RecordMetaData to load from. Should have been produced by the given record reader

Returns:

DataSet with the specified example

Throws:

IOException - If an error occurs during loading of the data

loadFromMetaData

public org.nd4j.linalg.dataset.DataSet loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list)
                                                 throws IOException

Load a multiple sequence examples to a DataSet, using the provided RecordMetaData instances.

Parameters:

list - List of RecordMetaData instances to load from. Should have been produced by the record reader provided to the SequenceRecordReaderDataSetIterator constructor

Returns:

DataSet with the specified examples

Throws:

IOException - If an error occurs during loading of the data