org.deeplearning4j.datasets.datavec

Class RecordReaderMultiDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>, org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

public class RecordReaderMultiDataSetIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator, Serializable

RecordReaderMultiDataSetIterator: A MultiDataSetIterator for data from one or more RecordReaders and SequenceRecordReaders
The idea: generate multiple inputs and multiple outputs from one or more Sequence/RecordReaders. Inputs and outputs may be obtained from subsets of the RecordReader and SequenceRecordReaders columns (for examples, some inputs and outputs as different columns in the same record/sequence); it is also possible to mix different types of data (for example, using both RecordReaders and SequenceRecordReaders in the same RecordReaderMultiDataSetIterator).
Uses a builder pattern (RecordReaderMultiDataSetIterator.Builder to specify the various inputs and subsets.

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RecordReaderMultiDataSetIterator.AlignmentMode When dealing with time series data of different lengths, how should we align the input/labels time series? For equal length: use EQUAL_LENGTH For sequence classification: use ALIGN_END
static class	RecordReaderMultiDataSetIterator.Builder

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported()
org.nd4j.linalg.dataset.api.MultiDataSetPreProcessor	getPreProcessor()
boolean	hasNext()
org.nd4j.linalg.dataset.api.MultiDataSet	loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list) Load a multiple sequence examples to a DataSet, using the provided RecordMetaData instances.
org.nd4j.linalg.dataset.api.MultiDataSet	loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData) Load a single example to a DataSet, using the provided RecordMetaData.
org.nd4j.linalg.dataset.api.MultiDataSet	next()
org.nd4j.linalg.dataset.api.MultiDataSet	next(int num)
org.nd4j.linalg.dataset.api.MultiDataSet	nextMultiDataSet(Map<String,List<List<org.datavec.api.writable.Writable>>> nextRRVals, Map<String,List<org.nd4j.linalg.api.ndarray.INDArray>> nextRRValsBatched, Map<String,List<List<List<org.datavec.api.writable.Writable>>>> nextSeqRRVals, List<org.datavec.api.records.metadata.RecordMetaDataComposableMap> nextMetas)
void	remove()
void	reset()
boolean	resetSupported()
void	setPreProcessor(org.nd4j.linalg.dataset.api.MultiDataSetPreProcessor preProcessor)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Method Detail

next

public org.nd4j.linalg.dataset.api.MultiDataSet next()

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>

remove

public void remove()

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>

next

public org.nd4j.linalg.dataset.api.MultiDataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

nextMultiDataSet

public org.nd4j.linalg.dataset.api.MultiDataSet nextMultiDataSet(Map<String,List<List<org.datavec.api.writable.Writable>>> nextRRVals,
                                                                 Map<String,List<org.nd4j.linalg.api.ndarray.INDArray>> nextRRValsBatched,
                                                                 Map<String,List<List<List<org.datavec.api.writable.Writable>>>> nextSeqRRVals,
                                                                 List<org.datavec.api.records.metadata.RecordMetaDataComposableMap> nextMetas)

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.MultiDataSetPreProcessor preProcessor)

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

getPreProcessor

public org.nd4j.linalg.dataset.api.MultiDataSetPreProcessor getPreProcessor()

Specified by:

getPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>

loadFromMetaData

public org.nd4j.linalg.dataset.api.MultiDataSet loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData)
                                                          throws IOException

Load a single example to a DataSet, using the provided RecordMetaData. Note that it is more efficient to load multiple instances at once, using loadFromMetaData(List)

Parameters:

recordMetaData - RecordMetaData to load from. Should have been produced by the given record reader

Returns:

DataSet with the specified example

Throws:

IOException - If an error occurs during loading of the data

loadFromMetaData

public org.nd4j.linalg.dataset.api.MultiDataSet loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list)
                                                          throws IOException

Load a multiple sequence examples to a DataSet, using the provided RecordMetaData instances.

Parameters:

list - List of RecordMetaData instances to load from. Should have been produced by the record reader provided to the SequenceRecordReaderDataSetIterator constructor

Returns:

DataSet with the specified examples

Throws:

IOException - If an error occurs during loading of the data