org.deeplearning4j.datasets.iterator.file

Class FileDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.iterator.file.BaseFileIterator<org.nd4j.linalg.dataset.DataSet,org.nd4j.linalg.dataset.api.DataSetPreProcessor>

org.deeplearning4j.datasets.iterator.file.FileDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class FileDataSetIterator
extends BaseFileIterator<org.nd4j.linalg.dataset.DataSet,org.nd4j.linalg.dataset.api.DataSetPreProcessor>
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Iterate over a directory (and optionally subdirectories) containing a number of DataSet objects that have previously been saved to files with DataSet.save(File).
This iterator supports the following (optional) features, depending on the constructor used:
- Recursive listing of all files (i.e., include files in subdirectories)
- Filtering based on a set of file extensions (if null, no filtering - assume all files are saved DataSet objects)
- Randomization of iteration order (default enabled, if a Random instance is provided
- Combining and splitting of DataSets (disabled by default, or if batchSize == -1. If enabled, DataSet objects will be split or combined as required to ensure the specified minibatch size is returned. In other words, the saved DataSet objects can have a different number of examples vs. those returned by the iterator.

Author:

Alex BLack

See Also:

Serialized Form

Field Summary

Fields inherited from class org.deeplearning4j.datasets.iterator.file.BaseFileIterator

batchSize, list, order, position, preProcessor, rng

Constructor Summary

Constructors

Constructor and Description
FileDataSetIterator(File... rootDirs) Create a FileDataSetIterator with the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining) - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileDataSetIterator(File rootDir) Create a FileDataSetIterator with the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining) - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileDataSetIterator(File[] rootDirs, boolean recursive, Random rng, int batchSize, String... validExtensions) Create a FileDataSetIterator with all settings specified
FileDataSetIterator(File rootDir, boolean recursive, Random rng, int batchSize, String... validExtensions) Create a FileDataSetIterator with all settings specified
FileDataSetIterator(File rootDir, int batchSize) Create a FileDataSetIterator with the specified batch size, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileDataSetIterator(File rootDir, int batchSize, String... validExtensions) Create a FileDataSetIterator with the specified batch size, filtering based on file extensions, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed
FileDataSetIterator(File rootDir, String... validExtensions) Create a FileDataSetIterator with filtering based on file extensions, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining)

Method Summary

Modifier and Type	Method and Description
protected void	applyPreprocessor(org.nd4j.linalg.dataset.DataSet toPreProcess)
int	batch()
int	cursor()
int	inputColumns()
protected org.nd4j.linalg.dataset.DataSet	load(File f)
protected org.nd4j.linalg.dataset.DataSet	merge(List<org.nd4j.linalg.dataset.DataSet> toMerge)
org.nd4j.linalg.dataset.DataSet	next(int num)
int	numExamples()
protected int	sizeOf(org.nd4j.linalg.dataset.DataSet of)
protected List<org.nd4j.linalg.dataset.DataSet>	split(org.nd4j.linalg.dataset.DataSet toSplit)
int	totalExamples()
int	totalOutcomes()

Methods inherited from class org.deeplearning4j.datasets.iterator.file.BaseFileIterator

asyncSupported, hasNext, mergeAndStoreRemainder, next, remove, reset, resetSupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported, getLabels, getPreProcessor, reset, resetSupported, setPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining, hasNext, next, remove

Constructor Detail

FileDataSetIterator

public FileDataSetIterator(File rootDir)

Create a FileDataSetIterator with the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDir - Root directory containing the DataSet objects

FileDataSetIterator

public FileDataSetIterator(File... rootDirs)

Create a FileDataSetIterator with the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDirs - Root directories containing the DataSet objects. DataSets from all of these directories will be included in the iterator output

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           int batchSize)

Create a FileDataSetIterator with the specified batch size, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDir - Root directory containing the saved DataSet objects

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           String... validExtensions)

Create a FileDataSetIterator with filtering based on file extensions, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)

Parameters:

rootDir - Root directory containing the saved DataSet objects

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           int batchSize,
                           String... validExtensions)

Create a FileDataSetIterator with the specified batch size, filtering based on file extensions, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed

Parameters:

rootDir - Root directory containing the saved DataSet objects

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           boolean recursive,
                           Random rng,
                           int batchSize,
                           String... validExtensions)

Create a FileDataSetIterator with all settings specified

Parameters:

rootDir - Root directory containing the saved DataSet objects

recursive - If true: include files in subdirectories

rng - May be null. If non-null, use this to randomize order

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileDataSetIterator

public FileDataSetIterator(File[] rootDirs,
                           boolean recursive,
                           Random rng,
                           int batchSize,
                           String... validExtensions)

Create a FileDataSetIterator with all settings specified

Parameters:

rootDirs - Root directories containing the DataSet objects. DataSets from all of these directories will be included in the iterator output

recursive - If true: include files in subdirectories

rng - May be null. If non-null, use this to randomize order

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

Method Detail

load

protected org.nd4j.linalg.dataset.DataSet load(File f)

Specified by:

load in class BaseFileIterator<org.nd4j.linalg.dataset.DataSet,org.nd4j.linalg.dataset.api.DataSetPreProcessor>

sizeOf

protected int sizeOf(org.nd4j.linalg.dataset.DataSet of)

Specified by:

sizeOf in class BaseFileIterator<org.nd4j.linalg.dataset.DataSet,org.nd4j.linalg.dataset.api.DataSetPreProcessor>

split

protected List<org.nd4j.linalg.dataset.DataSet> split(org.nd4j.linalg.dataset.DataSet toSplit)

Specified by:

split in class BaseFileIterator<org.nd4j.linalg.dataset.DataSet,org.nd4j.linalg.dataset.api.DataSetPreProcessor>

merge

protected org.nd4j.linalg.dataset.DataSet merge(List<org.nd4j.linalg.dataset.DataSet> toMerge)

Specified by:

merge in class BaseFileIterator<org.nd4j.linalg.dataset.DataSet,org.nd4j.linalg.dataset.api.DataSetPreProcessor>

applyPreprocessor

protected void applyPreprocessor(org.nd4j.linalg.dataset.DataSet toPreProcess)

Specified by:

applyPreprocessor in class BaseFileIterator<org.nd4j.linalg.dataset.DataSet,org.nd4j.linalg.dataset.api.DataSetPreProcessor>

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

batch

public int batch()

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

cursor

public int cursor()

Specified by:

cursor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

numExamples

public int numExamples()

Specified by:

numExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator