org.deeplearning4j.datasets.iterator.impl

Class EmnistDataSetIterator

java.lang.Object

org.nd4j.linalg.dataset.api.iterator.BaseDatasetIterator

org.deeplearning4j.datasets.iterator.impl.EmnistDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class EmnistDataSetIterator
extends org.nd4j.linalg.dataset.api.iterator.BaseDatasetIterator

EMNIST DataSetIterator
EMNIST is similar to the common MNIST dataset (available via MnistDataSetIterator), with 6 different splits/ variants, specified by EmnistDataSetIterator.Set:

• COMPLETE: Also known as 'ByClass' split. 814,255 examples total (train + test), 62 classes
• MERGE: Also known as 'ByMerge' split. 814,255 examples total. 47 unbalanced classes. Combines lower and upper case characters (that are difficult to distinguish) into one class for each letter (instead of 2), for letters C, I, J, K, L, M, O, P, S, U, V, W, X, Y and Z
• BALANCED: 131,600 examples total. 47 classes (equal number of examples in each class)
• LETTERS: 145,600 examples total. 26 balanced classes
• DIGITS: 280,000 examples total. 10 balanced classes
• MNIST: 70,000 examples total. 10 balanced classes. Equivalent to the original MNIST dataset in MnistDataSetIterator

See: https://www.nist.gov/itl/iad/image-group/emnist-dataset and https://arxiv.org/abs/1702.05373

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	EmnistDataSetIterator.Set EMNIST dataset has multiple different subsets.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batch
protected EmnistDataSetIterator.Set	dataSet
protected int	numExamples
protected org.nd4j.linalg.dataset.api.DataSetPreProcessor	preProcessor

Fields inherited from class org.nd4j.linalg.dataset.api.iterator.BaseDatasetIterator

fetcher

Constructor Summary

Constructors

Constructor and Description
EmnistDataSetIterator(EmnistDataSetIterator.Set dataSet, int batch, boolean train)
EmnistDataSetIterator(EmnistDataSetIterator.Set dataSet, int batch, boolean binarize, boolean train, boolean shuffle, long rngSeed) Get the specified number of MNIST examples (test or train set), with optional shuffling and binarization.
EmnistDataSetIterator(EmnistDataSetIterator.Set dataSet, int batchSize, boolean train, long seed)

Method Summary

Modifier and Type	Method and Description
List<String>	getLabels() Get the labels as a List
static List<String>	getLabels(EmnistDataSetIterator.Set dataSet) Get the label assignments for the given set as a List
static char[]	getLabelsArray(EmnistDataSetIterator.Set dataSet) Get the label assignments for the given set as a character array.
char[]	getLabelsArrays() Get the labels as a character array
static boolean	isBalanced(EmnistDataSetIterator.Set dataSet) Are the labels balanced in the training set (that is: are the number of examples for each label equal?)
static int	numExamplesTest(EmnistDataSetIterator.Set dataSet) Get the number of test examples for the specified subset
static int	numExamplesTrain(EmnistDataSetIterator.Set dataSet) Get the number of training examples for the specified subset
static int	numLabels(EmnistDataSetIterator.Set dataSet) Get the number of labels for the specified subset

Methods inherited from class org.nd4j.linalg.dataset.api.iterator.BaseDatasetIterator

asyncSupported, batch, cursor, getPreProcessor, hasNext, inputColumns, next, next, numExamples, remove, reset, resetSupported, setPreProcessor, totalExamples, totalOutcomes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

dataSet

protected EmnistDataSetIterator.Set dataSet

batch

protected int batch

numExamples

protected int numExamples

preProcessor

protected org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor

Constructor Detail

EmnistDataSetIterator

public EmnistDataSetIterator(EmnistDataSetIterator.Set dataSet,
                             int batch,
                             boolean train)
                      throws IOException

Throws:

IOException

EmnistDataSetIterator

public EmnistDataSetIterator(EmnistDataSetIterator.Set dataSet,
                             int batchSize,
                             boolean train,
                             long seed)
                      throws IOException

Throws:

IOException

EmnistDataSetIterator

public EmnistDataSetIterator(EmnistDataSetIterator.Set dataSet,
                             int batch,
                             boolean binarize,
                             boolean train,
                             boolean shuffle,
                             long rngSeed)
                      throws IOException

Get the specified number of MNIST examples (test or train set), with optional shuffling and binarization.

Parameters:

batch - Size of each patch

binarize - whether to binarize the data or not (if false: normalize in range 0 to 1)

train - Train vs. test set

shuffle - whether to shuffle the examples

rngSeed - random number generator seed to use when shuffling examples

Throws:

IOException

Method Detail

numExamplesTrain

public static int numExamplesTrain(EmnistDataSetIterator.Set dataSet)

Get the number of training examples for the specified subset

Parameters:

dataSet - Subset to get

Returns:

Number of examples for the specified subset

numExamplesTest

public static int numExamplesTest(EmnistDataSetIterator.Set dataSet)

Get the number of test examples for the specified subset

Parameters:

dataSet - Subset to get

Returns:

Number of examples for the specified subset

numLabels

public static int numLabels(EmnistDataSetIterator.Set dataSet)

Get the number of labels for the specified subset

Parameters:

dataSet - Subset to get

Returns:

Number of labels for the specified subset

getLabelsArrays

public char[] getLabelsArrays()

Get the labels as a character array

Returns:

Labels

getLabels

public List<String> getLabels()

Get the labels as a List

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Overrides:

getLabels in class org.nd4j.linalg.dataset.api.iterator.BaseDatasetIterator

Returns:

Labels

getLabelsArray

public static char[] getLabelsArray(EmnistDataSetIterator.Set dataSet)

Get the label assignments for the given set as a character array.

Parameters:

dataSet - DataSet to get the label assignment for

Returns:

Label assignment and given dataset

getLabels

public static List<String> getLabels(EmnistDataSetIterator.Set dataSet)

Get the label assignments for the given set as a List

Parameters:

dataSet - DataSet to get the label assignment for

Returns:

Label assignment and given dataset

isBalanced

public static boolean isBalanced(EmnistDataSetIterator.Set dataSet)

Are the labels balanced in the training set (that is: are the number of examples for each label equal?)

Parameters:

dataSet - Set to get balanced value for

Returns:

True if balanced dataset, false otherwise