org.apache.druid.inputsource.hdfs

Class HdfsInputSource

java.lang.Object

org.apache.druid.data.input.AbstractInputSource

org.apache.druid.inputsource.hdfs.HdfsInputSource

All Implemented Interfaces:

SplittableInputSource<List<org.apache.hadoop.fs.Path>>, InputSource

public class HdfsInputSource
extends AbstractInputSource
implements SplittableInputSource<List<org.apache.hadoop.fs.Path>>

Field Summary

Fields inherited from interface org.apache.druid.data.input.impl.SplittableInputSource

DEFAULT_SPLIT_HINT_SPEC

Constructor Summary

Constructors

Constructor and Description
HdfsInputSource(Object inputPaths, org.apache.hadoop.conf.Configuration configuration)

Method Summary

Modifier and Type	Method and Description
static List<String>	coerceInputPathsToList(Object inputPaths, String propertyName)
Stream<InputSplit<List<org.apache.hadoop.fs.Path>>>	createSplits(InputFormat inputFormat, SplitHintSpec splitHintSpec)
int	estimateNumSplits(InputFormat inputFormat, SplitHintSpec splitHintSpec)
protected InputSourceReader	formattableReader(InputRowSchema inputRowSchema, InputFormat inputFormat, File temporaryDirectory)
static Collection<org.apache.hadoop.fs.Path>	getPaths(List<String> inputPaths, org.apache.hadoop.conf.Configuration configuration)
boolean	needsFormat()
SplittableInputSource<List<org.apache.hadoop.fs.Path>>	withSplit(InputSplit<List<org.apache.hadoop.fs.Path>> split)

Methods inherited from class org.apache.druid.data.input.AbstractInputSource

fixedFormatReader, reader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.data.input.impl.SplittableInputSource

getSplitHintSpecOrDefault, isSplittable

Methods inherited from interface org.apache.druid.data.input.InputSource

reader

Constructor Detail

HdfsInputSource

public HdfsInputSource(Object inputPaths,
                       org.apache.hadoop.conf.Configuration configuration)

Method Detail

coerceInputPathsToList

public static List<String> coerceInputPathsToList(Object inputPaths,
                                                  String propertyName)

getPaths

public static Collection<org.apache.hadoop.fs.Path> getPaths(List<String> inputPaths,
                                                             org.apache.hadoop.conf.Configuration configuration)
                                                      throws IOException

Throws:

IOException

formattableReader

protected InputSourceReader formattableReader(InputRowSchema inputRowSchema,
                                              InputFormat inputFormat,
                                              @Nullable
                                              File temporaryDirectory)

Overrides:

formattableReader in class AbstractInputSource

createSplits

public Stream<InputSplit<List<org.apache.hadoop.fs.Path>>> createSplits(InputFormat inputFormat,
                                                                        @Nullable
                                                                        SplitHintSpec splitHintSpec)
                                                                 throws IOException

Specified by:

createSplits in interface SplittableInputSource<List<org.apache.hadoop.fs.Path>>

Throws:

IOException

estimateNumSplits

public int estimateNumSplits(InputFormat inputFormat,
                             @Nullable
                             SplitHintSpec splitHintSpec)
                      throws IOException

Specified by:

estimateNumSplits in interface SplittableInputSource<List<org.apache.hadoop.fs.Path>>

Throws:

IOException

withSplit

public SplittableInputSource<List<org.apache.hadoop.fs.Path>> withSplit(InputSplit<List<org.apache.hadoop.fs.Path>> split)

Specified by:

withSplit in interface SplittableInputSource<List<org.apache.hadoop.fs.Path>>

needsFormat

public boolean needsFormat()

Specified by:

needsFormat in interface InputSource