org.apache.hawq.pxf.plugins.hdfs

Class HdfsDataFragmenter

java.lang.Object

org.apache.hawq.pxf.api.utilities.Plugin

org.apache.hawq.pxf.api.Fragmenter

org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter

public class HdfsDataFragmenter
extends Fragmenter

Fragmenter class for HDFS data resources. Given an HDFS data source (a file, directory, or wild card pattern) divide the data into fragments and return a list of them along with a list of host:port locations for each.

Field Summary

Fields inherited from class org.apache.hawq.pxf.api.Fragmenter

fragments

Fields inherited from class org.apache.hawq.pxf.api.utilities.Plugin

inputData

Constructor Summary

Constructors

Constructor and Description
HdfsDataFragmenter(InputData md) Constructs an HdfsDataFragmenter object.

Method Summary

Modifier and Type	Method and Description
java.util.List<Fragment>	getFragments() Gets the fragments for a data source URI that can appear as a file name, a directory name or a wildcard.
FragmentsStats	getFragmentsStats() Default implementation of statistics for fragments.

Methods inherited from class org.apache.hawq.pxf.api.utilities.Plugin

isThreadSafe

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HdfsDataFragmenter

public HdfsDataFragmenter(InputData md)

Constructs an HdfsDataFragmenter object.

Parameters:

md - all input parameters coming from the client

Method Detail

getFragments

public java.util.List<Fragment> getFragments()
                                      throws java.lang.Exception

Gets the fragments for a data source URI that can appear as a file name, a directory name or a wildcard. Returns the data fragments in JSON format.

Specified by:

getFragments in class Fragmenter

Returns:

list of data fragments

Throws:

java.lang.Exception - if fragment list could not be retrieved

getFragmentsStats

public FragmentsStats getFragmentsStats()
                                 throws java.lang.Exception

Description copied from class: Fragmenter

Default implementation of statistics for fragments. The default is:

• number of fragments - as gathered by Fragmenter.getFragments()
• first fragment size - 64MB
• total size - number of fragments times first fragment size

Each fragmenter implementation can override this method to better match its fragments stats.

Overrides:

getFragmentsStats in class Fragmenter

Returns:

default statistics

Throws:

java.lang.Exception - if statistics cannot be gathered