org.apache.hawq.pxf.plugins.hive

Class HiveDataFragmenter

java.lang.Object

org.apache.hawq.pxf.api.utilities.Plugin

org.apache.hawq.pxf.api.Fragmenter

org.apache.hawq.pxf.plugins.hive.HiveDataFragmenter

Direct Known Subclasses:

HiveInputFormatFragmenter

public class HiveDataFragmenter
extends Fragmenter

Fragmenter class for HIVE tables.
Given a Hive table and its partitions divide the data into fragments (here a data fragment is actually a HDFS file block) and return a list of them. Each data fragment will contain the following information:

1. sourceName: full HDFS path to the data file that this data fragment is part of
2. hosts: a list of the datanode machines that hold a replica of this block
3. userData: file_input_format_name_DELIM_serde_name_DELIM_serialization_properties

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	filterInFragmenter

Fields inherited from class org.apache.hawq.pxf.api.Fragmenter

fragments

Fields inherited from class org.apache.hawq.pxf.api.utilities.Plugin

inputData

Constructor Summary

Constructors

Constructor and Description
HiveDataFragmenter(InputData inputData) Constructs a HiveDataFragmenter object.
HiveDataFragmenter(InputData inputData, java.lang.Class<?> clazz) Constructs a HiveDataFragmenter object.

Method Summary

Modifier and Type	Method and Description
java.util.List<Fragment>	getFragments() Gets the fragments of a given path (source name and location of each fragment).
FragmentsStats	getFragmentsStats() Returns statistics for Hive table.
static org.apache.hadoop.mapred.InputFormat<?,?>	makeInputFormat(java.lang.String inputFormatName, org.apache.hadoop.mapred.JobConf jobConf) Creates the partition InputFormat.

Methods inherited from class org.apache.hawq.pxf.api.utilities.Plugin

isThreadSafe

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

filterInFragmenter

protected boolean filterInFragmenter

Constructor Detail

HiveDataFragmenter

public HiveDataFragmenter(InputData inputData)

Constructs a HiveDataFragmenter object.

Parameters:

inputData - all input parameters coming from the client

HiveDataFragmenter

public HiveDataFragmenter(InputData inputData,
                          java.lang.Class<?> clazz)

Constructs a HiveDataFragmenter object.

Parameters:

inputData - all input parameters coming from the client

clazz - Class for JobConf

Method Detail

getFragments

public java.util.List<Fragment> getFragments()
                                      throws java.lang.Exception

Description copied from class: Fragmenter

Gets the fragments of a given path (source name and location of each fragment). Used to get fragments of data that could be read in parallel from the different segments.

Specified by:

getFragments in class Fragmenter

Returns:

list of data fragments

Throws:

java.lang.Exception - if fragment list could not be retrieved

makeInputFormat

public static org.apache.hadoop.mapred.InputFormat<?,?> makeInputFormat(java.lang.String inputFormatName,
                                                                        org.apache.hadoop.mapred.JobConf jobConf)
                                                                 throws java.lang.Exception

Creates the partition InputFormat.

Parameters:

inputFormatName - input format class name

jobConf - configuration data for the Hadoop framework

Returns:

a InputFormat derived object

Throws:

java.lang.Exception - if failed to create input format

getFragmentsStats

public FragmentsStats getFragmentsStats()
                                 throws java.lang.Exception

Returns statistics for Hive table. Currently it's not implemented.

Overrides:

getFragmentsStats in class Fragmenter

Returns:

default statistics

Throws:

java.lang.Exception - if statistics cannot be gathered