org.apache.hawq.pxf.plugins.hive

Class HiveInputFormatFragmenter

java.lang.Object

org.apache.hawq.pxf.api.utilities.Plugin

org.apache.hawq.pxf.api.Fragmenter

org.apache.hawq.pxf.plugins.hive.HiveDataFragmenter

org.apache.hawq.pxf.plugins.hive.HiveInputFormatFragmenter

public class HiveInputFormatFragmenter
extends HiveDataFragmenter

Specialized Hive fragmenter for RC and Text files tables. Unlike the HiveDataFragmenter, this class does not send the serde properties to the accessor/resolvers. This is done to avoid memory explosion in Hawq. For RC use together with HiveRCFileAccessor/ HiveColumnarSerdeResolver. For Text use together with HiveLineBreakAccessor/HiveStringPassResolver.
Given a Hive table and its partitions, divide the data into fragments (here a data fragment is actually a HDFS file block) and return a list of them. Each data fragment will contain the following information:

1. sourceName: full HDFS path to the data file that this data fragment is part of
2. hosts: a list of the datanode machines that hold a replica of this block
3. userData: inputformat name, serde names and partition keys

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HiveInputFormatFragmenter.PXF_HIVE_INPUT_FORMATS Defines the Hive input formats currently supported in pxf
static class	HiveInputFormatFragmenter.PXF_HIVE_SERDES Defines the Hive serializers (serde classes) currently supported in pxf

Field Summary

Fields

Modifier and Type	Field and Description
static int	TOK_FILTER_DONE
static int	TOK_KEYS
static int	TOK_SERDE

Fields inherited from class org.apache.hawq.pxf.plugins.hive.HiveDataFragmenter

filterInFragmenter

Fields inherited from class org.apache.hawq.pxf.api.Fragmenter

fragments

Fields inherited from class org.apache.hawq.pxf.api.utilities.Plugin

inputData

Constructor Summary

Constructors

Constructor and Description
HiveInputFormatFragmenter(InputData inputData) Constructs a HiveInputFormatFragmenter.

Method Summary

Modifier and Type	Method and Description
FragmentsStats	getFragmentsStats() Returns statistics for Hive table.
static java.lang.String[]	parseToks(InputData input, java.lang.String... supportedSerdes) Extracts the user data: serde, partition keys and whether filter was included in fragmenter
static java.lang.String	toHiveType(DataType type, java.lang.String name) Converts HAWQ type to hive type.

Methods inherited from class org.apache.hawq.pxf.plugins.hive.HiveDataFragmenter

getFragments, makeInputFormat

Methods inherited from class org.apache.hawq.pxf.api.utilities.Plugin

isThreadSafe

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOK_SERDE

public static final int TOK_SERDE

See Also:

Constant Field Values

TOK_KEYS

public static final int TOK_KEYS

See Also:

Constant Field Values

TOK_FILTER_DONE

public static final int TOK_FILTER_DONE

See Also:

Constant Field Values

Constructor Detail

HiveInputFormatFragmenter

public HiveInputFormatFragmenter(InputData inputData)

Constructs a HiveInputFormatFragmenter.

Parameters:

inputData - all input parameters coming from the client

Method Detail

parseToks

public static java.lang.String[] parseToks(InputData input,
                                           java.lang.String... supportedSerdes)
                                    throws UserDataException

Extracts the user data: serde, partition keys and whether filter was included in fragmenter

Parameters:

input - input data from client

supportedSerdes - supported serde names

Returns:

parsed tokens

Throws:

UserDataException - if user data contains unsupported serde or wrong number of tokens

toHiveType

public static java.lang.String toHiveType(DataType type,
                                          java.lang.String name)

Converts HAWQ type to hive type. The supported mappings are:

• BOOLEAN -> boolean
• SMALLINT -> smallint (tinyint is converted to smallint)
• BIGINT -> bigint
• TIMESTAMP, TIME -> timestamp
• NUMERIC -> decimal
• BYTEA -> binary
• INTERGER -> int
• TEXT -> string
• REAL -> float
• FLOAT8 -> double

All other types (both in HAWQ and in HIVE) are not supported.

Parameters:

type - HAWQ data type

name - field name

Returns:

Hive type

Throws:

UnsupportedTypeException - if type is not supported

getFragmentsStats

public FragmentsStats getFragmentsStats()
                                 throws java.lang.Exception

Returns statistics for Hive table. Currently it's not implemented.

Overrides:

getFragmentsStats in class HiveDataFragmenter

Returns:

default statistics

Throws:

java.lang.Exception - if statistics cannot be gathered