org.apache.hawq.pxf.plugins.hdfs.utilities

Class HdfsUtilities

java.lang.Object

org.apache.hawq.pxf.plugins.hdfs.utilities.HdfsUtilities

public class HdfsUtilities
extends java.lang.Object

HdfsUtilities class exposes helper methods for PXF classes.

Constructor Summary

Constructors

Constructor and Description
HdfsUtilities()

Method Summary

Modifier and Type	Method and Description
static java.lang.String	absoluteDataPath(java.lang.String dataSource) Hdfs data sources are absolute data paths.
static org.apache.avro.Schema	getAvroSchema(org.apache.hadoop.conf.Configuration conf, java.lang.String dataSource) Accessing the Avro file through the "unsplittable" API just to get the schema.
static org.apache.hadoop.io.compress.CompressionCodec	getCodec(org.apache.hadoop.conf.Configuration conf, java.lang.String name) Helper routine to get compression codec through reflection.
static boolean	isSplittableCodec(org.apache.hadoop.fs.Path path) Returns true if the needed codec is splittable.
static boolean	isThreadSafe(java.lang.String dataDir, java.lang.String compCodec) Checks if requests should be handle in a single thread or not.
static org.apache.hadoop.mapred.FileSplit	parseFragmentMetadata(InputData inputData) Parses fragment metadata and return matching FileSplit.
static byte[]	prepareFragmentMetadata(org.apache.hadoop.mapred.FileSplit fsp) Prepares byte serialization of a file split information (start, length, hosts) using ObjectOutputStream.
static java.lang.String	toString(java.util.List<OneField> complexRecord, java.lang.String delimiter) Returns string serialization of list of fields.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HdfsUtilities

public HdfsUtilities()

Method Detail

absoluteDataPath

public static java.lang.String absoluteDataPath(java.lang.String dataSource)

Hdfs data sources are absolute data paths. Method ensures that dataSource begins with '/'.

Parameters:

dataSource - The HDFS path to a file or directory of interest. Retrieved from the client request.

Returns:

an absolute data path

getCodec

public static org.apache.hadoop.io.compress.CompressionCodec getCodec(org.apache.hadoop.conf.Configuration conf,
                                                                      java.lang.String name)

Helper routine to get compression codec through reflection.

Parameters:

conf - configuration used for reflection

name - codec name

Returns:

generated CompressionCodec

isSplittableCodec

public static boolean isSplittableCodec(org.apache.hadoop.fs.Path path)

Returns true if the needed codec is splittable. If no codec is needed returns true as well.

Parameters:

path - path of the file to be read

Returns:

if the codec needed for reading the specified path is splittable.

isThreadSafe

public static boolean isThreadSafe(java.lang.String dataDir,
                                   java.lang.String compCodec)

Checks if requests should be handle in a single thread or not.

Parameters:

dataDir - hdfs path to the data source

compCodec - the fully qualified name of the compression codec

Returns:

if the request can be run in multi-threaded mode.

prepareFragmentMetadata

public static byte[] prepareFragmentMetadata(org.apache.hadoop.mapred.FileSplit fsp)
                                      throws java.io.IOException

Prepares byte serialization of a file split information (start, length, hosts) using ObjectOutputStream.

Parameters:

fsp - file split to be serialized

Returns:

byte serialization of fsp

Throws:

java.io.IOException - if I/O errors occur while writing to the underlying stream

parseFragmentMetadata

public static org.apache.hadoop.mapred.FileSplit parseFragmentMetadata(InputData inputData)

Parses fragment metadata and return matching FileSplit.

Parameters:

inputData - request input data

Returns:

FileSplit with fragment metadata

getAvroSchema

public static org.apache.avro.Schema getAvroSchema(org.apache.hadoop.conf.Configuration conf,
                                                   java.lang.String dataSource)
                                            throws java.io.IOException

Accessing the Avro file through the "unsplittable" API just to get the schema. The splittable API (AvroInputFormat) which is the one we will be using to fetch the records, does not support getting the Avro schema yet.

Parameters:

conf - Hadoop configuration

dataSource - Avro file (i.e fileName.avro) path

Returns:

the Avro schema

Throws:

java.io.IOException - if I/O error occurred while accessing Avro schema file

toString

public static java.lang.String toString(java.util.List<OneField> complexRecord,
                                        java.lang.String delimiter)

Returns string serialization of list of fields. Fields of binary type (BYTEA) are converted to octal representation to make sure they will be relayed properly to the DB.

Parameters:

complexRecord - list of fields to be stringified

delimiter - delimiter between fields

Returns:

string of serialized fields using delimiter