public class HdfsUtilities
extends java.lang.Object
Constructor and Description |
---|
HdfsUtilities() |
Modifier and Type | Method and Description |
---|---|
static java.lang.String |
absoluteDataPath(java.lang.String dataSource)
Hdfs data sources are absolute data paths.
|
static org.apache.avro.Schema |
getAvroSchema(org.apache.hadoop.conf.Configuration conf,
java.lang.String dataSource)
Accessing the Avro file through the "unsplittable" API just to get the
schema.
|
static org.apache.hadoop.io.compress.CompressionCodec |
getCodec(org.apache.hadoop.conf.Configuration conf,
java.lang.String name)
Helper routine to get compression codec through reflection.
|
static boolean |
isSplittableCodec(org.apache.hadoop.fs.Path path)
Returns true if the needed codec is splittable.
|
static boolean |
isThreadSafe(java.lang.String dataDir,
java.lang.String compCodec)
Checks if requests should be handle in a single thread or not.
|
static org.apache.hadoop.mapred.FileSplit |
parseFragmentMetadata(InputData inputData)
Parses fragment metadata and return matching
FileSplit . |
static byte[] |
prepareFragmentMetadata(org.apache.hadoop.mapred.FileSplit fsp)
Prepares byte serialization of a file split information (start, length,
hosts) using
ObjectOutputStream . |
static java.lang.String |
toString(java.util.List<OneField> complexRecord,
java.lang.String delimiter)
Returns string serialization of list of fields.
|
public static java.lang.String absoluteDataPath(java.lang.String dataSource)
dataSource
- The HDFS path to a file or directory of interest.
Retrieved from the client request.public static org.apache.hadoop.io.compress.CompressionCodec getCodec(org.apache.hadoop.conf.Configuration conf, java.lang.String name)
conf
- configuration used for reflectionname
- codec namepublic static boolean isSplittableCodec(org.apache.hadoop.fs.Path path)
path
- path of the file to be readpublic static boolean isThreadSafe(java.lang.String dataDir, java.lang.String compCodec)
dataDir
- hdfs path to the data sourcecompCodec
- the fully qualified name of the compression codecpublic static byte[] prepareFragmentMetadata(org.apache.hadoop.mapred.FileSplit fsp) throws java.io.IOException
ObjectOutputStream
.fsp
- file split to be serializedjava.io.IOException
- if I/O errors occur while writing to the underlying
streampublic static org.apache.hadoop.mapred.FileSplit parseFragmentMetadata(InputData inputData)
FileSplit
.inputData
- request input datapublic static org.apache.avro.Schema getAvroSchema(org.apache.hadoop.conf.Configuration conf, java.lang.String dataSource) throws java.io.IOException
conf
- Hadoop configurationdataSource
- Avro file (i.e fileName.avro) pathjava.io.IOException
- if I/O error occurred while accessing Avro schema filepublic static java.lang.String toString(java.util.List<OneField> complexRecord, java.lang.String delimiter)
complexRecord
- list of fields to be stringifieddelimiter
- delimiter between fields