org.kitesdk.data

Class DatasetDescriptor

java.lang.Object

org.kitesdk.data.DatasetDescriptor

@Immutable
public class DatasetDescriptor
extends Object

The structural definition of a Dataset.

Each Dataset has an associated Schema and optional PartitionStrategy defined at the time of creation. You use instances of this class to hold this information. You are strongly encouraged to use the inner DatasetDescriptor.Builder to create new instances.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DatasetDescriptor.Builder A fluent builder to aid in the construction of DatasetDescriptors.

Constructor Summary

Constructors

Constructor and Description
DatasetDescriptor(Schema schema, URI schemaUri, Format format, URI location, Map<String,String> properties, PartitionStrategy partitionStrategy, ColumnMapping columnMapping, CompressionType compressionType) Create an instance of this class with the supplied Schema, optional URL, Format, optional location URL, optional PartitionStrategy, optional ColumnMapping, and optional CompressionType.
DatasetDescriptor(Schema schema, URL schemaUrl, Format format, URI location, Map<String,String> properties, PartitionStrategy partitionStrategy) Create an instance of this class with the supplied Schema, optional URL, Format, optional location URL, and optional PartitionStrategy.
DatasetDescriptor(Schema schema, URL schemaUrl, Format format, URI location, Map<String,String> properties, PartitionStrategy partitionStrategy, ColumnMapping columnMapping) Create an instance of this class with the supplied Schema, optional URL, Format, optional location URL, optional PartitionStrategy, and optional ColumnMapping.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	equals(Object obj)
ColumnMapping	getColumnMapping() Get the ColumnMapping.
CompressionType	getCompressionType() Get the CompressionType
Format	getFormat() Get the associated Format the data is stored in.
URI	getLocation() Get the URL location where the data for this Dataset is stored (optional).
PartitionStrategy	getPartitionStrategy() Get the PartitionStrategy, if this dataset is partitioned.
String	getProperty(String name) Get a named property.
Schema	getSchema() Get the associated Schema.
URL	getSchemaUrl() Get a URL from which the Schema can be retrieved (optional).
int	hashCode()
boolean	hasProperty(String name) Check if a named property exists.
boolean	isColumnMapped() Returns true if an associated dataset is column mapped (that is, has an associated ColumnMapping), false otherwise.
boolean	isPartitioned() Returns true if an associated dataset is partitioned (that is, has an associated PartitionStrategy), false otherwise.
Collection<String>	listProperties() List the names of all custom properties set.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

DatasetDescriptor

public DatasetDescriptor(Schema schema,
                 @Nullable
                 URL schemaUrl,
                 Format format,
                 @Nullable
                 URI location,
                 @Nullable
                 Map<String,String> properties,
                 @Nullable
                 PartitionStrategy partitionStrategy)

Create an instance of this class with the supplied Schema, optional URL, Format, optional location URL, and optional PartitionStrategy.

DatasetDescriptor

public DatasetDescriptor(Schema schema,
                 @Nullable
                 URL schemaUrl,
                 Format format,
                 @Nullable
                 URI location,
                 @Nullable
                 Map<String,String> properties,
                 @Nullable
                 PartitionStrategy partitionStrategy,
                 @Nullable
                 ColumnMapping columnMapping)

Create an instance of this class with the supplied Schema, optional URL, Format, optional location URL, optional PartitionStrategy, and optional ColumnMapping.

Since:

0.14.0

DatasetDescriptor

public DatasetDescriptor(Schema schema,
                 @Nullable
                 URI schemaUri,
                 Format format,
                 @Nullable
                 URI location,
                 @Nullable
                 Map<String,String> properties,
                 @Nullable
                 PartitionStrategy partitionStrategy,
                 @Nullable
                 ColumnMapping columnMapping,
                 @Nullable
                 CompressionType compressionType)

Create an instance of this class with the supplied Schema, optional URL, Format, optional location URL, optional PartitionStrategy, optional ColumnMapping, and optional CompressionType.

Since:

0.17.0

Method Detail

getSchema

public Schema getSchema()

Get the associated Schema. Depending on the underlying storage system, this schema can be simple (that is, records made up of only scalar types) or complex (that is, containing other records, lists, and so on). Validation of the supported schemas is performed by the managing repository, not the dataset or descriptor itself.

Returns:

the schema

getSchemaUrl

@Nullable
public URL getSchemaUrl()

Get a URL from which the Schema can be retrieved (optional). This method might return null if the schema is not stored at a persistent URL (for example, if it were constructed from a literal string).

Returns:

a URL from which the schema can be retrieved

Since:

0.3.0

getFormat

public Format getFormat()

Get the associated Format the data is stored in.

Returns:

the format

Since:

0.2.0

getLocation

@Nullable
public URI getLocation()

Get the URL location where the data for this Dataset is stored (optional).

Returns:

a location URL or null if one is not set

Since:

0.8.0

getProperty

@Nullable
public String getProperty(String name)

Get a named property.

Parameters:

name - the String property name to get.

Returns:

the String value of the property, or null if it does not exist.

Since:

0.8.0

hasProperty

public boolean hasProperty(String name)

Check if a named property exists.

Parameters:

name - the String property name.

Returns:

true if the property exists, false otherwise.

Since:

0.8.0

listProperties

public Collection<String> listProperties()

List the names of all custom properties set.

Returns:

a Collection of String property names.

Since:

0.8.0

getPartitionStrategy

public PartitionStrategy getPartitionStrategy()

Get the PartitionStrategy, if this dataset is partitioned. Calling this method on a non-partitioned dataset is an error. Instead, use the isPartitioned() method prior to invocation.

getColumnMapping

public ColumnMapping getColumnMapping()

Get the ColumnMapping.

Returns:

ColumnMapping

Since:

0.14.0

getCompressionType

public CompressionType getCompressionType()

Get the CompressionType

Returns:

the compression format

Since:

0.17.0

isPartitioned

public boolean isPartitioned()

Returns true if an associated dataset is partitioned (that is, has an associated PartitionStrategy), false otherwise.

isColumnMapped

public boolean isColumnMapped()

Returns true if an associated dataset is column mapped (that is, has an associated ColumnMapping), false otherwise.

Since:

0.14.0

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(@Nullable
             Object obj)

Overrides:

equals in class Object

toString

public String toString()

Overrides:

toString in class Object