org.kitesdk.data

Class Datasets

java.lang.Object

org.kitesdk.data.Datasets

public class Datasets
extends Object

Methods for working with Dataset instances.

URIs

All methods require a URI that identifies a dataset, view, or repository. The URI must begin with the scheme dataset:, view:, or repo:. The remainder of the URI is implementation specific, depending on the dataset scheme.

For example, the URI dataset:hive:movies/ratings references a dataset named ratings in the movies namespace, stored in Hive.

The URI view:hive:movies/ratings?year=2015&month=3 references a view of the same ratings dataset. The view is filtered to include records from only March, 2015.

See Dataset and View URIs for the available URI patterns.

Dataset Descriptors

Some methods require a DatasetDescriptor that encapsulates metadata about a dataset. Descriptors are built using a descriptor builder.

Entities

Entities are analagous to records in database terminology. The term is used in the API to emphasize that an entity can include not only primitive objects, but also complex objects such as hash maps.

Some methods accept an entity class that will be used by Kite when returning entities from a dataset or view.

Since:

0.8.0

Constructor Summary

Constructors

Constructor and Description
Datasets()

Method Summary

Methods

Modifier and Type	Method and Description
static <V extends View<GenericRecord>> V	create(String uri, DatasetDescriptor descriptor) Create a Dataset for the given dataset or view URI string.
static <E,V extends View<E>> V	create(String uri, DatasetDescriptor descriptor, Class<E> type) Create a Dataset for the given dataset or view URI string.
static <V extends View<GenericRecord>> V	create(URI uri, DatasetDescriptor descriptor) Create a Dataset for the given dataset or view URI.
static <E,V extends View<E>> V	create(URI uri, DatasetDescriptor descriptor, Class<E> type) Create a Dataset for the given dataset or view URI.
static boolean	delete(String uri) Delete a Dataset identified by the given dataset URI string.
static boolean	delete(URI uri) Delete a Dataset identified by the given dataset URI.
static boolean	exists(String uri) Check whether a Dataset identified by the given URI string exists.
static boolean	exists(URI uri) Check whether a Dataset identified by the given URI exists.
static Collection<URI>	list(String uri) List the Dataset URIs in the repository identified by the URI string.
static Collection<URI>	list(URI uri) List the Dataset URIs in the repository identified by the URI.
static <V extends View<GenericRecord>> V	load(String uriString) Load a Dataset or View for the given URI.
static <E,V extends View<E>> V	load(String uriString, Class<E> type) Load a Dataset or View for the given URI.
static <V extends View<GenericRecord>> V	load(URI uri) Load a Dataset or View for the given URI.
static <E,V extends View<E>> V	load(URI uri, Class<E> type) Load a Dataset or View for the given URI.
static <D extends Dataset<GenericRecord>> D	update(String uri, DatasetDescriptor descriptor) Update a Dataset for the given dataset or view URI string.
static <E,D extends Dataset<E>> D	update(String uri, DatasetDescriptor descriptor, Class<E> type) Update a Dataset for the given dataset or view URI string.
static <D extends Dataset<GenericRecord>> D	update(URI uri, DatasetDescriptor descriptor) Update a Dataset for the given dataset or view URI.
static <E,D extends Dataset<E>> D	update(URI uri, DatasetDescriptor descriptor, Class<E> type) Update a Dataset for the given dataset or view URI.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Datasets

public Datasets()

Method Detail

load

public static <E,V extends View<E>> V load(URI uri,
                           Class<E> type)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of View expected

Parameters:

uri - a Dataset or View URI

type - a Java class that represents an entity in the dataset

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

NullPointerException - if any arguments are null

IllegalArgumentException - if uri is not a dataset or view URI

load

public static <V extends View<GenericRecord>> V load(URI uri)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

V - the type of View expected

Parameters:

uri - a Dataset or View URI

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

NullPointerException - if any arguments are null

IllegalArgumentException - if uri is not a dataset or view URI

load

public static <E,V extends View<E>> V load(String uriString,
                           Class<E> type)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of View expected

Parameters:

uriString - a Dataset or View URI

type - a Java class that represents an entity in the dataset

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

NullPointerException - if any arguments are null

IllegalArgumentException - if uri is not a dataset or view URI

load

public static <V extends View<GenericRecord>> V load(String uriString)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

V - the type of View expected

Parameters:

uriString - a Dataset or View URI

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

NullPointerException - if any arguments are null

IllegalArgumentException - if uri is not a dataset or view URI

create

public static <E,V extends View<E>> V create(URI uri,
                             DatasetDescriptor descriptor,
                             Class<E> type)

Create a Dataset for the given dataset or view URI. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI

type - a Java class that represents an entity in the dataset

Returns:

a newly created Dataset responsible for the given URI

Throws:

NullPointerException - if uri, descriptor, or type is null

IllegalArgumentException - if uri is not a dataset or view URI

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

create

public static <V extends View<GenericRecord>> V create(URI uri,
                                       DatasetDescriptor descriptor)

Create a Dataset for the given dataset or view URI. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI

Returns:

a newly created Dataset responsible for the given URI

Throws:

NullPointerException - if uri or descriptor is null

IllegalArgumentException - if uri is not a dataset or view URI

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

create

public static <E,V extends View<E>> V create(String uri,
                             DatasetDescriptor descriptor,
                             Class<E> type)

Create a Dataset for the given dataset or view URI string. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI string

type - a Java class that represents an entity in the dataset

Returns:

a newly created Dataset responsible for the given URI

Throws:

NullPointerException - if uri, descriptor, or type is null

IllegalArgumentException - if uri is not a dataset or view URI

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

create

public static <V extends View<GenericRecord>> V create(String uri,
                                       DatasetDescriptor descriptor)

Create a Dataset for the given dataset or view URI string. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI string

Returns:

a newly created Dataset responsible for the given URI

Throws:

NullPointerException - if uri or descriptor is null

IllegalArgumentException - if uri is not a dataset or view URI

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

update

public static <E,D extends Dataset<E>> D update(URI uri,
                                DatasetDescriptor descriptor,
                                Class<E> type)

Update a Dataset for the given dataset or view URI.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

E - the type used for readers and writers created by this Dataset

D - the type of Dataset expected

Parameters:

uri - a Dataset URI

type - a Java class that represents an entity in the dataset

Returns:

a Dataset for the given URI

Throws:

NullPointerException - if uri, descriptor, or type is null

IllegalArgumentException - if uri is not a dataset URI

DatasetNotFoundException - if there is no dataset for the given URI

UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

update

public static <D extends Dataset<GenericRecord>> D update(URI uri,
                                          DatasetDescriptor descriptor)

Update a Dataset for the given dataset or view URI.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

D - the type of Dataset expected

Parameters:

uri - a Dataset URI

Returns:

a Dataset for the given URI

Throws:

NullPointerException - if uri or descriptor is null

IllegalArgumentException - if uri is not a dataset URI

DatasetNotFoundException - if there is no dataset for the given URI

UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

update

public static <E,D extends Dataset<E>> D update(String uri,
                                DatasetDescriptor descriptor,
                                Class<E> type)

Update a Dataset for the given dataset or view URI string.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

E - the type used for readers and writers created by this Dataset

D - the type of Dataset expected

Parameters:

uri - a Dataset URI string

type - a Java class that represents an entity in the dataset

Returns:

a Dataset for the given URI

Throws:

NullPointerException - if uri, descriptor, or type is null

IllegalArgumentException - if uri is not a dataset URI

DatasetNotFoundException - if there is no dataset for the given URI

UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

update

public static <D extends Dataset<GenericRecord>> D update(String uri,
                                          DatasetDescriptor descriptor)

Update a Dataset for the given dataset or view URI string.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

D - the type of Dataset expected

Parameters:

uri - a Dataset URI string

Returns:

a Dataset for the given URI

Throws:

NullPointerException - if uri or descriptor is null

IllegalArgumentException - if uri is not a dataset URI

DatasetNotFoundException - if there is no dataset for the given URI

UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

delete

public static boolean delete(URI uri)

Delete a Dataset identified by the given dataset URI.

When you call this method using a dataset URI, both data and metadata are deleted. After you call this method, the dataset no longer exists, unless an exception is thrown.

When you call this method using a view URI, data in that view is deleted. The dataset's metadata is not changed. This can throw an UnsupportedOperationException if the delete requires additional work. For example, if some, but not all, of the data in an underlying data file must be removed, then the implementation is allowed to reject the deletion rather than copy the remaining records to a new file. An implementation must document under what conditions it accepts deletes, and under what conditions it rejects them.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI

Returns:

true if any data or metadata is removed, false otherwise

Throws:

NullPointerException - if uri is null

IllegalArgumentException - if uri is not a dataset URI

delete

public static boolean delete(String uri)

Delete a Dataset identified by the given dataset URI string.

When you call this method using a dataset URI, both data and metadata are deleted. After you call this method, the dataset no longer exists, unless an exception is thrown.

When you call this method using a view URI, data in that view is deleted. The dataset's metadata is not changed. This can throw an UnsupportedOperationException if the delete requires additional work. For example, if some, but not all, of the data in an underlying data file must be removed, then the implementation is allowed to reject the deletion rather than copy the remaining records to a new file. An implementation must document under what conditions it accepts deletes, and under what conditions it rejects them.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI string

Returns:

true if any data or metadata is removed, false otherwise

Throws:

NullPointerException - if uri is null

IllegalArgumentException - if uri is not a dataset URI

exists

public static boolean exists(URI uri)

Check whether a Dataset identified by the given URI exists.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI

Returns:

true if the dataset exists, false otherwise

Throws:

NullPointerException - if uri is null

IllegalArgumentException - if uri is not a dataset URI

exists

public static boolean exists(String uri)

Check whether a Dataset identified by the given URI string exists.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI string

Returns:

true if the dataset exists, false otherwise

Throws:

NullPointerException - if uri is null

IllegalArgumentException - if uri is not a dataset URI

list

public static Collection<URI> list(URI uri)

List the Dataset URIs in the repository identified by the URI.

URI formats are defined by Dataset implementations. The repository URIs you pass to this method must begin with repo:. For example, to list the Dataset URIs for the Hive repository, provide the URI repo:hive.

Parameters:

uri - a DatasetRepository URI

Returns:

the URIs present in the DatasetRepository

Throws:

NullPointerException - if uri is null

IllegalArgumentException - if uri is not a repository URI

list

public static Collection<URI> list(String uri)

List the Dataset URIs in the repository identified by the URI string.

URI formats are defined by Dataset implementations. The repository URIs you pass to this method must begin with repo:. For example, to list the Dataset URIs for the Hive repository, provide the URI repo:hive.

Parameters:

uri - a DatasetRepository URI string

Returns:

the URIs present in the DatasetRepository

Throws:

NullPointerException - if URI is null

IllegalArgumentException - if uri is not a repository URI