REST Resources

Data Avenue REST interface is accessible at URL https://data-avenue.eu/blacktop-dev/rest, followed by one of the resources listed below:

• /directory - for directory (folder, bucket, container) lising, creation, deletion

• /file - for file (object) upload, download, deletion

• /attributes - for getting/setting file, directory attributes (permissions, size, date of last modification, etc.)

• /aliases - for creating/deleting aliases for remote files (accessed via HTTP tunneling)

• /transfers - for starting/aborting/getting status of inter-storage data tranfers

• /protocols - for getting the set of supported storage protocols

• /authentication - for getting possible authentication mechanisms for storage protocols

• /operations - for getting possible operation types implemented in Data Avenue

• /version - for getting Data Avenue version

REST interface can be used with command line tools as well such as curl. For example:

returns Data Avenue version (-k is to ignore certicate verification).

Data Avenue Authentication (Key)

Each REST call must contain a special header field, named "X-Key", containing the access key (a.k.a. "ticket") required to use Data Avenue services. It can be requested on page: ticket request form.

You can use curl's -H (--header) option set the key for the request:

To avoid saving your key into command history, you can create a file key with a single line "X-Key: de305d54-75b4-431b-adb2-eb6b9e546014", and use cat in curl:

(Storage) Credentials

In order to allow Data Avenue to access the storage of your choice, you have to temporarily delegate your credentials for successful mediation. Such credentials are passed via HTTP header elements. The most common authentication type is the username-password authentication (access key/secret key pair, in the case of S3), but you can also authenticate using x509 proxy (e.g., GridFTP). In the latter case, proxy need to be base64 encoded in the header. These credentials have to put into headers named "X-Username", "X-Password", "X-Proxy" as appropriate.

To pass username and password with curl, use -H option:

To add a proxy, stored in file x509up,  to HTTP header you can use the base64 encoder inline:

URIs

 Data Avenue identifies remote files residing on storage resources using URIs (Uniform Resource Identifiers) of the form: protocol://storage-address/path/[file]. (Directory URIs end with /.)

Examples of URIs are:

Data Avenue REST operations (file, directory, attributes, transfers) require passing the URI of the remote file on which some operation needs to be performed. The URI is passed in "X-URI" HTTP header field.

With curl you can do this like:

/directory (resource)

GET

Returns the list of directory enty names (files and subdirectories) of the spefified remote directory (X-URI header) in the form of JSON array (application/json response).

POST

Creates the specified directory (X-URI header).

DELETE

Deletes the specified directory (X-URI header), recursively.

/file (resource)

GET

Download file contents of the specified remote file (X-URI header) as application/octet-stream.

POST/PUT

Uploads a new remote file (X-URI header) to a remote storage.

DELETE

Deletes the specified remote file (X-URI header).

/attributes (resource)

GET

Returns the attributes of the specified remote file or directory (X-URI header). The result is a JSON object with keys: name, size (in bytes, in the case of file), date of last modification (epoch in millis), permissions (in linux-like format, e.g., xrw-rw-rw, in owner, group, all users permissions order), and comment (optional).

POST

Returns the set of attributes of the set of specified remote files or directories in the specified remote directory (X-URI header). The set of entries of interest should be given in entiry body as a JSON array. The result is a JSON array of JSON objects with keys descibed in GET. This operation is used typically together with directory listing (GET /directory) to get entry details.

/aliases (resource)

POST

Creates an (HTTP) alias for the specified remote file (X-URI header), so the file contents of the remote file can be read/written by simply reading/writing (HTTP GET/PUT) the alias URL, which points to the Data Avenue server, via HTTP tunneling. Optionally, parameters can be specified for alias creation (in the entiry body in JSON format): lifetime (of the alias in seconds, default: 1 hour), read (read-only alias, boolean, default: true), redirect (allowed if applicable, default: true), and archive (extract archive on reading/writing, boolean,  default: false). The method returns alias details in entiry body in JSON format with keys: (alias) id, URL (on Data Avenue server), direct URL (direct storage URL, if applicable).

DELETE

Deletes the specified alias (path parameter) - before expiration.

/transfers (resource)

GET

Returns the status of the specified transfer (path parameter). The response is a JSON object (entity body) with keys: source, target URIs, state (CREATED, TRANSFERRING, COMPETED, FAILED), bytes transferred (so far, in bytes), size (file size in bytes), started, ended, server time (in epoch), failure cause (on error).

POST

Starts a new data transfer task, which copies or moves the source file or directory (X-URI) from one storage to another, target storage. Request entiry body specifies the target in JSON format with keys: target (URI, file or directory, required), credentials (for target storage, in JSON format, e.g. {Type:UserPass, UserID:username, UserPass:password}), move (optional, boolean, default: false), overwrite (optional, boolean, default: false)   (specified in the entiry body) 

DELETE

Aborts a currently active transfer (path parameter).

/protocols (resource)

GET

Returns the list of storage protocols that Data Avenue supports in the response body as a JSON array.

/authentications (resource)

GET

Returns the list of authentication types supported by the specified storage protocol (path parameter).

GET

Returns Data Avenue version (in plain text).