REST Resources
Data Avenue REST interface is accessible at URL https://data-avenue.eu/blacktop-dev/rest, followed by one of the resources listed below:
-
/directory - for directory (folder, bucket, container) lising, creation, deletion
-
/file - for file (object) upload, download, deletion
-
/attributes - for getting/setting file, directory attributes (permissions, size, date of last modification, etc.)
-
/aliases - for creating/deleting aliases for remote files (accessed via HTTP tunneling)
-
/transfers - for starting/aborting/getting status of inter-storage data tranfers
-
/protocols - for getting the set of supported storage protocols
-
/authentication - for getting possible authentication mechanisms for storage protocols
-
/operations - for getting possible operation types implemented in Data Avenue
-
/version - for getting Data Avenue version
REST interface can be used with command line tools as well such as curl. For example:
returns Data Avenue version (-k is to ignore certicate verification).
Data Avenue Authentication (Key)
Each REST call must contain a special header field, named "X-Key", containing the access key (a.k.a. "ticket") required to use Data Avenue services. It can be requested on page: ticket request form.
You can use curl's -H (--header) option set the key for the request:
To avoid saving your key into command history, you can create a file key with a single line "X-Key: de305d54-75b4-431b-adb2-eb6b9e546014", and use cat in curl:
(Storage) Credentials
In order to allow Data Avenue to access the storage of your choice, you have to temporarily delegate your credentials for successful mediation. Such credentials are passed via HTTP header elements. The most common authentication type is the username-password authentication (access key/secret key pair, in the case of S3), but you can also authenticate using x509 proxy (e.g., GridFTP). In the latter case, proxy need to be base64 encoded in the header. These credentials have to put into headers named "X-Username", "X-Password", "X-Proxy" as appropriate.
To pass username and password with curl, use -H option:
To add a proxy, stored in file x509up, to HTTP header you can use the base64 encoder inline:
URIs
Data Avenue identifies remote files residing on storage resources using URIs (Uniform Resource Identifiers) of the form: protocol://storage-address/path/[file]. (Directory URIs end with /.)
Examples of URIs are:
Data Avenue REST operations (file, directory, attributes, transfers) require passing the URI of the remote file on which some operation needs to be performed. The URI is passed in "X-URI" HTTP header field.
With curl you can do this like:
/directory (resource)
GET
Returns the list of directory enty names (files and subdirectories) of the spefified remote directory (X-URI header) in the form of JSON array (application/json response).
POST
Creates the specified directory (X-URI header).
DELETE
Deletes the specified directory (X-URI header), recursively.
/file (resource)
GET
Download file contents of the specified remote file (X-URI header) as application/octet-stream.
POST/PUT
Uploads a new remote file (X-URI header) to a remote storage.
DELETE
Deletes the specified remote file (X-URI header).
/attributes (resource)
GET
Returns the attributes of the specified remote file or directory (X-URI header). The result is a JSON object with keys: name, size (in bytes, in the case of file), date of last modification (epoch in millis), permissions (in linux-like format, e.g., xrw-rw-rw, in owner, group, all users permissions order), and comment (optional).
POST
Returns the set of attributes of the set of specified remote files or directories in the specified remote directory (X-URI header). The set of entries of interest should be given in entiry body as a JSON array. The result is a JSON array of JSON objects with keys descibed in GET. This operation is used typically together with directory listing (GET /directory) to get entry details.
/aliases (resource)
POST
Creates an (HTTP) alias for the specified remote file (X-URI header), so the file contents of the remote file can be read/written by simply reading/writing (HTTP GET/PUT) the alias URL, which points to the Data Avenue server, via HTTP tunneling. Optionally, parameters can be specified for alias creation (in the entiry body in JSON format): lifetime (of the alias in seconds, default: 1 hour), read (read-only alias, boolean, default: true), redirect (allowed if applicable, default: true), and archive (extract archive on reading/writing, boolean, default: false). The method returns alias details in entiry body in JSON format with keys: (alias) id, URL (on Data Avenue server), direct URL (direct storage URL, if applicable).
DELETE
Deletes the specified alias (path parameter) - before expiration.
/transfers (resource)
GET
Returns the status of the specified transfer (path parameter). The response is a JSON object (entity body) with keys: source, target URIs, state (CREATED, TRANSFERRING, COMPETED, FAILED), bytes transferred (so far, in bytes), size (file size in bytes), started, ended, server time (in epoch), failure cause (on error).
POST
Starts a new data transfer task, which copies or moves the source file or directory (X-URI) from one storage to another, target storage. Request entiry body specifies the target in JSON format with keys: target (URI, file or directory, required), credentials (for target storage, in JSON format, e.g. {Type:UserPass, UserID:username, UserPass:password}), move (optional, boolean, default: false), overwrite (optional, boolean, default: false) (specified in the entiry body)
DELETE
Aborts a currently active transfer (path parameter).
/protocols (resource)
GET
Returns the list of storage protocols that Data Avenue supports in the response body as a JSON array.
/authentications (resource)
GET
Returns the list of authentication types supported by the specified storage protocol (path parameter).
GET
Returns the list of operations implemented by Data Avenue over the specified storage protocol (path parameter).
GET
Returns Data Avenue version (in plain text).