REST Resources

Data Avenue REST interface is accessible at URL https://data-avenue.eu/blacktop-dev/rest, followed by one of the resources listed below:

  • /directory - for directory (folder, bucket, container) lising, creation, deletion

  • /file - for file (object) upload, download, deletion

  • /attributes - for getting/setting file, directory attributes (permissions, size, date of last modification, etc.)

  • /aliases - for creating/deleting aliases for remote files (accessed via HTTP tunneling)

  • /transfers - for starting/aborting/getting status of inter-storage data tranfers

  • /protocols - for getting the set of supported storage protocols

  • /authentication - for getting possible authentication mechanisms for storage protocols

  • /operations - for getting possible operation types implemented in Data Avenue

  • /version - for getting Data Avenue version

REST interface can be used with command line tools as well such as curlFor example:

curl -k https://data-avenue.eu/blacktop-dev/rest/version

returns Data Avenue version (-k is to ignore certicate verification).

Data Avenue Authentication (Key)

Each REST call must contain a special header field, named "X-Key", containing the access key (a.k.a. "ticket") required to use Data Avenue services. It can be requested on page: ticket request form.

You can use curl's -H (--header) option set the key for the request:

 curl -H "X-Key: de305d54-75b4-431b-adb2-eb6b9e546014" ...

To avoid saving your key into command history, you can create a file key with a single line "X-Key: de305d54-75b4-431b-adb2-eb6b9e546014", and use cat in curl:

curl -H "$(cat key)" ...
X-Key header will be denoted as <key> for short in the following.

(Storage) Credentials

In order to allow Data Avenue to access the storage of your choice, you have to temporarily delegate your credentials for successful mediation. Such credentials are passed via HTTP header elements. The most common authentication type is the username-password authentication (access key/secret key pair, in the case of S3), but you can also authenticate using x509 proxy (e.g., GridFTP). In the latter case, proxy need to be base64 encoded in the header. These credentials have to put into headers named "X-Username", "X-Password", "X-Proxy" as appropriate.

To pass username and password with curl, use -H option:

curl -H "X-Username: username" -H "X-Password: password" ...

To add a proxy, stored in file x509up,  to HTTP header you can use the base64 encoder inline:

curl -H "X-Proxy: $(base64 --wrap=0 x509up)" ...
Note that credentials passed over non-secured HTTP can be eavesdropped. Use secured Data Avenue endpoints (starting with URL https://), and avoid using curl's -k (--insecure) option, unless you are absolutely sure in that the network link between you and the Data Avenue server is protected.
Storage credential headers will be denoted as <credentials> for short in the following.

URIs

 Data Avenue identifies remote files residing on storage resources using URIs (Uniform Resource Identifiers) of the form: protocol://storage-address/path/[file]. (Directory URIs end with /.)

Examples of URIs are:

sftp://192.168.152.46/tmp/mydir/
s3://s3.lpds.sztaki.hu/mys3bucket/myfile

Data Avenue REST operations (file, directory, attributes, transfers) require passing the URI of the remote file on which some operation needs to be performed. The URI is passed in "X-URI" HTTP header field.

With curl you can do this like:

curl -H "X-URI: s3://aws.amazon.com/mybucket/file2download"

/directory (resource)

GET

Returns the list of directory enty names (files and subdirectories) of the spefified remote directory (X-URI header) in the form of JSON array (application/json response).

curl <key> <credentials> -H "X-URI: sftp://192.168.152.46/tmp/" http://data-avenue.eu/blacktop/rest/directory
["file1","file2",...]

POST

Creates the specified directory (X-URI header).

curl --request POST <key> <credentials> -H "X-URI: sftp://192.168.152.46/tmp/newdir/" http://data-avenue.eu/blacktop/rest/directory

DELETE

Deletes the specified directory (X-URI header), recursively.

curl <key> <credentials> --request DELETE -H "X-URI: sftp://192.168.152.46/tmp/dir2delete/" http://data-avenue.eu/blacktop/rest/directory

/file (resource)

GET

Download file contents of the specified remote file (X-URI header) as application/octet-stream.

curl <key> <credentials> "X-URI: sftp://192.168.152.46/tmp/file2download-o downloaded http://data-avenue.eu/blacktop/rest/file

POST/PUT

Uploads a new remote file (X-URI header) to a remote storage.

curl <key> <credentials> --request POST --header "Content-Type: application/octet-stream" --data-binary @file2upload "X-URI: sftp://192.168.152.46/tmp/uploaded" http://data-avenue.eu/blacktop/rest/file
If you also add header "X-Accept-Redirects: yes" and the storage resource supports direct URLs (pre-signed URLs, in the case of Amazon S3), file up- and downloads are redirected to the storage resource bypassing Data Avenue, so data transfers can be much more efficient. Use -L (--location) option of curl to allow redirection.
Using redirects in the case of S3 requires using the PUT HTTP method. Some S3 storages also require setting empty Content-Type header or using "binary/octet-stream". It is thus recommended to use curl command's --upload-file option (--data-binary sets "application/x-www-form-urlencoded") and to omit the Content-Type header setting.

DELETE

Deletes the specified remote file (X-URI header).

curl <key> <credentials> --request DELETE "X-URI: sftp://192.168.152.46/tmp/file2delete" http://data-avenue.eu/blacktop/rest/file

/attributes (resource)

GET

Returns the attributes of the specified remote file or directory (X-URI header). The result is a JSON object with keys: name, size (in bytes, in the case of file), date of last modification (epoch in millis), permissions (in linux-like format, e.g., xrw-rw-rw, in owner, group, all users permissions order), and comment (optional).

curl <key> <credentials> "X-URI: sftp://192.168.152.46/tmp/filehttp://data-avenue.eu/blacktop/rest/attributes
{"lastModified":1444641344000,"name":"file","permissions":"rwx---r--","sizeUnit":"B","size":3649}

POST

Returns the set of attributes of the set of specified remote files or directories in the specified remote directory (X-URI header). The set of entries of interest should be given in entiry body as a JSON array. The result is a JSON array of JSON objects with keys descibed in GET. This operation is used typically together with directory listing (GET /directory) to get entry details.

curl <key> <credentials> --request POST "X-URI: sftp://192.168.152.46/tmp/-H "Content-Type: application/json" --data "[file1, file2]" http://data-avenue.eu/blacktop/rest/attributes
[{"lastModified":1403481750000,"name":"file1","permissions":"rw----r--","sizeUnit":"B","size":104857600}, {"lastModified":1403481914000,"name":"file2","permissions":"rw----r--","sizeUnit":"B","size":10485760}]

/aliases (resource)

POST

Creates an (HTTP) alias for the specified remote file (X-URI header), so the file contents of the remote file can be read/written by simply reading/writing (HTTP GET/PUT) the alias URL, which points to the Data Avenue server, via HTTP tunneling. Optionally, parameters can be specified for alias creation (in the entiry body in JSON format): lifetime (of the alias in seconds, default: 1 hour), read (read-only alias, boolean, default: true), redirect (allowed if applicable, default: true), and archive (extract archive on reading/writing, boolean,  default: false). The method returns alias details in entiry body in JSON format with keys: (alias) id, URL (on Data Avenue server), direct URL (direct storage URL, if applicable).

curl <key> <credentials> --request POST "X-URI: s3://s3.lpds.sztaki.hu/s3bucket//file2upload-H "Content-Type: application/json" --data "{lifetime:3600, read:false, redirect:true}" http://data-avenue.eu/blacktop/rest/aliases
{"id":"16a5db84-db48-4747-9a6f-7df0fe515103", "URL":"https://data-avenue.eu//blacktop/aliases/16a5db84-db48-4747-9a6f-7df0fe515103", "directURL":"https://s3.lpds.sztaki.hu/s3bucket/file2upload?Expires=1445645656&AWSAccessKeyId=username..."}

DELETE

Deletes the specified alias (path parameter) - before expiration.

curl <key> <credentials> --request DELETE http://data-avenue.eu/blacktop/rest/aliases/0a6921bf-4599-4548-8595-497ed9cd01e9

/transfers (resource)

GET

Returns the status of the specified transfer (path parameter). The response is a JSON object (entity body) with keys: source, target URIs, state (CREATED, TRANSFERRING, COMPETED, FAILED), bytes transferred (so far, in bytes), size (file size in bytes), started, ended, server time (in epoch), failure cause (on error).

curl <key> <credentials> http://data-avenue.eu/blacktop/rest/transfers/0a6921bf-4599-4548-8595-497ed9cd01e9
{"bytesTransferred":104857600,"source":"s3://s3.lpds.sztaki.hu/s3bucket/100MB.dat","status":"DONE","serverTime":1446719043792,"target":"sftp://192.168.152.46/tmp/100MB.dat","ended":1446718355963,"started":1446718343133,"size":104857600}

POST

Starts a new data transfer task, which copies or moves the source file or directory (X-URI) from one storage to another, target storage. Request entiry body specifies the target in JSON format with keys: target (URI, file or directory, required), credentials (for target storage, in JSON format, e.g. {Type:UserPass, UserID:username, UserPass:password}), move (optional, boolean, default: false), overwrite (optional, boolean, default: false)   (specified in the entiry body) 

curl <key> <credentials> --request POST "X-URI: s3://s3.lpds.sztaki.hu/s3bucket/100MB.dat"  -H "Content-ype: application/json" --data "{target:'sftp://192.168.152.46/tmp/100MB.dat', credentials:{Type:UserPass, UserID:username, UserPass:password}}" http://data-avenue.eu/blacktop/rest/transfers
{"id":"0a6921bf-4599-4548-8595-497ed9cd01e9"}

DELETE

Aborts a currently active transfer (path parameter).

curl <key> <credentials> --request DELETE http://data-avenue.eu/blacktop/rest/transfers/0a6921bf-4599-4548-8595-497ed9cd01e9

/protocols (resource)

GET

Returns the list of storage protocols that Data Avenue supports in the response body as a JSON array.

curl http://data-avenue.eu/blacktop/rest/protocols
["http","https","sftp","gsiftp","srm","irods","cassandra","s3"]

/authentications (resource)

GET

Returns the list of authentication types supported by the specified storage protocol (path parameter).

curl http://data-avenue.eu/blacktop/rest/authentication/sftp
["UserPass","SSH"]
/operations (resource)

GET

Returns the list of operations implemented by Data Avenue over the specified storage protocol (path parameter).

curl http://data-avenue.eu/blacktop/rest/operations/sftp
["LIST", "MKDIR", "RMDIR", "DELETE", "RENAME", "INPUT_STREAM", "OUTPUT_STREAM"]
/version (resource)

GET

Returns Data Avenue version (in plain text).

curl http://data-avenue.eu/blacktop/rest/version
3.0.0 (build: 04/11/2015-15:17)