Introduction

Data Avenue services are accessible via direct web service calls as well. The WSDL of the Data Avenue web service can be found here: https://data-avenue.eu/blacktop2/ws/dataavenue?wsdl. If using Java, it is probably more convenient to use the Java API provided as a single jar file data-avenue.core.wsclient-2.0.2.jar or use it as a maven dependency (see later).

After creating a web service client, you can already use functions that Data Avenue provides by invoking operations such as copy, list, mkdir, delete, etc. In order to use the Data Avenue API first you need to request for a "ticket" (also known as API key) by filling out the ticket request form.

The code of a directory listing below illustrates how simple is to use Data Avenue web services:

 
import hu.sztaki.lpds.dataavenue.ws.*;

public class List {
	public static void main(String[] args) throws Exception {
		String wsdlURL = "https://data-avenue.eu/blacktop2/ws/dataavenue?wsdl"; // WSDL URL
		DataAvenueService ws = DataAvenueClient.getUnsecureClient(wsdlURL); // create client
		String ticket = "YOURTICKET"; // insert your ticket here
		String directoryURI = "http://xenia.sote.hu/ftp/mirrors/www.apache.org/tomcat/";
		for (DirEntry item : ws.list(ticket, directoryURI, null))
			System.out.println(item.getName()); // print directory entry name
	}
}

Running the application:

$ java -cp .:data-avenue.core.wsclient-2.0.2.jar List
maven-plugin
taglibs
tomcat-6
tomcat-7
tomcat-8
tomcat-connectors
 
 

Creating a Client

Typically, the web service client is simply created by invoking DataBridgeClient.getClient(wsdlUrl), where wsdlUrl is the URL of Data Avenue web services' WSDL file.

NOTE: Data Avenue server uses self-signed certificate. The lines below are no longer needed, but ensure that you use "unsecure client" (DataBridgeClient.getUnsecureClient(wsdlUrl)) and the latest client lib of version 2.0.2

As Data Avenue is accessible over secure HTTP, first, you have to register the certificate of RapidSSL CA as trusted (if not yet done) by downloading its certificate (https://knowledge.rapidssl.com/library/VERISIGN/ALL_OTHER/RapidSSL%20Intermediate/RapidSSL_CA_bundle.pem) and importing it into your Java's local certificates repository:

$ keytool -importcert -trustcacerts -alias rapidsslcabundle -file RapidSSL_CA_bundle.pem -keystore
 <JAVA_HOME>/jre/lib/security/cacerts -storepass changeit

 

NOTE: If using Java6, in spite of the above certificate import, host key verification must be turned off, which is done if creating client via DataBridgeClient.getUsecureClient(wsdlUrl). It is due to the missing SNI functionality in Java6, but the communication will still be secured.

To test connectivity use the command:

$ java -cp data-avenue.core.wsclient.jar hu.sztaki.lpds.dataavenue.ws.DataAvenueClient <ticket> https://data-avenue.eu/blacktop2/ws/dataavenue?wsdl

 

Uniform Resource Identifiers (URIs)

All entities that Data Avenue can handle are identified by uniform resource identifiers (URIs) of the form: protocol://host[:port]/path, where path contains (ends with) either a file or a directory name. In the case of directories, paths should end with a slash symbol: '/'. If omitted, port is the protocol default (e.g., defaults to 80 in the case of HTTP). Available protocols can be obtained dynamically by invoking client's getSuppotedProtocols method (see later), which are: http, https, sftp, gsiftp, srm and s3, at the time of writing this document.

Credentials

Accessing different storage resources may require authentication data, called credentials, which must be provided for Data Avenue, which will behaves as a mediator. Depending on the selected protocol, you can choose among different authentication types, which require different authentication data, called credential attributes to be specified. For example, a site may require HTTP Basic Authentication, so you choose "UserPass" authentication type, and set "UserID" and "UserPass" attributes. The list of protocols, authentication types and credential attributes are available here. All credential attributes must be specified as strings, so credential data available in files locally (e.g., x509up proxy file) must be read up and set as string in credentials.

We use "Globus" authentication to create a new directory in a storage element accessible via gsiftp in the example below:

import hu.sztaki.lpds.dataavenue.ws.*;
import java.io.*;
import java.util.*;

public class Mkdir {
	public static void main(String[] args) throws Exception {
		String wsdlURL = "https://data-avenue.eu/blacktop2/ws/dataavenue?wsdl"; // WSDL URL
		DataAvenueService ws = DataAvenueClient.getUnsecureClient(wsdlURL); // create ws client
		String ticket = "YOURTICKET"; // insert yours here
		CredentialAttributes credentials = new CredentialAttributes(); // create credentials object
		credentials.setType("Globus"); // set authentication type to "Globus"
		credentials.setUserProxy(contentOf("./x509up")); // set x509up file content as UserProxy
		String newDirectory = "gsiftp://gsiftp.host.eu:2811/tmp/test/"; // create new subdir 'test' in '/tmp'
		ws.mkdir(ticket, newDirectory, credentials); // create the new directory
	}

	private static String contentOf(String fileName)
			throws FileNotFoundException {
		Scanner scanner = new Scanner(new File(fileName));
		scanner.useDelimiter("\\Z");
		StringBuilder sb = new StringBuilder();
		while (scanner.hasNext())
			sb.append(scanner.next());
		return sb.toString();
	}
}

Sessions

At accessing a specific storage, tickets and credentials data need to be passedat the very first call only, which are then maintained in your web service "session" by Data Avenue. This way you may omit these parameters in subsequent calls (null values can be passed) as long as your session is alive. After a specific time (5 minutes) of inactivity, your session will expire (you get SessionExpiredException), when you have to set ticket and credentials data again.

NOTE: There are two exceptions when credentials have to be passed: copy/move operation and alias creation. Copy tasks create new, individual session as well as aliases (download/upload), since their lifetime may exceed user session (may take several hours to complete).

Operations

There are a number of functions, operations provided by Data Avenue, all available as client methods for API users. These functions consist of four groups: meta-information retrieval, synchronous- and asynchronous operations (commands), and "alias-related" functions.

Meta-information retrieval

These methods return meta-information about Data Avenue protocols, possible authentication types, and supported operations (per protocol).

  • List<String> getSupportedProtocols(String ticket);

Returns the list of protocols that Data Avenue can handle. Using a protocol prefix in a URI other than provided by the returned list, you will get NotSupportedProtocolException.

  • List<String> getSupportedAuthenticationTypes(String ticket, String protocol);

Returns the list of supported authentication types that can be used for the given protocol. (Using an unsupported authentication type for a given protocol, you will get CredentialException.)

  • List<OperationTypesEnum> getSupportedOperations(String ticket, String protocol);

Returns the list of operations supported by Data Avenue for the given protocol. This is a subset of possible operations: LIST, MKDIR, RMDIR, DELETE, and RENAME. These operations a detailed below.

NOTE: Not all protocols necessarily offer all the possible operations (e.g., it is not possible to create directory on an HTTP site, and only LIST is available for HTTP.)

Synchronous commands

Except copy and move operations, these commands are called synchronous, as they are done at the time of client method invocation (when the control returns).

  • List<DirEntry> list(String ticket, String uri, CredentialAttributes creds);

Returns the list of directory contents (file and subdirectory entries). Details about files and subdirectories (such as type, last modification date, size) can be queried using the appropriate DirEntry methods.

  • void mkdir(String ticket, String uri, CredentialAttributes creds);

Creates a new directory at the specified location.

  • void rmdir(String ticket, String uri, CredentialAttributes creds);

Deletes the specified directory.

  • void delete(String ticket, String uri, CredentialAttributes creds);

Deletes the specified file.

  • void rename(String ticket, String uri, String newName, CredentialAttributes creds);

Renames the specified file or directory to the given new name.

Asynchronous commands

These are called asynchronous operations, as these tasks, data transfers may take a longer time to complete (possibly hours or days), so these methods calls returns transfer identifiers ("handles") associated with background tasks initiated by the call. These ids can be used to poll transfer state or cancel the task later.

  • String copy(String ticket, String srcUri, CredentialAttributes srcCreds, String destUriString, CredentialAttributes destCreds, boolean overwrite);

Copies the specified source file to the destination location. The method returns the identifier of the transfer, which can be used in getState or cancel operations (see below). overwrite flag indicates to replace the previous file with the same name at the destination location. destUri may be a directory; in this case, the name of the source file will be used as file name on the destination location. Note source- (srcCreds) and destination credentials (destCreds) are specified to authenticate on different source and destination resources, respectively. 

  • String move(String ticket, String srcUri, CredentialAttributes srcCreds, String destUri, CredentialAttributes destCreds, boolean overwrite);

Moves the specified source file to the destination location (source will be deleted after copy). The method returns the identifier of the transfer (task).

 

The following methods are not "asynchronous", but they are related to managing asynchronous tasks.

  • TransferDetails getState(String ticket, String id);

Returns state details about the transfer having the specified identifier. TransferDetails record returned contains status (CREATED, SCHEDULED, TRANSFERRING, DONE, FAILED, CANCELED), time (created, started, ended), size and progress information (size, bytes transferred), failure message (if the task has failed for some reason) information.

  • void cancel (String ticket, String id);

Stops an active transfer, and set its state to CANCELED.

 

Aliases

The aliasing-technique implemented by Data Avenue provides HTTP tunneling possibility to access (read or write) files on different storage elements. When creating an "alias", you specify the file on a storage –accessible with a specific protocol – along with the appropriate credentials, and you will get an ID back, from which an HTTP URL can be constructed, available on the blacktop: https://data-avenue.eu/blacktop2/<ID>. Reading this URL (by HTTP GET), or writing this URL (by HTTP PUT or POST), respectively, will read or write the resource, mediated by Data Avenue – without the need of the client (wget, curl, HTTP form) to be aware of the specific protocol.

NOTE: you may use aliases for a longer period of time (days, weeks). Therefore it is very important to provide credentials that are still valid at the time of their latest potential use (creation time + lifetime)!

  • String createAlias(String ticket, String uriString, CredentialAttributes creds, boolean read, int lifetime, boolean archive);

Creates an HTTP alias for the specified resource. read flag specifies whether this file will be read (HTTP GET) or written (HTTP PUT/POST). The method returns an ID; appended to blacktop URL gives the full HTTP URL, i.e. the alias of the resource. The alias will be accessible for lifetime seconds. (archive option is reserved for future development, use false value.)

The example below creates an HTTP alias for a file "myfile" accessible on an SFTP server and then downloads its content into a local file "downloaded":

import java.io.*;
import java.net.*;
import hu.sztaki.lpds.dataavenue.ws.*;

public class Download {
	public static void main(String[] args) throws Exception {
		String wsdlURL = "https://data-avenue.eu/blacktop2/ws/dataavenue?wsdl"; // WSDL URL
		DataAvenueService ws = DataAvenueClient.getUnsecureClient(wsdlURL); // create ws client
		String ticket = "YOURTICKET"; // insert yours here
		CredentialAttributes credentials = new CredentialAttributes(); // create credentials object
		credentials.setType("UserPass"); // set authentication type to "UserPass"
		credentials.setUserID("username"); // username
		credentials.setUserPass("password"); // password
		String file = "sftp://sftp.host.eu/tmp/myfile";
		String id = ws.createAlias(ticket, file, credentials, true, 60, false); // create alias for reading for 1 min
		saveHTTPUrlContent("https://data-avenue.eu/blacktop2/" + id, "downloaded");
	}

	private static void saveHTTPUrlContent(String url, String file) throws Exception {
		URLConnection urlConnection = new URL(url).openConnection();
		InputStream in = urlConnection.getInputStream();
		OutputStream out = new FileOutputStream(file);
		int data;
		while ((data = in.read()) > 0) out.write(data);
		in.close();
		out.close();
	}
}

Maven

To use Data Avenue web services in your Java application use the following repository and dependecy entries shown below:

<repositories>
  ...
  <repository>
    <id>sztaki</id>
    <name>MTA SZTAKI LPDS Maven repository</name>
    <url>https://guse.sztaki.hu/nexus/content/repositories/releases/</url> 
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
  </repository>
  ...
</repositories>
 
<dependencies>
  ...
  <dependency>
    <groupId>hu.sztaki.lpds.dataavenue.core</groupId>
    <artifactId>data-avenue.core.wsclient</artifactId>
    <version>2.0.2</version>
  </dependency>
  ...
</dependencies>