Data Discovery/Server Discovery Service Definition

In order to find a solution for the visualisation of big data files at the SRC Net data lake without the latency of moving data from different repositories before starting the parsing the following approach is proposed:

• Visualisation tool servers will be deployed in all the SRCs with big data files to be visualised (e.g. big data cubes) (ideally, containerised versions)
• A REST (or similar) service will be used to discover the server that is close to a particular data entity 
• The visualisation tool will connect to this selected remote server to start the visualisation

This could be integrated into an ObsCore response in-line with the VLKB Obscore service definition

For every record (observation or file level), the service invocation would provide one entry with the location of the server to be contacted to parse the remote file or if there are two or more copies of the data across the network for high availability, an equivalent number of entries with primary and secondary locations. In case multiple copies are found with a similar quality of service on the storage repository I/O rate, the IP used to invoke the service could find out the closer server location but this is an enhancement not really needed for a proof of concept (a possible idea for the science data challenge)

Notes:

• Tools should send a user-agent into the HTTP/HTTPS invocation, so this could be used to personalise the response depending on the tool, pointing to its relevant server locations and adding any needed metadata to allow the connection (e.g. server ports or any other server signature parameters)
• The internal hierarchy of this service could be inspired by the IVOA Data Link standard https://www.ivoa.net/documents/DataLink/
  • Data Link client is implemented in python inside pyVO which could simplify the parsing operations by reusing this library
  • https://pyvo.readthedocs.io/en/latest/dal/index.html#datalink
    *
• That also implies that a list of SRC->servers should be present in a queryable database or similar. Also, not really needed for the proof of concept

This feature only covers the definition of the service but not the implementation itself