Support hierarchical data with scientific metadata in Rucio

Rucio does not natively support hierarchical data (that is to say, there is no single logical data type that can be used to encapsulate a nested directory of files and subdirectories). As radio astronomy outputs are often best represented in some hierarchical format e.g. visibilities and measurement sets, this means there is a gap in the data management functionality.

As part of previous work, an extended client for Rucio was written to explore how hierarchical data could be managed using two approaches, one by a mixture of logical data types and the other using metadata to retain the structural information of the data. This client was capable of uploading and downloading hierarchical data from the datalake using both methods, though the former was found to have limitations which would have to be addressed in Rucio's upstream (https://github.com/rucio/rucio/issues/6049) for it to be feasible.

This client does not currently have any support for setting metadata other than structural metadata. For data to subsequently appear in standardised IVOA (International Virtual Observatory Alliance) discovery services, e.g. here, work would have to be done to add functionality to the client to parse/attach the necessary ObsCore (IVOA standard) metadata fields to the corresponding data identifiers. This is the work that this feature is proposing.

AC1: Make a proposal for changes to the extended client to support upload of hierarchical data with ObsCore metadata and discuss with Magenta.

AC2: Amend the client with these changes.

AC3: Demo an actual upload of hierarchical data with ObsCore metadata and show it as discoverable it via IVOA SCS (simple cone search).