IVOA compliant metadata ingestion with Rucio

A fundamental requirement for SRCs will be the ability to expose externally accessible search and retrieve interfaces to data ingested into the distributed data management (DDM) ecosystem. For this purpose, there already exists a standardised set of astronomical interfaces and protocols under the umbrella of the International Virtual Observatory Alliance (IVOA). As such, any DDM solution being considered should ideally be capable of supplying metadata conforming to standards set out by the IVOA.

Through development recently undertaken by Cyan team, Rucio is now capable of storing metadata in standalone, external resources via plugins for e.g. postgresql, mongo. This functionality should enable a service to be deployed that formats and serves this metadata in a way that is compatible with IVOA standards.

As part of PI15 I&P, an MVP has been built that demonstrates (with toy services and mock data) synchronisation of an external Rucio metadata store with an IVOA ObsCore table, served by an externally accessible IVOA TAP service. The next step is to deploy this into a "production" environment.

This will require:

• Upgrading the existing SKAO Rucio prototype to >=1.29.2 so that the external postgres plugin is available,
• Deploying a separate, standalone postgres database accessible to both the Rucio prototype infrastructure and the infrastructure hosting the TAP service, and
• Creating mock data and ingesting it into the Rucio prototype

 Follow up work may include refactoring parts of the architecture after discussion with Dave Morris.