Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3302

Multi-site Observation Metadata Deployment: location agnostic data query with data access

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      This is a step further from SP-3297. If in SP-3297 the objective is to demonstrate the interoperability between the A&A and the distributed data systems, here we will demonstrate how to do queries based on scientific metadata as search criteria and how this can be done by both authorised and anonymous users. In this way we are adding a new layer for the mini-SRCNet concept. 

      Show
      This is a step further from SP-3297 . If in SP-3297 the objective is to demonstrate the interoperability between the A&A and the distributed data systems, here we will demonstrate how to do queries based on scientific metadata as search criteria and how this can be done by both authorised and anonymous users. In this way we are adding a new layer for the mini-SRCNet concept. 
    • Hide

      AC1: Documentation and Demo on how to access to the SI metadata for an authorised user can query ObsCore by using TOPCAT/astroquery.
      AC2: Documentation and Demo on how to access to the SI data for an authorised user can query files with astroquery.

      Show
      AC1: Documentation and Demo on how to access to the SI metadata for an authorised user can query ObsCore by using TOPCAT/astroquery. AC2: Documentation and Demo on how to access to the SI data for an authorised user can query files with astroquery.
    • 2
    • 0
    • Team_CORAL
    • Sprint 5
    • 20.1
    • PI24 - UNCOVERED

    • PI18-PB SRC-DataAccess SRC-SI
    • SRC-G2

    Description

      Develop a demonstrator of 3 use cases:

      a) An authorised user can query on ObsCore table with TOPCAT or astroquery, in order to access metadata of datasets stored in the distributed data system based on SI.

      b) An authorised user can query and access files from the storage inventory with astroquery.

      c) show support for anonymous access to public data.

      Basic concept: there is a primary metadata database with all the observation metadata. Mini-SRC prototype sites will make a complete local copy of the metadata. Unlike storage inventory, this is single-direction sync from a primary to clones.

      Softare dev: add grants endpoint to metadata repository service and provide a stand alone grants service for mini-src nodes.

      Deploy primary metadata site: metadata database (PostgreSQL) and metadata repository service; add to global service registry; populate SI science metadata database identified in SP-2840 but with storage identifiers changed to refer to the SRCNet global inventory in SP-2829.

      Software dev: the CAOM metadata sync tool is a command line tool used at CADC and ESAC; need to package it up in a docker image for easier config and deployment.

      Software dev: add a StorageResolver for the `ska` artifact scheme; this is used by the CAOM DataLink service to generate links for accessing data in storage.

      Mini-SRC metadata sites: deploy metadata database and metadata synchcronisation from primary; deploy CAOM services: TAP, DataLink, grant service (readGrantProvider for use by local minoc)

       

      This prototype involves four CAOM software components; CADC will provide docker images for all components: repo service, sync tool, TAP service, DataLink service.

       

      Mini-SRCNet Demonstrator Roadmap: https://docs.google.com/spreadsheets/d/1Kri0Bc0n9PE30Nv7b1LkeUhh88ZatuFuYYJ2T2CSRUU/edit?usp=sharing

      Attachments

        Issue Links

          Structure

            Activity

              People

                M.Parra Parra, Manuel
                M.Parra Parra, Manuel
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel