Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3492

OpenCADC Multi-site Science Data Discovery and Access

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      Access to scientific metadata is crucial for scientific work. In this feature, we propose to continue the work done in the previous IP18 and to offer authorized users the capability to query the ObsCore table using tools such as TOPCAT or astroquery, granting access to valuable metadata of datasets stored within the distributed data system, leveraging the SI infrastructure. Furthermore, authorized users are also able to query the storage inventory and retrieve files utilizing astroquery. In addition to authorized access, the system demonstrates its commitment to openness and inclusivity by providing support for anonymous access to public data, allowing seamless exploration and retrieval of datasets without the need for user authentication.

      Show
      Access to scientific metadata is crucial for scientific work. In this feature, we propose to continue the work done in the previous IP18 and to offer authorized users the capability to query the ObsCore table using tools such as TOPCAT or astroquery, granting access to valuable metadata of datasets stored within the distributed data system, leveraging the SI infrastructure. Furthermore, authorized users are also able to query the storage inventory and retrieve files utilizing astroquery. In addition to authorized access, the system demonstrates its commitment to openness and inclusivity by providing support for anonymous access to public data, allowing seamless exploration and retrieval of datasets without the need for user authentication.
    • Hide

      AC1: Demonstration that a user can query metadata. A demo showing a user querying the ObsCore table using TOPCAT. Alternatively a python notebook using astroquery to query the metadata  of datasets stored in the distributed data system based on SI. The retrieved metadata should include essential information such as dataset descriptions, observation parameters, and data availability details.

      AC2: Demonstration that an authorised user can access private data and an anonymous user can access public data. Based on the output of query demonstrated in AC1, the system should accurately retrieve and provide access to the requested files. The retrieval process should be efficient and reliable, ensuring that the authorized user can seamlessly access and download the files they require from the storage inventory.

      Show
      AC1: Demonstration that a user can query metadata. A demo showing a user querying the ObsCore table using TOPCAT. Alternatively a python notebook using astroquery to query the metadata  of datasets stored in the distributed data system based on SI. The retrieved metadata should include essential information such as dataset descriptions, observation parameters, and data availability details. AC2: Demonstration that an authorised user can access private data and an anonymous user can access public data. Based on the output of query demonstrated in AC1, the system should accurately retrieve and provide access to the requested files. The retrieval process should be efficient and reliable, ensuring that the authorized user can seamlessly access and download the files they require from the storage inventory.
    • 2
    • 2
    • 0
    • Team_CORAL, Team_RED
    • Sprint 5
    • Show
      Presentation: https://docs.google.com/presentation/d/1WJAsGfZbB4-rIZ12bubf1SWgFOLViUwTFYIT_VLMOpI/edit?usp=sharing Demo: -  https://skatelescope.zoom.us/rec/share/wUt8XpI1vx4tzFWHYU_CakXV2r3tJmjXz2DSntkGmryzcZ474OnZyj_N1cUA65m-.yNrrP_xxvqk8pzPN   Passcode: B&w.Hm5D
    • 20.6
    • Stories Completed, Demonstrated, Accepted by FO
    • PI24 - UNCOVERED

    • PI19-PB SRC-DM SRC-DataAccess SRC-MiniNode SRC-Multi-Team Team_CORAL Team_RED

    Description

      During PI18 we (Coral team with the support from Red/CADC team) deployed a OpenCADC platform with 3 Storage Inventory sites ( ESP, SWE, CH) and a global site. At the time of writing those sites are being populated with several data sets according to the data distribution described here: https://confluence.skatelescope.org/display/SRCSC/COR-221+Multi-site+Storage+Inventory%3A+Data+distribution+plan

      This distribution plan defines 6 namespaces. Some of them will be configured as public, and others as non-public, which will be  only accessed by users belonging to "prototyping-groups/mini-src/data-ops" group (group registration via  https://ska-iam.stfc.ac.uk/login ).

      This feature will gather the activities to finish the data distribution and to ensure the correct metadata harvesting, in order to enable 2 use cases:

      • A user can query the metadata system ( TAP, CAOM and ObsCore data models, and DataLink service with links to files in the Storage inventory) through tools like Topcat or Astroquery
      • A user can query and access files stored in the Storage Inventory by using the access url that is returned. The access will take into account if the user has permissions to access the files. Both anonymous and non-anonymous access will be possible.

      This feature is related to https://jira.skatelescope.org/browse/SP-3302

      Attachments

        Issue Links

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                M.Parra Parra, Manuel
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete2937.5
                  Total2937.5

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel