Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4605

Development of a scalable, multi-container processing pipeline in CADC CANFAR

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide
      • The current examples of using CANFAR show the deployment of pipelines within a single container
      • In this work we will demonstrate the deployment of astronomy data post-processing pipelines that coordinate the execution of separate containers on a CANFAR deployment
      • This will allow for scaling your compute resources at different stages of a workflow, which leads to more efficient resource usage
      • In this work we want to specifically:
        • Mount image cubes (.fits) from Rucio onto CANFAR containers
        • Run the Miriad imaging software in a container to generate a reduced image cube
        • Run a parallel source finding algorithm (sofia) on the image cube (embarrassingly parallel problem, run multiple containers of sofia on different sub-cubes)
        • Write the output product files back to Rucio with the appropriate metadata so it is discoverable via TAP
      • Read and write data products between Rucio and CANFAR containers in individual steps of a workflow
      Show
      The current examples of using CANFAR show the deployment of pipelines within a single container In this work we will demonstrate the deployment of astronomy data post-processing pipelines that coordinate the execution of separate containers on a CANFAR deployment This will allow for scaling your compute resources at different stages of a workflow, which leads to more efficient resource usage In this work we want to specifically: Mount image cubes (.fits) from Rucio onto CANFAR containers Run the Miriad imaging software in a container to generate a reduced image cube Run a parallel source finding algorithm (sofia) on the image cube (embarrassingly parallel problem, run multiple containers of sofia on different sub-cubes) Write the output product files back to Rucio with the appropriate metadata so it is discoverable via TAP Read and write data products between Rucio and CANFAR containers in individual steps of a workflow
    • Hide

      AC1: Upload of astronomy data products to the rucio datalake , volumes mounted from CASRC Rucio storage element onto a processing node on CANFAR 

      AC2: Upload of custom-build images to the CANFAR science platform so they can be used for batch processing

      AC3: Write a multi-stage data processing workflow that orchestrates CANFAR compute resources

      AC4: Demo and documentation so that the workflow can be re-run by other members of the SRCNet

      Show
      AC1: Upload of astronomy data products to the rucio datalake , volumes mounted from CASRC Rucio storage element onto a processing node on CANFAR  AC2: Upload of custom-build images to the CANFAR science platform so they can be used for batch processing AC3: Write a multi-stage data processing workflow that orchestrates CANFAR compute resources AC4: Demo and documentation so that the workflow can be re-run by other members of the SRCNet
    • 1
    • 1
    • 0
    • Team_LAVENDER
    • Sprint 5

    • PI24 - UNCOVERED

    • PI24-PB SRCNet0.x

    Description

      Working with CanSRC (Red team) to develop a data processing pipeline that uses the Rucio datalake and CADC CANFAR deployment for data processing. We authenticate to the CADC CANFAR platform with a OIDC provider. This will add to the test campaigns for the SRCNet, and demonstrate distributed data transfer and processing.

      AC1: Upload of astronomy data products to the rucio datalake , volumes mounted from CASRC Rucio storage element onto a processing node on CANFAR 

      AC2: Upload of custom-build images to the CANFAR science platform so they can be used for batch processing

      AC3: Write a multi-stage data processing workflow that orchestrates CANFAR compute resources

      AC4: Demo and documentation so that the workflow can be re-run by other members of the SRCNet

       

      Attachments

        Issue Links

          Child Of

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-4870 Science Enabling v0.1 - Roadmap

          • Not Assigned - Priority is not assigned yet.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4676 HPC-based Science Platform

          • Not Assigned - Priority is not assigned yet.
          • Funnel
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4640 Demonstrate job submission via the orchestration layer

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Program Backlog

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3954 Gather APIs of existing federated execution tools

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done
          mentioned in

          Page Loading...

          Structure

            Activity

              People

                j.collinson Collinson, James
                A.Shen Shen, Austin
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 1.0

                  IssuesStory Points
                  To Do22.0
                  In Progress   37.0
                  Complete00.0
                  Total59.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel