Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3510

SRC workflow example: Source Finding

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      SRCs will provide pre-built scripts for standard workflows that users expect to run. We aim to implement some of these, and test their effectiveness with regards to hardware limitations and data storage schemes. 

      This particular workflow looks at source finding using the PYBDSF algorithm which we have already containerised and used in Science Data Challenge 1. This workflow could test distributing images across a storage network (Rucio), and running source-finding on each of them simultaneously. The catalogues would then be retrieved and combined in a single location for the user.

      Show
      SRCs will provide pre-built scripts for standard workflows that users expect to run. We aim to implement some of these, and test their effectiveness with regards to hardware limitations and data storage schemes.  This particular workflow looks at source finding using the PYBDSF algorithm which we have already containerised and used in Science Data Challenge 1. This workflow could test distributing images across a storage network (Rucio), and running source-finding on each of them simultaneously. The catalogues would then be retrieved and combined in a single location for the user.
    • Hide

      AC1: Run a containerised version of PYBDSF to find sources and output a catalogue.

      AC2: Make this workflow interactive via Jupyterhub.

      AC3: Document this workflow on Git.

      Bonus-AC4: Distribute survey data across the Rucio storage network

      Bonus-AC5: Run source-finding on the distributed images simultaneously.

      Bonus-AC6: Retrieve the output catalogues across the storage network to a single location, filter and combine them into a master catalogue.

      Show
      AC1: Run a containerised version of PYBDSF to find sources and output a catalogue. AC2: Make this workflow interactive via Jupyterhub. AC3: Document this workflow on Git. Bonus-AC4: Distribute survey data across the Rucio storage network Bonus-AC5: Run source-finding on the distributed images simultaneously. Bonus-AC6: Retrieve the output catalogues across the storage network to a single location, filter and combine them into a master catalogue.
    • 0.5
    • 0.5
    • 0
    • Team_MAGENTA
    • Sprint 5
    • Hide

      Code and documentation at: https://gitlab.com/ska-telescope/src/src-scientific-use-cases/-/tree/master/tasks/source-finding-PYBDSF?ref_type=heads

      Demo: https://drive.google.com/file/d/1X9w6alrxfQ1Fu3BpiRTvIqEwZnZ4Vybz/view?usp=drive_link 

      AC2: SDC1 environment is available at https://jupyterhub.srcdev.skao.int/ 

      Show
      Code and documentation at: https://gitlab.com/ska-telescope/src/src-scientific-use-cases/-/tree/master/tasks/source-finding-PYBDSF?ref_type=heads Demo: https://drive.google.com/file/d/1X9w6alrxfQ1Fu3BpiRTvIqEwZnZ4Vybz/view?usp=drive_link   AC2: SDC1 environment is available at https://jupyterhub.srcdev.skao.int/  
    • 24.2
    • Accepted by FO

    • PI24 - UNCOVERED

    • PI19-PB

    Description

      Input data: single/multiple images.

      Output data: catalogue.

      Software involved: PYBDSF (others exist).

      Workflow steps: Run PYBDSF in python with input parameter file. If multiple images in single or distributed locations, will need to write a script to combine the catalogues afterwards. 

      State of existing workflows: PYBDSF has been Containerised and is used in the SDC1 workflow. It would be minimal work to split this out as a stand alone use case for source finding on a single image. For a use case where data is distributed and catalogues are combined after, this would need a moderate amount of work to distribute the data and write a script that filters and combines the catalogues after.

      Architecture and hardware: Multithreaded CPU. Flexible RAM. Data in a single location, or if multiple images, can distribute and then combine catalogues after.

      Attachments

        Issue Links

          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3496 Consolidate naming and structure of SRC example workflow repository

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                r.bolton Bolton, Rosie
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 0.5

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete33.5
                  Total33.5

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel