Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2392

Dask self-calibration pipeline demonstrating use SDP Processing Functions

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Must have
    • PI14
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      see description

      Show
      see description
    • Hide
      • Self-cal pipeline, designed to integrate new SDP Processing functions (preferably using the Python Bindings)
      • Identify suitable test data and develop test scripts for running the pipeline and for checking and analysing outputs
      • Include in a suitable SDP GitLab repo with consideration for CI, developer documentation, etc
      Show
      Self-cal pipeline, designed to integrate new SDP Processing functions (preferably using the Python Bindings) Identify suitable test data and develop test scripts for running the pipeline and for checking and analysing outputs Include in a suitable SDP GitLab repo with consideration for CI, developer documentation, etc
    • Intra Program
    • 5
    • 5
    • 13
    • 2.6
    • Team_ORCA
    • Sprint 5
    • Hide

      We have created a prototype imaging pipeline using the processing function library. Despite the title of this feature, it is not parallelised in Dask nor does it perform self-calibration. The only function available when the pipeline was created was the direct Fourier transform (DFT). The pipeline is formally a stand-alone code, but it depends on RASCIL data models and processing functions. Since the pipeline is a serial implementation we have used small test data sets from RASCIL in testing it.

      A comparison of the output images and timings can be found in Results comparison for stand-alone imaging.

      Show
      We have created a prototype imaging pipeline using the processing function library. Despite the title of this feature, it is not parallelised in Dask nor does it perform self-calibration. The only function available when the pipeline was created was the direct Fourier transform (DFT). The pipeline is formally a stand-alone code, but it depends on RASCIL data models and processing functions. Since the pipeline is a serial implementation we have used small test data sets from RASCIL in testing it. A comparison of the output images and timings can be found in Results comparison for stand-alone imaging .
    • 18.3
    • Stories Completed, Accepted by FO
    • PI22 - UNCOVERED

    • SPO-1592 SPO-1593

    Description

      Who?

      • SDP pipeline developers, Commissioning scientists, (eventually) wider radio-astronomy community

      What?

      • The first iteration of a SKA self-cal pipeline that is composed of SKA processing functions with clean levels of abstraction between the processing functions and Execution Engine application. (As title writes, would want to have this distributed in Dask. However first first thing is to have a non-distributed but Python version using the SDP Proc Fns)
      • Consideration for scaling to multiple cores, accelerators, and eventually nodes
      • Consideration for test data in the form of simulations and precursor data for focused quick CI testing and working towards being able to process an ~AA2 size data set.

      Why?

      • Demonstrates and explores approaches to SDP pipeline development, leveraging on work done in previous PIs and in PI14 on processing functions.

      Attachments

        Issue Links

          Structure

            Activity

              People

                b.nikolic Nikolic, Bojan
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 5.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete615.0
                  Total615.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel