Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4649

Expand spectral line pipeline capability and test scaling

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • High
    • PI24
    • COM SDP SW
    • Data Processing
    • Hide

      Highly parallelised spectral line imaging is required for processing spectral line data which will eventually require efficient per channel wide-field imaging with continuum subtraction. Whilst the scaling of this pipeline should be straightforward, it will account for a significant portion of the overall processing time. Development of this pipeline now will enable early understanding of necessary functionality, overall performance and maximise time for further development to meet our performance needs.

      Show
      Highly parallelised spectral line imaging is required for processing spectral line data which will eventually require efficient per channel wide-field imaging with continuum subtraction. Whilst the scaling of this pipeline should be straightforward, it will account for a significant portion of the overall processing time. Development of this pipeline now will enable early understanding of necessary functionality, overall performance and maximise time for further development to meet our performance needs.
    • Hide
      1. Add additional functionality as outlined in the diagram (scope to be agreed during planning discussions)
        1. An updated release of the pipeline
      2. Benchmark tests with data for both Low and Mid that test performance and scalability of current functionality
        1. Confluence page reports
      3. Discussion with relevant stakeholders to illicit feedback for future additions/improvements (Science Operations/ Commissioning Scientists)

       

      Inputs:

      • (Final DD calibration solutions from self-calibration)
      • Averaged/calibrated visibilities
      • Final Local sky model (produced post-continuum imaging pipeline)

      Outputs:

      • Spectral line cube Data Product with residual image cube
      • Line plots
      Show
      Add additional functionality as outlined in the diagram (scope to be agreed during planning discussions) An updated release of the pipeline Benchmark tests with data for both Low and Mid that test performance and scalability of current functionality Confluence page reports Discussion with relevant stakeholders to illicit feedback for future additions/improvements (Science Operations/ Commissioning Scientists)   Inputs: (Final DD calibration solutions from self-calibration) Averaged/calibrated visibilities Final Local sky model (produced post-continuum imaging pipeline) Outputs: Spectral line cube Data Product with residual image cube Line plots
    • 13
    • 13
    • 0
    • Team_DHRUVA
    • Sprint 5

    Description

      Following first release(s) of the spectral line pipeline in PI23, this work should add additional functionality that is currently missing, potentially including deconvolution, image-based subtraction, improvements to the sky model representation and use of an updated gridder. The final scope should be agreed with the FO.

      The work should also include further tests of the pipeline that include image sizes sufficiently large that they do not fit in memory to more fully test the distribution and performance for both Mid and Low use case tests.

       

      Links / References

      1. Current repository: https://gitlab.com/ska-telescope/sdp/science-pipeline-workflows/ska-sdp-spectral-line-imaging
      2. Current documentation: https://developer.skao.int/projects/ska-sdp-spectral-line-imaging/en/latest/?badge=latest

       

       

      See frame on DP ART board: https://miro.com/app/board/uXjVK6Lrdw4=/?moveToWidget=3458764597808238287&cot=14 

      Attachments

        Issue Links

          Structure

            Activity

              People

                D.Fenech Fenech, Danielle
                f.graser Graser, Ferdl
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (6.52%)

                  Feature Estimate: 13.0

                  IssuesStory Points
                  To Do1030.0
                  In Progress   413.0
                  Complete13.0
                  Total1546.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel