Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4350

Develop and realease combined ICAL pipeline

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Must have
    • PI23
    • COM SDP SW
    • Data Processing
    • Hide

      Development and release of a combined ICAL pipeline will

      • result in a configurable pipeline that can process both Low and Mid data by simply changing its configuration;
      • avoid code duplication;
      • enable expert users like domain specialists, system scientists and commissioning scientists to exercise the pipeline and provide feedback.
      Show
      Development and release of a combined ICAL pipeline will result in a configurable pipeline that can process both Low and Mid data by simply changing its configuration; avoid code duplication; enable expert users like domain specialists, system scientists and commissioning scientists to exercise the pipeline and provide feedback.
    • Hide
      1. Support functions for the Low and Mid pipelines are moved into a single repository so that they can be used by both pipelines and code-duplication is avoided.
      2. The iterative calibration (ICAL) and/or low pipeline adopt the configuration strategy of the Mid pipeline.
      3. (Stretch) The Low and Mid pipeline are merged / combined into a single Python-based pipeline that can process both Low and Mid data depending on configuration.
      4. (Stretch) This is demonstrated by reproducing the Low and Mid benchmarking runs.
      5. Test data sets are made available along with pipeline configurations to process them.
      6. A release of the pipeline(s) is made so that system scientists, domain specialists and commissioning scientists can easily install the pipeline and experiment with it.
      Show
      Support functions for the Low and Mid pipelines are moved into a single repository so that they can be used by both pipelines and code-duplication is avoided. The iterative calibration (ICAL) and/or low pipeline adopt the configuration strategy of the Mid pipeline. (Stretch) The Low and Mid pipeline are merged / combined into a single Python-based pipeline that can process both Low and Mid data depending on configuration. (Stretch) This is demonstrated by reproducing the Low and Mid benchmarking runs. Test data sets are made available along with pipeline configurations to process them. A release of the pipeline(s) is made so that system scientists, domain specialists and commissioning scientists can easily install the pipeline and experiment with it.
    • Intra Program
    • 8
    • 8
    • 0
    • Team_SCHAAP
    • Sprint 5

    • PI23 - UNCOVERED

    • AA2

    Description

      The iterative calibration (ICAL) pipelines for Low and Mid have been developed as two independent products while their overall structure is fundamentally the same. Both developed pipelines have their own strengths. The Low pipeline is feature complete as demonstrated by processing actual LOFAR data, while the Mid pipeline is more mature in terms of configuring a multi-node compute environment and providing a mechanisme to define / configure the pipeline tasks. The teams have been exchanging these ideas, but this unavoidably results in code-duplication and duplicate efforts. Given that the intrinsic structure (iterative self-calibration loop) of both pipelines is the same, the aim is to combine both pipelines into a single ICAL pipeline that can process both Low and Mid data depending on how it si configured.

      This pipeline should be released so that end-users like domain specialists, system scientists and commissioning scientists can start exercising the pipeline and provide feedback to inform further evolution / development of the ICAL pipeline.

      Attachments

        Structure

          Activity

            People

              D.Fenech Fenech, Danielle
              D.Fenech Fenech, Danielle
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Feature Progress

                Story Point Burn-up: (53.33%)

                Feature Estimate: 8.0

                IssuesStory Points
                To Do27.0
                In Progress   00.0
                Complete28.0
                Total415.0

                Dates

                  Created:
                  Updated:

                  Structure Helper Panel