Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3748

Scaled up workflow

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Could have
    • PI20
    • None
    • SRCnet
    • Hide

      SRCs will provide pre-built scripts for standard workflows that users expect to run. We aim to implement some of these, and test their effectiveness with regards to hardware limitations and data storage schemes. 

      This looks at taking an existing workflow and scaling it up to run on a significantly larger dataset. This will test both the workflow and also the compute infrastructure required. It will become important in the future when performing batch-testing and profiling across the tasks. 

      Show
      SRCs will provide pre-built scripts for standard workflows that users expect to run. We aim to implement some of these, and test their effectiveness with regards to hardware limitations and data storage schemes.  This looks at taking an existing workflow and scaling it up to run on a significantly larger dataset. This will test both the workflow and also the compute infrastructure required. It will become important in the future when performing batch-testing and profiling across the tasks. 
    • Hide

      AC1: Make a sub-folder in at least one Gitlab task with a example code of how it can be run with a significantly larger dataset on a cloud compute site. 

      AC2: Record a demo of how this can be run.

      Show
      AC1: Make a sub-folder in at least one Gitlab task with a example code of how it can be run with a significantly larger dataset on a cloud compute site.  AC2: Record a demo of how this can be run.
    • 0.5
    • 0.5
    • 0
    • Overdue

    • PI23 - UNCOVERED

    • PI20 PI20-PB

    Description

      Existing workflows are mainly aimed at small data sets that run quickly and efficiently. This feature will look at running an existing workflow with a significantly larger dataset. This will help to test out compute infrastructure using a more intense workflow. There are 3 obvious choices:

      Source-finding: For this task, it could be extended to download the entire LOFAR survey and run source-finding on every image.

      Mosaicking: It could be extended to download the entire survey (individual fields, not mosaics) and produce mosaics of a defined size (e..g 10x10 degrees).

      Image-cutouts: It could be extended to produce cut-outs of thousands of sources from the LOFAR survey.

       

      Attachments

        Structure

          Activity

            People

              r.bolton Bolton, Rosie
              A.Clarke Clarke, Alex
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Feature Progress

                Story Point Burn-up: (0%)

                Feature Estimate: 0.5

                IssuesStory Points
                To Do00.0
                In Progress   00.0
                Complete00.0
                Total00.0

                Dates

                  Created:
                  Updated:

                  Structure Helper Panel