Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1572

Improve the Yandasoft memory footprint

Details

    • Data Processing
    • Hide

      Improved performance –
      Improved scaling –
      Improved flexibility –
      Decreased technical debt –

      Show
      Improved performance – Improved scaling – Improved flexibility – Decreased technical debt –
      1. Quantify the impact of the changes for a simulated SKA dataset
      2. Quantify any impact on performance (memory and/or runtime)
    • 2
    • 2
    • 3
    • Team_YANDA
    • Sprint 5
    • Hide

      AC#1: Slides presented at sysdemo 10.4:
      https://drive.google.com/file/d/1oRtag5BydJ4XEGhc29Kw1UrqYm6AnkkC/view?usp=sharing
      AC#2: Memory footprint was reduced by about a factor of 2; jupyter notebook, logs and a memory plot are attached to YAN-712

      Show
      AC#1: Slides presented at sysdemo 10.4: https://drive.google.com/file/d/1oRtag5BydJ4XEGhc29Kw1UrqYm6AnkkC/view?usp=sharing AC#2: Memory footprint was reduced by about a factor of 2; jupyter notebook, logs and a memory plot are attached to YAN-712
    • 10.6
    • Stories Completed, Integrated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
    • PI22 - UNCOVERED

    • Yandasoft
    • SPO-1002

    Description

      After the continuum imaging experiments it is clear that a few relatively small changes to yandasoft would be beneficial for ongoing work:

      1) Get EveryBeam into the codebase (to be done in SP-1539)
      2) Make the yandasoft WStack implementation more memory efficient. It should be possible to do this by resetting it in the snapshot imaging framework. It would allow us to retire some of the current, standalone w-stacking classes, reducing technical debt.
      3) Reduce the size of uv and image buffers. There are two front runners here: check that any buffers that duplicate data are needed (e.g. dp transforms or reformatting for a given task); and remove zero-padding for psf oversampling from all worker process buffers, adding it only after merging to the master for preconditioning and deconvolution. The latter is also an example of the former, since images are stored as both normal equation objects during the merge and params objects during deconvolution.

      This enabler will address points 2 and 3, in particular WStack modifications and removal of zero-padding from worker-node buffers.

      Attachments

        Issue Links

          Structure

            Activity

              People

                b.mort Mort, Ben
                d.mitchell Mitchell, Daniel
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1021.0
                  Total1021.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel