Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2084

Make first set of SDP processing functions ready for release

Details

    • Enabler
    • Must have
    • PI13
    • COM SDP SW
    • None
    • True
    • Data Processing
    • Hide

      The key aspect of SDP architecture is a maintainable set of processing functions which are then used for all data processing through execution engines/workflows. This is expected to contribute to the long-term sustainability of the SKA software and to allow easier upgrades when for example characteristics of hardware change. In order to achieve this it is necessary to have a single repository to which multiple developers and eventually multiple teams can contribute the code for these functions. The first release of this repository signals the point when it is possible for multiple teams to begin contributing to it.

      Show
      The key aspect of SDP architecture is a maintainable set of processing functions which are then used for all data processing through execution engines/workflows. This is expected to contribute to the long-term sustainability of the SKA software and to allow easier upgrades when for example characteristics of hardware change. In order to achieve this it is necessary to have a single repository to which multiple developers and eventually multiple teams can contribute the code for these functions. The first release of this repository signals the point when it is possible for multiple teams to begin contributing to it.
    • Hide
      1. Ready for use, specifically for multiple teams to begin contributing to the repository
      2. Documentation, build system, Python bindings in reasonable shape (at least as advanced as in https://gitlab.com/ska-telescope/sdp/ska-sdp-func-iotest  but free to use other solutions if desired)
      Show
      Ready for use, specifically for multiple teams to begin contributing to the repository Documentation, build system, Python bindings in reasonable shape (at least as advanced as in https://gitlab.com/ska-telescope/sdp/ska-sdp-func-iotest   but free to use other solutions if desired)
    • Intra Program
    • 3
    • 3
    • 13
    • 4.333
    • Team_HIPPO
    • Sprint 5
    • Hide

      The repository for processing function library with documentation, build system, and Python bindings based on ctypes was created.

      Repository:
      https://gitlab.com/ska-telescope/sdp/ska-sdp-func

      Documentation is available here:
      https://ska-telescope.gitlab.io/sdp/ska-sdp-func

      Documentation also has a detailed description of how to add a new processing function:
      https://ska-telescope.gitlab.io/sdp/ska-sdp-func/adding_new_functions.html

      Python bindings that are based on ctypes are for example shown here:
      https://gitlab.com/ska-telescope/sdp/ska-sdp-func/-/blob/main/python/ska/sdp/func/vector.py

      Show
      The repository for processing function library with documentation, build system, and Python bindings based on ctypes was created. Repository: https://gitlab.com/ska-telescope/sdp/ska-sdp-func Documentation is available here: https://ska-telescope.gitlab.io/sdp/ska-sdp-func Documentation also has a detailed description of how to add a new processing function: https://ska-telescope.gitlab.io/sdp/ska-sdp-func/adding_new_functions.html Python bindings that are based on ctypes are for example shown here: https://gitlab.com/ska-telescope/sdp/ska-sdp-func/-/blob/main/python/ska/sdp/func/vector.py
    • 14.4
    • Stories Completed, Integrated, Outcomes Reviewed, Demonstrated
    • PI22 - UNCOVERED

    • SPO-1454

    Description

      Based on an existing solution, a processing function library release should contain a (possibly simple) set of processing functions that is

      • ready for usage (i.e. documented, with clear Python & C++ API )
      • internally well factored (i.e. following clear design and language patterns and supporting robust testing and packaging via CI)

      Such that it can be used as a template for future processing function releases. The approach should have been reviewed with relevant (developer) stakeholders.

      OLD Notes for review:

      populated with a skeleton structure for source folders, docs, and CI pipeline

      Expect to need spike to review, update, and agree on structure following <TODO link!> straw man proposal.

      (this will likely need 1FP - ~10+ people for a few hours each)

      The remaining FP is intended to include

      • Initial repo set up in gitlab under `sdp/ska-sdp-func` 
      • Initial source structure (folders) defined with readme's in each to describe what goes into each folder.
      • Initial doc skeleton (user and developer) structure set up with build steps (manual + CI pipeline)
      • Initial build system set up for C/C++ package (including use of CUDA kernels) and Python bindings
      • Initial skeleton CI pipeline developed (for build, test, lint, docs stages)

      Where at all possible this should adopt the SKA standardised project structure or otherwise follow from existing projects rather than try to invent an entirely new approach.

      This does have to be perfect and can be iterated on but needs to be in place to allow the development of this library to proceed with contributions from multiple developers/teams in parallel. 

       __ 

      Attachments

        Issue Links

          Structure

            Activity

              People

                b.nikolic Nikolic, Bojan
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 3.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1317.0
                  Total1317.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel