Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1985

Decompose and integrate imaging-iotest processing functions

    XporterXMLWordPrintable

Details

    • Enabler
    • Should have
    • PI12
    • COM SDP SW
    • None
    • Services
    • Hide

      The processing functions of the I/O tests are fairly simple, yet integrating them opens up a number of interesting possibilities:

      • We can use them to test other execution engines (specifically their capability of dealing with high throughput and memory requirements)
      • As the stages (distributed FFT, degridding) are fairly standard, there is a good chance we might be able to interface this with other processing functions in the future.
      Show
      The processing functions of the I/O tests are fairly simple, yet integrating them opens up a number of interesting possibilities: We can use them to test other execution engines (specifically their capability of dealing with high throughput and memory requirements) As the stages (distributed FFT, degridding) are fairly standard, there is a good chance we might be able to interface this with other processing functions in the future.
    • Hide
      • Introduce / finish factoring processing functions out from imaging-iotest
        • Degridding (mostly done)
        • FFT & w-shifting/w-towers (started)
        • "recombination" distributed FFT
      • Add necessary functional tests, as well as systematic performance tests of at least one processing function**
      • Write developer documentation and generate API documentation (breathe/Sphinx?)
      • Stretch: Re-introduce into an execution engine (likely easiest to just refactor existing MPI implementation)
      • Stretch: Add Python bindings
      Show
      Introduce / finish factoring processing functions out from imaging-iotest Degridding (mostly done) FFT & w-shifting/w-towers (started) "recombination" distributed FFT Add necessary functional tests, as well as systematic performance tests of at least one processing function** Write developer documentation and generate API documentation (breathe/Sphinx?) Stretch: Re-introduce into an execution engine (likely easiest to just refactor existing MPI implementation) Stretch: Add Python bindings
    • 3
    • 8
    • 2.667
    • Sprint 4
    • 17.4

    • PI22 - UNCOVERED

    Description

      The imaging-io test is a prototype of a distributed (currently MPI) image-based visibility predict code that has been demonstrated to achieve high data throughput and presents a model that may be of interest to SDP pipelines.

      Decomposing this into standalone processing functions (which will provide the building blocks for SDP pipeline development), and once developed, testing these within this framework is a natural next step to investigate the approach taken in this code and would provide an excellent test of how we might design interfaces to processing functions.

      A start of this has been made at https://gitlab.com/scpmw/ska-sdp-func

      Attachments

        Issue Links

          Parent Of

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1826 Establish the SKA SDP Processing Function Library

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          mentioned on

          GitLab Commit - Merge branch 'sp-1985-fix-wheel-build-arrow-7.0' into 'master'

          GitLab Commit - Merge branch 'sp-1985-packaging' into 'master'

          GitLab Commit - Merge branch 'sp-1985-refactoring-and-generalisation' into 'master'

          GitLab Commit - Merge branch 'sp-1985-tests-and-docs' into 'master'

          GitLab Commit - SP-1985 Actually copy the conversion scripts into the image

          GitLab Commit - SP-1985 Actually produce clang-tidy.out

          GitLab Commit - SP-1985 Add "newaxis" support

          GitLab Commit - SP-1985 Add axis defaults to gridder as well

          GitLab Commit - SP-1985 Add CI stage to build Conan packages

          GitLab Commit - SP-1985 Add Conan files

          GitLab Commit - SP-1985 Add defaulting of dimension names

          GitLab Commit - SP-1985 Add DumpTensor

          GitLab Commit - SP-1985 Add essentials to make "python setup.py" work

          GitLab Commit - SP-1985 Add FFT broadcast dimension

          GitLab Commit - SP-1985 Add h5py to dependencies

          GitLab Commit - SP-1985 Add note about data copying to guidelines

          GitLab Commit - SP-1985 Add parameter names to Pybind11 bindings

          GitLab Commit - SP-1985 Add pip-installation of cppcheck-junit and junit-xml

          GitLab Commit - SP-1985 Add proper gridding tests

          GitLab Commit - SP-1985 Add pybind11_mkdoc to dependencies

          GitLab Commit - SP-1985 Add pybind11_mkdoc to dependencies

          GitLab Commit - SP-1985 Add Python dependencies to Dockerfile

          GitLab Commit - SP-1985 Add Python Sphinx documentation

          GitLab Commit - SP-1985 Add tests for Python bindings

          GitLab Commit - SP-1985 Add uvw_grid to Python interface

          GitLab Commit - SP-1985 Add warning about using pre-C++11 ABI

          GitLab Commit - SP-1985 Additional uvw_grid refactoring

          GitLab Commit - SP-1985 Also add git to CI container

          GitLab Commit - SP-1985 Also silence c-ares search

          GitLab Commit - SP-1985 Also update pip

          GitLab Commit - SP-1985 Another attempt at making code that runs without AVX2

          GitLab Commit - SP-1985 Another attempt at rebuilding the CI Docker image

          GitLab Commit - SP-1985 Attempt to allow pip to build its own wheel?

          GitLab Commit - SP-1985 Attempt to build (fixed-up) wheels

          GitLab Commit - SP-1985 Attempt to build Python packages for different Python vers

          GitLab Commit - SP-1985 Attempt to build Python packages for different Python vers

          GitLab Commit - SP-1985 Attempt to build Python packages for different Python vers

          GitLab Commit - SP-1985 Attempt to disable publishing pages on non-master branches

          GitLab Commit - SP-1985 Attempt to disable publishing pages on non-master branches

          GitLab Commit - SP-1985 Attempt to disable publishing pages on non-master branches

          GitLab Commit - SP-1985 Attempt to fix test_as_subproject

          GitLab Commit - SP-1985 Attempt to make types more consistent

          GitLab Commit - SP-1985 Attempt to re-enable coverage reports

          GitLab Commit - SP-1985 Attempt to use ctest2junit output directly?

          GitLab Commit - SP-1985 Code coverage: Also export per-file coverage reports

          GitLab Commit - SP-1985 Compatibility

          GitLab Commit - SP-1985 Copy Mahendra's JUnit conversion scripts

          GitLab Commit - SP-1985 Disable CMake verbose output

          GitLab Commit - SP-1985 Document the wheel building process a bit

          GitLab Commit - SP-1985 Document workaround in README.md

          GitLab Commit - SP-1985 Document workaround in README.md

          GitLab Commit - SP-1985 Documentation restructuring

          GitLab Commit - SP-1985 double_tensor_ptr -> tensor_ptr<double>

          GitLab Commit - SP-1985 First attempt at UVW generation, using Arrow tables

          GitLab Commit - SP-1985 First Python gridder tests

          GitLab Commit - SP-1985 First UVW handling, using Arrow tables

          GitLab Commit - SP-1985 First version of Python bindings

          GitLab Commit - SP-1985 Fix ctest-to-junit.py execution permissions

          GitLab Commit - SP-1985 Fix finding Arrow with Conan

          GitLab Commit - SP-1985 Fix finding Arrow with Conan

          GitLab Commit - SP-1985 Fix FindPyarrow for when wheel is using system arrow

          GitLab Commit - SP-1985 Fix h5py package name

          GitLab Commit - SP-1985 Fix kernel generationn

          GitLab Commit - SP-1985 Fix macros

          GitLab Commit - SP-1985 Fix Python tests

          GitLab Commit - SP-1985 Fix typo

          GitLab Commit - SP-1985 Fix warning about creating "build"

          GitLab Commit - SP-1985 Fix wheel build for Arrow 7.0

          GitLab Commit - SP-1985 Fixed some du/dv confusion

          GitLab Commit - SP-1985 Flip UVW sign to correspond with casacore convention

          GitLab Commit - SP-1985 Have automatic PyBind11 conversion for NumPy arrays

          GitLab Commit - SP-1985 Have Conan run tests as part of the build process

          GitLab Commit - SP-1985 Import pyarrow bits when Python module is initialised

          GitLab Commit - SP-1985 Improve code coverage (mostly triggering exceptions)

          GitLab Commit - SP-1985 Install headers into subdirectories

          GitLab Commit - SP-1985 Install libarrow-python-dev as well

          GitLab Commit - SP-1985 Install Pyarrow+Pybind11 into CI container

          GitLab Commit - SP-1985 Install Pyarrow+Pybind11 into CI container

          GitLab Commit - SP-1985 Install Pyarrow+Pybind11 into CI container

          GitLab Commit - SP-1985 Introduce processing component destructors

          GitLab Commit - SP-1985 Make Conan build work again

          GitLab Commit - SP-1985 Make ctest generate output if a test case fails

          GitLab Commit - SP-1985 Make ctest generate output if a test case fails

          GitLab Commit - SP-1985 Move gridder to use exceptions as well

          GitLab Commit - SP-1985 Prevent cppcheck from aborting lint stage

          GitLab Commit - SP-1985 Properly allocate numpy array used for complex conversion

          GitLab Commit - SP-1985 Put wheel audit into separate for loop

          GitLab Commit - SP-1985 Pybind11 type casters name tensor element type

          GitLab Commit - SP-1985 Query pyarrow properly for library directory

          GitLab Commit - SP-1985 Refactor double_tensor_ptr -> tensor_ptr<double>

          GitLab Commit - SP-1985 Refactor pybind11 caster a bit

          GitLab Commit - SP-1985 Refactor tensor interface

          GitLab Commit - SP-1985 Refactor tensor interface

          GitLab Commit - SP-1985 Refactor tensor interface

          GitLab Commit - SP-1985 Refactor wshift_fft using tensor_dim_info

          GitLab Commit - SP-1985 Register Python test files separately with CTest

          GitLab Commit - SP-1985 Remove debug logs

          GitLab Commit - SP-1985 Remove generated file

          GitLab Commit - SP-1985 Remove lint step in CMake

          GitLab Commit - SP-1985 Remove now-uneeded macros

          GitLab Commit - SP-1985 Remove reference to grid.cc

          GitLab Commit - SP-1985 Remove step0 from degrid, increase test parameter coverage

          GitLab Commit - SP-1985 Remove unneeded du/dv/dw members

          GitLab Commit - SP-1985 Remove unused macros

          GitLab Commit - SP-1985 Rename fft.cc -> bindings.cc

          GitLab Commit - SP-1985 Replace assertions by proper exceptions

          GitLab Commit - SP-1985 Replace std::pair by tensor_dim_info, simplify uvw_grid

          GitLab Commit - SP-1985 Revert to C++11 standard

          GitLab Commit - SP-1985 Revise how Pybind11 is looked for

          GitLab Commit - SP-1985 Revise solution for AVX2/FMA kernels again

          GitLab Commit - SP-1985 Second attempt at manylinux2014 build...

          GitLab Commit - SP-1985 Second attempt at manylinux2014 build...

          GitLab Commit - SP-1985 Shorten dimension names

          GitLab Commit - SP-1985 Silence CMake warnings about gRPC and re2

          GitLab Commit - SP-1985 Slightly refactor C++ tests

          GitLab Commit - SP-1985 Some CMake refactoring

          GitLab Commit - SP-1985 Some fixed to make building as a shared library work

          GitLab Commit - SP-1985 Support strided Numpy Arrays and dimensionality mismatches

          GitLab Commit - SP-1985 Switch to new CI Docker image

          GitLab Commit - SP-1985 Switch to obtaining Arrow from the Python wheel

          GitLab Commit - SP-1985 Switch to obtaining Arrow from the Python wheel

          GitLab Commit - SP-1985 Switch to using new CI image

          GitLab Commit - SP-1985 Test FFT with irregular resolution

          GitLab Commit - SP-1985 Untabify

          GitLab Commit - SP-1985 Update CI Docker build

          GitLab Commit - SP-1985 Use complex() instead of a+b*I

          GitLab Commit - SP-1985 Use exceptions for indicating parameter errors

          GitLab Commit - SP-1985 Use image from CAR for build base

          GitLab Commit - SP-1985 Whitespace changes

          GitLab Merge request - SP-1985 Add building of Conan packages

          GitLab Merge request - SP-1985 Fix wheel build for Arrow 7.0

          GitLab Merge request - SP-1985 Refactoring & fixes, add broadcasting behaviour

          GitLab Merge request - SP-1985 Tests and documentation...

          Structure

            Activity

              People

                p.wortmann Wortmann, Peter
                f.graser Graser, Ferdl
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 3.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete13.0
                  Total13.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel