Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1290

Develop Initial implementation of Plasma Casacore Storage Manager

Details

    • Data Processing
    • Hide

      Providing an integration between Apache Arrow/Plasma and Casacore-based tools would enable reusing existing production-ready software.

      Show
      Providing an integration between Apache Arrow/Plasma and Casacore-based tools would enable reusing existing production-ready software.
    • Hide
      • An initial implementation of a Plasma-based casacore storage manager that compiles against the latest casacore release.
      • Simple example program reading existing Arrow data from Plasma using the new casacore storage manager (i.e., using the casacore API).
      • Stretch: An initial implementation of a casacore Table that is comprised only of Plasma-based columns, binding together several columns (i.e., Apache tensors) living in the same Plasma store, and presenting them as a single entity for reading.
      • Stretch: Show a casacore-based application interacting with such tables/columns.
      Show
      An initial implementation of a Plasma-based casacore storage manager that compiles against the latest casacore release. Simple example program reading existing Arrow data from Plasma using the new casacore storage manager (i.e., using the casacore API). Stretch : An initial implementation of a casacore Table that is comprised only of Plasma-based columns, binding together several columns (i.e., Apache tensors) living in the same Plasma store, and presenting them as a single entity for reading. Stretch : Show a casacore-based application interacting with such tables/columns.
    • 2
    • 2
    • Team_YANDA
    • Sprint 3
    • Show
      Repo: https://gitlab.com/ska-telescope/plasma-storage-manager Slides: https://docs.google.com/presentation/d/1kxWvy3U91tkj6fdlyJ6DIcm9THbjDwO1wX4PL8JVaEQ/edit
    • 8.4
    • Stories Completed, Outcomes Reviewed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
    • PI22 - UNCOVERED

    • goal_D1

    Description

      Preamble: This feature was cloned/split from SP-841 following a discussion with b.mort and p.wortmann during the PI8 planning meeting. It provides a lightweight implementation, which is the best that we felt can be done given that SP-1174 has not been pulled from the program board.

      After the initial work done in SP-1023, usage of Apache Plasma/Arrow for data interchange between processing components is being extended. An interesting avenue of exploration is to provide an initial implementation of an casacore Storage Manager that can read Apache Arrow tensors off an Apache Plasma server, and expose their data as casacore Columns. This can be then further composed into representing Tables and Measurement Sets. This work would build a bridge between current prototyping activities and existing production software, further expanding our ability to reuse existing software.

      This initial implementation is meant to be very simple and only demonstrate that the two technologies can be integrated, and that existing Arrow data from Plasma can be read and presented as a column. As a stretch, more advanced features like reading full tables and MSs might see some advance.

      Attachments

        Issue Links

          Structure

            Activity

              People

                b.mort Mort, Ben
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete912.0
                  Total912.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel