Details
-
Enabler
-
Not Assigned
-
Data Processing
-
-
-
2
-
2
-
Team_YANDA
-
Sprint 3
-
-
-
-
8.4
-
Stories Completed, Outcomes Reviewed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
-
-
goal_D1
Description
Preamble: This feature was cloned/split from SP-841 following a discussion with b.mort and p.wortmann during the PI8 planning meeting. It provides a lightweight implementation, which is the best that we felt can be done given that SP-1174 has not been pulled from the program board.
After the initial work done in SP-1023, usage of Apache Plasma/Arrow for data interchange between processing components is being extended. An interesting avenue of exploration is to provide an initial implementation of an casacore Storage Manager that can read Apache Arrow tensors off an Apache Plasma server, and expose their data as casacore Columns. This can be then further composed into representing Tables and Measurement Sets. This work would build a bridge between current prototyping activities and existing production software, further expanding our ability to reuse existing software.
This initial implementation is meant to be very simple and only demonstrate that the two technologies can be integrated, and that existing Arrow data from Plasma can be read and presented as a column. As a stretch, more advanced features like reading full tables and MSs might see some advance.
Attachments
Issue Links
- clones
-
SP-841 Develop library code that implements data models and schemas following SP-1174
- Discarded
- relates to
-
SP-1479 Add Table reading support to Plasma Storage Manager
- Done
- split from
-
SP-841 Develop library code that implements data models and schemas following SP-1174
- Discarded