Consolidate design for Data Management in early array releases

A necessity is emerging to manage large data sets during the construction phase of the SKA1. This is not correctly captured in our current set of requirements and in the design and it represents a gap that needs to be addressed.

In the low telescope we are already experiencing in the AAVS prototype how a great quantity of data needs to managed as an output of the station verification activity. But this activity shall not be limited to the MCCS design. 

The problem is multi-faceted and it needs to be addressed with a system view as many systems interoperate in order to realise this capability. In particular, this will involve at least: MCCS, Dish, TMC, SDP and the Network. 

It will be essential to establish: 

• What intermediate data products need to be stored and curated by each different subsystem. In doing this analysis it will be important to evaluate the data size and the related processing needs. i.e. we will have to establish if data products can be stored onsite or if they need to be transferred somewhere else for further analysis. Note that this applies also to all data sets that will be used for testing purposes where the traceability to such data sets needs to be maintained in the long term. these results will be collected at https://confluence.skatelescope.org/display/SE/SS-53+consolidate+design+for+Data+Management+in+early+array+releases This can easily be applied to: 
  • MCCS 
  • Dish
  • LOW CBF
  • MID CBF
• Understand the impact of analysing beamformed data in early array releases using PSS/PST software. A tradeoff exists between deploying PSS/PST components onsite or just offloading data produced by the CBF to external locations for further analysis. In each case, we need to consider impacts on network bandwidth and rack space for storage and execution. 
• Review the estimates related to EDA data size and availability. This might be just a quick review exercise based on existing documents. It will also be useful to understand if replication of such data off-site is immediately needed in the early stages of commissioning and operations and to estimate the related bandwidth necessary so to inform the requirements on the network infrastructure. 
  • Other use cases for different EDA deployments: PSI, ITF. 
• Science data produced onsite using the SDP system. What is the expected size of such data products and where are they transferred and stored. 
• SDP Delivery architecture: can this be analysed so that it provides a solution for the different subsystems to store intermediate products off-site? What is an alternative solution? 
• evaluate network bandwidth requirements for Mid and Low.