SDP Data Management

We have partially implemented SDP data management. We chose to pursue the idea of using a single shared volume to store the data that is written by SDP processing blocks. ADR-55 proposes how the data (and metadata) should should be managed: data products are written to a directory whose path contains the execution block and processing block IDs. On a Kubernetes cluster the storage would ideally be realised as a persistent volume shared between the control and processing namespaces, so the control system and services can see the data written by the processing deployments. This has so far proved impossible to implement on the DP cluster, which delayed the completion of this feature. In order to make progress with the visibility receive processing script, we implemented a temporary workaround in some of the Orca team namespaces to use non-shared volumes. We have created a draft MR to update the SDP integration tests to use an existing persistent volume for the visibility receive test, where is is available, with the intention of running it on the persistent deployment in the dp-shared(-p) namespaces. At the moment it runs on a persistent deployment in Nijin's namespaces. The MR should be updated when it is decided what to do about the dp-shared(-p) namespaces. We could apply the workaround and use non-shared volumes for the time being. However, it would be preferable to find a solution that allows us to share persistent volumes between namespaces and implement that straight away. This needs the assistance of the System team.