Simplify visibility receive processing script parameters

The visibility receive processing script implements the most important function that the SDP is expected to provide at AA0.5. At present the parameters are complicated and they expose too many implementation details to the users.

This is a result of the way the receive Helm chart, which makes the deployment of the receiver components, is written. It was originally written years ago, when the receiver itself was much simpler than it is today. Many of the concepts that now are a core part of a full receiver deployment (e.g., the plasma store, the different processors that attach to it, an external PVC, etc) were not present in the chart; what happens instead is that the receiver chart gives full flexibility and responsibility to users, who need to provide all these extra bits of information in many different places. On top of that, the vis-receive script tries to fill some of these gaps itself, leading to two layers of complexity that need to be dealt with when writing PBs that run the vis-receive script.

As an example, at the moment a user wanting to use the mswriter processor that takes visibilities from the plasma store, needs to put the following in the PB parameters:

"extraContainers":[
   {  
      "name":"plasma-processor",
      "image":"artefact.skao.int/ska-sdp-realtime-receive-processors:0.3.0",
      "args":[
         "--max-payloads",
         "12",
         "--readiness-file",
         "/tmp/processor_ready",
         "/mnt/data/output.ms"
      ], 
      "volumeMounts":[
         {  
            "name":"plasma-storage-volume",
            "mountPath":"/plasma"
         }, 
         {  
            "mountPath":"/mnt/data",
            "name":"testing"
         }  
      ]  
   }  
]                                                                                                                                                                                                                                                                                                                              

There are a few issues already here:

• These two volume mounts need to be manually specified (with information the user is unlikely to know), even though the corresponding volume definitions are automatically created. Instead, the volumeMounts should also be automatically created.
• The volume for the first volumeMount is added by the vis-receive script, while the second is created by the receiver chart. They should both be created by the receiver char.
• The name of the second volumeMount must match the SKA_DATA_SDP_PVC name. The unit tests using this file modify this field after loading this example, which is why tests work.

This feature is to rewrite the receive chart and simplify the logic based on these changes, leading to simpler parameters for the vis-receive script.

The receive chart should be easier to configure and use, with as much setup as possible happening within the chart itself instead of leaving details to users of the chart. In particular, as few chart details as possible should spill out into the processing block parameters, which should deal mostly with domain-specific matters.

Since the changes to the receive chart will not be small, backwards-compatibility will be lost. The current infrastructure does not allow for this, because

• processing scripts (or scripting library) cannot specify the chart version they want to use, and
• the Helm deployer chart repository contains only one version of each chart.

These two issues should be tackled, which has the added benefit that refactoring deployments should become easier in future.

Another minor issue to tackle is the that the stateful set always uses "receive" as the service name, which means that multiple deployments cannot be made simultaneously, even in different processing blocks on different subarrays.

Who?

• SDP user
• AIV engineer
• Commissioning scientist

What?

• New version of the vis-receive processing script with simplified parameters.
• New version of the receive Helm chart with simplified parameters.
• Scripting library allows version of Helm charts to be specified.
• Helm deployer chart repository can contain multiple versions of a chart.

Why?

• Visibility receive processing block parameters are complicated and expose too many implementation details to users of SDP.
• Allowing scripting library / processing scripts to specify version of the Helm charts internally will make it easier to refactor things in future (e.g. replacing plasma with another technology).