TM Interface via SDP evolutionary prototype: Processing Block support in DALiuGE

• Build understanding of the architecture. Maybe plan a spike for some Q&A to make sure we have a shared understanding of how this is supposed to work?
• Once ORCA-26 and ORCA-73 are progressed far enough (likely next sprint at the earliest) it should be possible to put a rough first version of a workflow together (containerised DALiuGE, workflow script, JSON template for workflow parameters)

Outcomes related to acceptance criteria:

• The first point in the acceptance criteria ("Build understanding of the architecture") has been achieved mainly through the reading of the old SIP code, but also through the discussions with Peter, and reading the architecture documents pointed to by him in this ticket.
• The second point ("... put a rought first version of a workflow together") has also been delivered, and actually demonstrated during https://confluence.skatelescope.org/display/SE/2019-08-08+System+Demo+%233. The link contains the demo agenda, where a full recording of the demo can be found, alongside with the couple of slides that were shown during this presentation.
• The integration was done by providing a different Processing Block Controller implementation (available at https://github.com/ska-telescope/daliuge-pbc), but we know this is not going to be exactly how different execution frameworks will be plugged into the rest of the SDP prototype in the future; instead, this logic will most likely be split between the "Workflow Control Libraries" and "Execution Framework Interface" components of the architecture.
• A patch was sent to the SIP codebase to have the celery infrastructure working with non-SIP code (PR here: https://github.com/SKA-ScienceDataProcessor/integration-prototype/pull/103). The patch was accepted as it might be useful in the SDP prototype code, even though the two code bases don't share code.

Not tackled (future work, capturing in SP-469):

• Acquiring of resources. This was done via docker compose/swarm; this should be taken care of by the "Control & Configuration Scripts" and "Processing Block Controller" components in the architecture.
• Pipeline/workflow/PB parameters were not tested. The artificial Processing Blocks didn't contain any, and our dummy pipeline didn't require any either. However, in principle DALiuGE supports this already. This point was also dropped from the acceptance criteria early during the process, and therefore does not harm the value of this feature.
• Adding more interesting pipeline logic (e.g., SPEAD reception of visibilities + MS writing).

Others:

• The Processing Block states seemed to be free strings, at least on the SIP code. Is this still the same in the SDP prototype?