Develop visibility receive benchmarks

A large part of this effort consisted of developing and extending an “integration repository”.  SKAO / Science Data Processor / SDP Realtime Receive Integration · GitLab This repository allows for automatic integration tests without complicated orchestration.  The intent being to provide lightweight infrastructure that can be easily deployed by visibility receive developers for the first stages of ‘integration’ within the visibility receive component. Although there is probably a need to streamline the integration at a level higher than this. For example some early comments were made by the FO that new processing scripts could be adapted to provide this functionality — the decision was made to limit the scope of this particular feature to a new repository within the component. The Acceptance Criteria were satisfied within this framework Acceptance Criteria: 1) Write benchmark processing scripts that involve all the pieces of the full ingest pipeline, including: cbf-emulator, vis-receiver, plasma and one or more processors (including, at least, mswriter and qametrics). These benchmarks would work with dummy data, and be able to scale along the different data dimensions. This is satisfied by a number of linked tickets that widen the capability of the integration repo including: Initial tickets to build the repo and move some integration tests from elsewhere in the visbility receive domain and consolidate them here (YAN-1511, YAN-1516, YAN-1517, YAN-1541, YAN-1576) The functionality of the cbf-emulator has also been extended to allow for more configurable data dimensions (YAN-1540) including dummy data as configurable dimensions and a test was written to exercise this (YAN-1552). 2) Check that strided spectral windows / channel definitions work throughout the pipeline as expected This has been widely tested revealing some corner cases and other issues solved by YAN-1584, with more fixes required in YAN-1433 and YAN-1581. 3) Tackle currently known bottlenecks. This work is still in progress - but it was deemed more practical to close out this feature and deal with more performance bottlenecks as they arise in PI 21. It is likely that there will be a follow-on feature. Some preliminary profiling was performed in YAN-1574, which backed up our intuition that memory copies are the primary receiver bottleneck. It was also observed that aggregation might need to be tunable to maximise plasma usage (e.g. >1GB aggregated payloads in a 2GB plasma store only allows one in-flight RPC call).