Complete integration of signal displays for visibility receive

Simple, configurable displays will be one of the first functions needed by a PSI/AA operator when deploying and debugging SDP in the context of the wider system.

This will allow the operator to have a quick-look functionality to establish if the CBF output and SDP inputs are behaving as expected without having to access and inspect data products.

• Complete helm scripts for integration with sdp-prototype element within SKAMP - (We will be doing a direct integration of the python metric generator with the CBF-SDP-Emulator (which will be integrated with sdp-prototype), which should already have its own helm script(s)). –
• Contribution to simple system test that validates the displays are working without having to manually inspect the dashboard (by querying the database?) - (We will use localized flat files for validation of working metric generator  - NZAPP-251)

• Review options for generating the following widget types, including (@Andrew Ensor)

• • Amp/phase baseline spectrum plot
  • Waterfall plots
• Stress test of metrics displays to PSI data sizes and rates (and beyond to SKA) (@Andrew Ensor)

• • Theoretical review
  • Benchmark tests
• Document dashboard configuration for selecting different baselines and widgets (@Andrew Ensor)
• Code and configuration should meet SKA DoD (or at a minimum report against TBD CI / code quality metrics) so an assessment of technical debt can be made and addressed in future PIs  (Clean up of technical debt, update source code for compliance as per developer documentation, linting, code coverage, and updating source code documentation. Continuous integration is in theory taken care of by the CBD-SDP-emulator, which we will be integrating with - NZAPP-252)
• System demo aimed at key stakeholders (@Andrew Ensor)

• • PSI Operator
  • System scientist / Commissioning scientist
• Further socialisation with other workflow development teams with a view towards extending this for other QA displays in future. (@Andrew Ensor)
• Extend existing python metric generator to allow for configurable data endpoints (ie: flatfile, or time-series DB) - NZAPP-253.
• Extend existing python metric generator to provide dynamic control over active metrics (ie: create, read, update). Requires discussion with Steve Ord regarding best approach to expose this functionality to end users (ie: API) - NZAPP-254.
• Finalize integration of python metric generator with CBD-SDP-emulator (meeting with Steve Ord to discuss best approach) - NZAPP-255.

[NZAPP-253]: The metric generator is now able to be configured externally utilizing a YAML file, or by passing the configuration args through to the metric generator Docker container (ie: database connection details).

[NZAPP-254]: The metric generator now exposes functions which allow for metric management (ie: update, delete, query, etc. of metrics). However, these need "hooks" to be called externally from a running instance of the metric generator (TANGO?).

[NZAPP-252]: Code compliance has been performed, and kindly reviewed by the YANDA team. Documentation is in the form of the repository README, covering use cases, examples, and descriptions of operation. Code coverage for testing sits at ~92%, as we have not yet been able to unit test the database functionality without a centrally located instance of InfluxDB (to be decided in the future by someone at the Systems team level?).

[NZAPP-251, NZAPP-255]: Discussions with the YANDA team have provided us with an approach for dynamic integration with the CBF-SDP-Emulator. They have implemented the metric generator as a custom consumer object, and we have "containerized" the metric generator using the CBF-SDP-Emulator as a base image. Integration with the "sdp-prototype" should technically come "for free", given that we dynamically integrate with the emulator, and the emulator integrates with the prototype.

[NZAPP-277]: Implemented documentation in the form of README, completed containerization via Dockerfile. This has been mentioned already.

[NZAPP-279]: Preliminary performance results were obtained, and reported on the linked ticket. Note these timings are crude, such that they are the total execution time of the container.

[NZAPP-260, NZAPP-268, NZAPP-280]: Grafana dashboard has been implemented for the metric generator demonstration, this has been uploaded into the "dashboards" section of the metric generator repository.

[NZAPP-278]: System demonstration was performed by Seth Hall. Demonstration of containerized CBF-SDP-Emulator send/receive with metric generator was performed, dispatching metrics to a locally hosted InfluxDB, and viewed in a web browser using the Grafana visualizer (custom dashboard).

Demonstration: https://confluence.skatelescope.org/display/SE/2020-11-18+DP+ART+System+Demo+8.6

Repository: https://gitlab.com/ska-telescope/cbf-sdp-emulator-metrics-generator