Example of tracing TANGO command events throughout the system

Timeboxing this into a Spike for PI5. Further work could be done as a follow up captured in Feature SP710 (Clone)

Syslog and the Elastic stack were identified for logging solution in Pre-construction and the containerisation standards, with TANGO as telescope SCADA.

Come up with solution for distributed tracing events throughout the system, including what is handled via e.g. ELK, and TANGO.  For instance, tracing from scheduling block execution to downstream events.

As a stretch, this ticket could be expanded to also integrate proposed/selected solutions for monitoring of virtual and physical compute infrastructure as well as logs and TANGO.

Example solutions (to give an idea of what is meant by tracing)

• ISTIO
• Using APM agents, Elastic APM and APM Kibana GUI supports distributed tracing.  Elastic APM also automatically collects unhandled errors and exceptions.

This feature will be tested as follows:

A design review of the documentation will be held with Architects after which all of their observations are addressed. However, the design review should occur in an iterative fashion to get continuos and quick feedback. The test will, therefore, be a demonstration that all observations from architects were addressed and found acceptable. The documentation will need to be made official and be seen as part of solution intend.