Build up the user-facing tools for tracing and diagnosing SKAMPI

Several tools exist for tracing and diagnosing on SKAMPI:

• Transaction IDs have been added
• Logging has been standardized
• Error handling has also been standardized

However, there is not much tooling available for diagnosing problems with SKAMPI, such as identifying when SKAMPI is stuck because transitions have not happened, or if a forbidden transition has occurred.

This capability aims to resolve that by providing tooling at both OMC and DP ART levels that can display to a user together the flow of information and state across SKAMPI.

We expect ALL major TANGO devices — i.e., at least the Master and Leaf nodes — to:

• Create a clear entry point for finding logging on SKAMPI through the landing page.
• Add logging for all events and commands following the SKA Logging standards.
• The ability to force transitions in several components when they are stuck, with meaningful Abort/Restart transitions implemented.
• Create a Log Consumer component that can react to information dumped in the logs, and can construct a health view of the system.
• Uncommitted on PI10: A transaction display similar to ElasticSearch APM that shows how data and state is flowing across SKAMPI.
• Uncommitted on PI10: creating a WebJive/Taranta dashboard to be provided by each TANGO Device Server. That could be part of a device_dashboard_json attribute, or at least a WebJive/Taranta .json file in the repository for the Device Server.

The ability to run SKAMPI from a selection of charts in a remote environment, and the ability to interact with it, is enabled by the multi-tenant SKAMPI Feature, and it is a pre-requisite for this Capability.