Archiving and visualising engineering parameters with the EDA

Why?

Being able to achieve engineering parameters will help with observability and debugging the system. 

Providing simple query (and visualisation) capability is important for usability.

Implementing this function now is valuable as it will be needed as soon as multiple successive observations are executed in a persistent environment (cloud --> ITF --> AA0.5)

Who?

1. AIV engineer
2. Developer
3. Commissioning Scientist 
4. (Engineering Operator)

What? (outcomes?)

Archiving engineering parameters.

• As an 'operator' (AIV engineer, commissioning scientist) I want to be able to query a record of the health of the various subsystems over time that has been recorded when executing an observation script which performs functions such as executing a series of scans that results in ingest of visibility data from the (simulated) CBF.
• When configuring the system, I want to be able to select what set of engineering parameters are archived and with what policy. For example, as a minimum, one would expect to be able to archive the set of metrics related to the underlying hardware with a given polling frequency. Clear instructions should be provided to all teams developing control products about how to configure the archiving for the single subsystems. 
• In order to enable this capability, the necessary database(s) and TANGO infrastructure need to be in place. TANGO Archivers are deployed and the database backend is deployed as decided in ADR-61 AA0.5 EDA Design and Deployment 

Starting with archival of TMC parameters, it will be important that for every product there is a plan to start working on the initial archiving configuration. Depending on the planning activities, it would be good to start extending the archiving capabilities in PI16 to the different subsystems. This will create dependencies with possibly all teams that are working on products that present a control interface.

Important NFRs and Success Metrics

Usability: the main focus in this PI should be on ease of configuration for the different subsystems in specifying their archiving policy. Correctly distributing this function is crucial to advance the development activity in a distributed way without creating bottlenecks. Initial considerations must also involve ease of data retrieval and visualisation in order to functionally enable consultation and querying of the archived data. 

Accuracy: for each subsystem configured, it is important to correctly set archival rates and configuration. The control system guidelines (or a different document TBD) shall provide guidance on how to correctly configure the mapping of archiving functions to hardware subsystems. 

Scalability: at the moment this is more related to the horizontal distribution of data and sharding. If multiple databases are deployed in AA0.5 for different TANGO hosts, we need to consider how different archivers are deployed in relation to this configuration and how access is provided to every instance for querying. It is essential to have a way to access all data from a central location at this stage. It will be important in the meantime to design a way to aggregate the data in a single database or querying interface. Vertical scalability is also to be considered in trying to configure each system for archiving at the proper rate. In this context, it will be essential to provide a summary of the expected data rate for each subsystem that is configured as part of this implementation.

References

1. ADR-61 AA0.5 EDA Design and Deployment 
   #