Pairwise testing of control interfaces (continued from SS-116)

Miro Backlog frame: https://miro.com/app/board/uXjVN4N0K4c=/?moveToWidget=3458764578056020785&cot=14

Who & Why? (Benefit hypothesis)

• Provide the ability to quickly and easily share the current test coverage of control system interfaces with stakeholders (AIV, other development teams, TDTs, system science, Systems Engineers) for review and to add implementation detail to the description of the interface.
• Increases confidence of correct behaviour and robustness of control and monitoring interfaces between subsystems.
• Another step toward full system Integration testing.

What?

• Provide a set of XRay Test Plans with test executions interfaces defined in the list of Test Plans below, which are present in the "Solution and System Acceptance Tests” (XTP) JIRA project. The test plans should cover:
  • commands needed for subarray state model transitions involved in running an observation.
  • and attributes/data provided and consumed by TMC alongside these commands (e.g. asserting that delay polynomials) are provided along with executing a scan in the correct format as defined in the ICD.
• Test plans are reviewed and agreed by Design Authorities of subsystems on both sides of the interface for completeness / correctness against the current ICD definition and interface behaviours, updating the current ICDs as needed.
• Improvements to test ware, test quality, and code quality are made.
• • (If time permits) Refactoring the current tests to take advantage of further test harness / test ware improvements made in PI22. 
  • Remove duplication across in the test harness and create a python package.
  • Continue to review and improve test cases for consistency of terminology which should also reduce unnecessary code duplication
  • Review and improve allocation of scenarios to feature file definitions.
  • Consideration for unhappy path tests such as those occurring when the underlying subsystem does not respond or is sent invalid data as part of a command.
• Subsystem teams conduct some exploratory testing with the TMC, including injecting faults in the interfacing subsystem to assess behaviour. (SDP, CSP.LMC, Dish.LMC).
• Documentation/descriptions of the API provided by products that TMC interfaces with is complete and sufficient to describe the interface commands, attributes, and behaviours.

Expected Outcomes

1. Design Authorities, AIV, TDTs, System Sci, System Eng, can easily assess the current status and interface coverage of TMC integration with other subsystems by inspection of a set of Test plans.
2. The confidence in the robustness of the telescope controls increases as the scope of the test coverage expands beyond state transition, to cover expected and unexpected scenarios.

Test Plans for pairwise testing of AA0.5 control system interfaces

1. Low TMC CSP LMC: https://jira.skatelescope.org/browse/XTP-29682
2. Low TMC - SDP: https://jira.skatelescope.org/browse/XTP-29227
3. Low TMC - MCCS: https://jira.skatelescope.org/browse/XTP-30488
4. Mid TMC CSP LMC: https://jira.skatelescope.org/browse/XTP-29583
5. Mid TMC - SDP: https://jira.skatelescope.org/browse/XTP-29381
6. Mid TMC - Dish LMC: https://jira.skatelescope.org/browse/XTP-29778
7. TBD: OSO - TMC: <TO DO>
8. TBD: CBF (on the assumption that not all data going to CSP goes via CSP LMC)

Success Measures?

1. A set of Test plans for TMC - * interfaces are available in XRay and have been reviewed and accepted by DAs (and/or DDAs) for validity against the current ICD and associated requirements (small 'r' not necessarily the ones in JAMA right away) for AA0.5. Covering at minimum all Controls State transitions (ADR-8), some unhappy path testing and initial 'higher-level' functionality (e.g. ensure delays are provided to CBF at expected location etc).
2. Test Plans linked to REL tickets for future releases as part of the regression test suite.
3. Initial exploratory testing of >1 subordinate subsystem is conducted

References

• (Low PI21 work) https://confluence.skatelescope.org/display/SE/%5BDRAFT%5DTMC-SDP+scenario+for+Low+telescope
• (Mid PI21 work) https://confluence.skatelescope.org/display/SE/%5BDRAFT%5DMid+BDD+Scenarios+for+pair+wise+testing