Loading...

Change Owns to Parent Ofs

Set start and due date...

Xporter

XML

Word

Printable

Details

Type: Enabler
Priority: Must have
Fix Version/s: PI19
Component/s: COM TMC SW
Labels:

ARTs:

Obs Mgt & Controls
Benefit hypothesis:
Hide

we can abstract the detail of test fixture setup (which is time-consuming, but will be used 1000s of times over the project lifetime, thus realising us long-term savings).

we can provide a common abstraction for testing TANGO devices that also allows us to work with the FAULT states and the more complete state machine implementations we have now.

such an abstraction can be used as a trail-blazer for other sub-systems/components.

test scenarios can be more easily understood because we can implement scenarios' steps using such an abstraction, and if this is done according to a "clean code" approach, steps implementations will be readable, contain only relevant stuff, and therefore more understandable.

test scenarios that are better formulated means that it is easier to understand what they do, and we can reuse them as living documentation.

we can improve the quality of software releases by supporting a better monitoring of the quality status of the TMC.
Show
we can abstract the detail of test fixture setup (which is time-consuming, but will be used 1000s of times over the project lifetime, thus realising us long-term savings). we can provide a common abstraction for testing TANGO devices that also allows us to work with the FAULT states and the more complete state machine implementations we have now. such an abstraction can be used as a trail-blazer for other sub-systems/components. test scenarios can be more easily understood because we can implement scenarios' steps using such an abstraction, and if this is done according to a "clean code" approach, steps implementations will be readable, contain only relevant stuff, and therefore more understandable. test scenarios that are better formulated means that it is easier to understand what they do, and we can reuse them as living documentation. we can improve the quality of software releases by supporting a better monitoring of the quality status of the TMC.
Acceptance criteria:
Hide

an abstraction layer for TMC sub-system and component level tests has been provided; this layer hides details about Tango and exposes "control-system"-related concepts (names and verbs).

this abstraction layer is used for a dozen of tests of state transitions in the TMC

the results of running these tests are clearly visible during development and as part of a TMC release

the abstraction layer and the tests have been demo'd and discussed at the Testing CoP

in the Xray representation of the test scenarios or test plans it is possible to identify such tests (perhaps via an appropriate label).

Out of scope:

Unit tests

Integration tests for integration of the TMC sub-system with other sub-systems (like CSP or SDP)

a refactoring of skallop is out of scope (though we will keep an eye on that).
Show
an abstraction layer for TMC sub-system and component level tests has been provided; this layer hides details about Tango and exposes "control-system"-related concepts (names and verbs). this abstraction layer is used for a dozen of tests of state transitions in the TMC the results of running these tests are clearly visible during development and as part of a TMC release the abstraction layer and the tests have been demo'd and discussed at the Testing CoP in the Xray representation of the test scenarios or test plans it is possible to identify such tests (perhaps via an appropriate label). Out of scope: Unit tests Integration tests for integration of the TMC sub-system with other sub-systems (like CSP or SDP) a refactoring of skallop is out of scope (though we will keep an eye on that).
Feature Points:
8
Initial Size:
8
WSJF:
0
Epic Link:
AA0.5 Observation Execution
Agile Teams:

Team_HIMALAYA, Team_SAHYADRI
Due Sprint:
Sprint 5
Story Point Burn-up:
Overdue:
Outcomes:
Hide

Implemented Test Suite using pure Python classes abstracting all Tango specific code.

Improved simulators for sub-system tango devices such as --> Csp Subarray, Sdp Subarray, Csp Master, Sdp Master and Dish Masters

Added Ability to inject delays in generating result

Added Ability to simulate transitions in given sequence.

Added ability to records all commands and argument for better debugging

Implemented common fixtures for SUT setup and teardown

Cleaner test case, better readability

Records all events in the SUT, via EventRecorder class which internally uses ska-tango-testing library for callbacks

Implemented ability to force subarray Observation State (obsState) to given value

Currently support testing of TMC Subarray Node and TMC Central Node

Below are the set of tests implemented

All the positive scenarios considering ObsState model as per revised ~~ADR-8~~

Verified all the possible observation state transitions like obsState.EMPTY, obsState.RESOURCING, obsState.IDLE, obsState.CONFIGURING,obsState.READY,obsState.SCANNING, obsState.READY, obsState.ABORTED

Negative scenarios, when TMC subarray stucks in Configuring obsState

Integration tests to verify SubarrayNode.healthState aggregation

Integration tests to verify CentralNode.telescopehealthState aggregation

Gherkins scenarios for healthState and telescopehealthState tests

Integration tests are parameterised to avoid code duplication

MR Link: https://gitlab.com/ska-telescope/ska-tmc/ska-tmc-integration/-/merge_requests/80

TMC negative scenarios and decision tables for healthStates: https://docs.google.com/spreadsheets/d/1XbNb8We7fK-EhmOcw3S-h0V_Pu-WAfPTkEd13MSmIns/edit#gid=0
Show
Implemented Test Suite using pure Python classes abstracting all Tango specific code. Improved simulators for sub-system tango devices such as --> Csp Subarray, Sdp Subarray, Csp Master, Sdp Master and Dish Masters Added Ability to inject delays in generating result Added Ability to simulate transitions in given sequence. Added ability to records all commands and argument for better debugging Implemented common fixtures for SUT setup and teardown Cleaner test case, better readability Records all events in the SUT, via EventRecorder class which internally uses ska-tango-testing library for callbacks Implemented ability to force subarray Observation State (obsState) to given value Currently support testing of TMC Subarray Node and TMC Central Node Below are the set of tests implemented All the positive scenarios considering ObsState model as per revised ADR-8 Verified all the possible observation state transitions like obsState.EMPTY, obsState.RESOURCING, obsState.IDLE, obsState.CONFIGURING,obsState.READY,obsState.SCANNING, obsState.READY, obsState.ABORTED Negative scenarios, when TMC subarray stucks in Configuring obsState Integration tests to verify SubarrayNode.healthState aggregation Integration tests to verify CentralNode.telescopehealthState aggregation Gherkins scenarios for healthState and telescopehealthState tests Integration tests are parameterised to avoid code duplication MR Link: https://gitlab.com/ska-telescope/ska-tmc/ska-tmc-integration/-/merge_requests/80 TMC negative scenarios and decision tables for healthStates: https://docs.google.com/spreadsheets/d/1XbNb8We7fK-EhmOcw3S-h0V_Pu-WAfPTkEd13MSmIns/edit#gid=0
Resolved PI.Sprint:
19.6

Feature Checklist:

BDD Testing Passes (no errors), Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

Requirement Status:

PI24 - UNCOVERED
Labels_MIRO:
Team_HIMALAYA Team_SAHYADRI testing

Description

Problems 1 &3 from the I&A workshops from PI19, and the Testing CoP retrospective from PI18 revealed some issues with the abstraction level we use for defining sub-system and system level tests (e.g. tests on the SDP, the TMC, or combinations thereof).
Giorgio & Verity are proposing to work closely with teams on the TMC sub-systems to improve the tests and also prototype a revised common abstraction for testing TANGO devices (to be socialised via the Testing CoP, once we deem it ok). This could be done as a refactoring/extension of skallop, or it could be a new product (where we've used skallop as our "build one to throw away" learning exercise). This decision would emerge from the design work on the Testing Harness product (skallop, or New-Product-Name-Here).
The design phase would involve multiple teams, and then be implemented by the selected teams as part of the coaching exercise.
This also works well with the overall goal of providing well-documented public APIs, well-documented, high quality tests, as part of sub-system releases to be deployed to the ITFs and to the sandbox environments to be tested as part of a larger system.
There is also the opportunity to provide some more examples of well-defined BDD tests and scenarios, with test steps and step definitions that can be reused by these teams and other teams working on the telescopes.

Attachments

Issue Links

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(1 mentioned in)

Structure

Activity

People

Assignee:: Brajnik, Giorgio

Reporter:: Allan, Verity

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Feature Progress

Story Point Burn-up: (100.00%)

Feature Estimate: 8.0

	Issues	Story Points
To Do	0	0.0
In Progress	0	0.0
Complete	10	76.0
Total	10	76.0

Dates

Created:: 02/Jun/23 11:03 AM

Updated:: 15/Apr/24 9:37 AM

Resolved:: 28/Aug/23 10:44 AM

I&A improvement: create a test harness for the TMC and improve quality of its integration/component tests