Loading...

Change Owns to Parent Ofs

Set start and due date...

Xporter

XML

Word

Printable

Details

Type: Feature
Priority: High
Fix Version/s: PI24
Component/s: COM SDP SW
Labels:
None

ARTs:

Data Processing
Benefit hypothesis:

Hide

As a AIV Engineer / Telescope Operator, I want to be informed immediately if there is a error in real-time processing, either in the processing scripts which orchestrate the execution of processing blocks, or in the execution engines and workflows that they deploy.

Show
As a AIV Engineer / Telescope Operator, I want to be informed immediately if there is a error in real-time processing, either in the processing scripts which orchestrate the execution of processing blocks, or in the execution engines and workflows that they deploy.
Acceptance criteria:
Hide

Each processing script monitors the execution engines and workflows it deploys, and reports any errors encountered, including in the script itself.

The processing controller monitors the processing scripts and, as a last resort, reports if a processing script has failed.

The Helm Deployer collects error messages from deployments and stores the information in deployment state.

Each subarray monitors its associated real-time processing blocks and reports errors via its errorMessage attribute.

Each subarray sets its ObsState to FAULT and HealthState to DEGRADED/FAILED when (agreed) conditions are met.

Implementation covers all six scenarios described in the Design link below (Confluence page).
Show
Each processing script monitors the execution engines and workflows it deploys, and reports any errors encountered, including in the script itself. The processing controller monitors the processing scripts and, as a last resort, reports if a processing script has failed. The Helm Deployer collects error messages from deployments and stores the information in deployment state. Each subarray monitors its associated real-time processing blocks and reports errors via its errorMessage attribute. Each subarray sets its ObsState to FAULT and HealthState to DEGRADED/FAILED when (agreed) conditions are met. Implementation covers all six scenarios described in the Design link below (Confluence page).
Expect Dependencies:

Intra Program
Feature Points:
3
Initial Size:
3
WSJF:
0
Agile Teams:

Team_ORCA
Due Sprint:
Sprint 5
Story Point Burn-up:
Overdue:

Requirement Status:

PI24 - UNCOVERED

Description

See Feature frame on DP ART board.

Context

See Capabilities SS-155, SS-156 and SS-185.

At present the SDP does not systematically monitor processing scripts and execution engines / workflows for any errors or failures that may occur. It only happens during some of the "long-running" subarray commands. The AssignResources command waits for the real-time processing scripts to indicate they have started successfully, and the Configure command waits for the execution engine deployments to succeed. If an error does occur it is not reported immediately, but only indirectly when the command times out. If an error occurs outside one of these commands, then it is not reported.

References

~~SP-4042~~ (Design spike in PI23)
- Design for reporting & monitoring of script and engine failure
MRs of initial implementation that need finishing:

Attachments

Issue Links

Child Of

SS-155 Low AA0.5 multi-station interferometry

Implementing

SS-185 Test basic Mid end-to-end interferometry scenarios

Implementing

SS-156 Low AA0.5/AA1 Control & Monitoring improvements

Implementing

mentioned in: Page Loading...; Page Loading...

Structure

Activity

People

Assignee:: Ashdown, Mark

Reporter:: Ashdown, Mark

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Feature Progress

Story Point Burn-up: (0%)

Feature Estimate: 3.0

	Issues	Story Points
To Do	1	0.0
In Progress	3	7.0
Complete	0	0.0
Total	4	7.0

Dates

Created:: 26/Jul/24 5:12 PM

Updated:: 3 days ago 10:59 PM

Due Sprint Date:: 19/Nov/24

Monitoring of processing scripts and workflows