Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3773

Testing Cheetah SPS modules with PSS test vectors

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Should have
    • PI20
    • COM PSS SW
    • None
    • Data Processing
    • Hide

      Improvements to the efficacy of the SPS pipeline using results from the PSS product testing framework, ProTest. Improvements to the SPS candidate metdata verification modules in ProTest. 

      Show
      Improvements to the efficacy of the SPS pipeline using results from the PSS product testing framework, ProTest. Improvements to the SPS candidate metdata verification modules in ProTest. 
    • Hide

      A document describing the outcome of a set of tests of the SPS pipelines using newly implemented tolerances and a range of test vectors

      Completion of pandas training

      Show
      A document describing the outcome of a set of tests of the SPS pipelines using newly implemented tolerances and a range of test vectors Completion of pandas training
    • 4
    • 4
    • 0
    • Team_PSS
    • Sprint 5
    • Hide

      The Klotski implementation of SPS has been fixed for the bugs found in earlier PIs (bug tickets are linked to this feature).
      MR including fixes: https://gitlab.com/ska-telescope/pss/ska-pss-cheetah/-/merge_requests/43

      We ran all test vectors in the ProTest SPS feature through the SPS code included in that MR, and plots of the single pulse detections are collected in this document:
      https://docs.google.com/document/d/1PeSS5OrMvkvSMvijnn_osg-zk71WET5hd_dH6TRF57I/edit?usp=sharing

      We have also turned on the previously defined single pulse tolerances in ProTest and checked how many pulses are now detected by ProTest. The results are written up in this document:
      https://docs.google.com/document/d/1SXMo4OZFqfcq3A2N8IyGOvzPCDDg4g7Xs-ihnmg4FHQ/edit?usp=sharing

      In summary, some of the pulses that ProTest expects are found in SPS outside of the tolerance values: ProTest detects 122 of the 136 pulses we passed through SPS. We believe that the SPS code is now all correct, but some of the tolerance values are too restrictive, in particular the DM tolerance. We will look into further refining the tolerances in future PIs.

      Ben Shaw also completed some training on pandas covering the basics of series, data frames, and most importantly, conditional filtering (which will allow us to instantly exclude all the candidates which fall outside our tolerances values, on any field). Also covered, various methods for performing statistical analysis on the data, and well as how to handle missing or null vales. We think this will be very useful for ProTest development in the future.

      Show
      The Klotski implementation of SPS has been fixed for the bugs found in earlier PIs (bug tickets are linked to this feature). MR including fixes: https://gitlab.com/ska-telescope/pss/ska-pss-cheetah/-/merge_requests/43 We ran all test vectors in the ProTest SPS feature through the SPS code included in that MR, and plots of the single pulse detections are collected in this document: https://docs.google.com/document/d/1PeSS5OrMvkvSMvijnn_osg-zk71WET5hd_dH6TRF57I/edit?usp=sharing We have also turned on the previously defined single pulse tolerances in ProTest and checked how many pulses are now detected by ProTest. The results are written up in this document: https://docs.google.com/document/d/1SXMo4OZFqfcq3A2N8IyGOvzPCDDg4g7Xs-ihnmg4FHQ/edit?usp=sharing In summary, some of the pulses that ProTest expects are found in SPS outside of the tolerance values: ProTest detects 122 of the 136 pulses we passed through SPS. We believe that the SPS code is now all correct, but some of the tolerance values are too restrictive, in particular the DM tolerance. We will look into further refining the tolerances in future PIs. Ben Shaw also completed some training on pandas covering the basics of series, data frames, and most importantly, conditional filtering (which will allow us to instantly exclude all the candidates which fall outside our tolerances values, on any field). Also covered, various methods for performing statistical analysis on the data, and well as how to handle missing or null vales. We think this will be very useful for ProTest development in the future.
    • 20.6
    • Stories Completed, Outcomes Reviewed, Satisfies Acceptance Criteria, Accepted by FO
    • PI22 - UNCOVERED

    Description

      This feature focuses on improvement of the Klotski implementations of the SPS pipeline through a series of tests combined with further development of the SPS in order to resolve issues previously flagged up by ProTest. These tests will be carried out both manually (i.e., executing the SPS pipeline on the command line directly) and by ProTest. Will we also improve ProTest's SPS verification modules. As part of this work, we will

      • Evaluate the output of the Klotski SPS pipeline using a wider range of test vectors that cover areas of parameter space not previously tested, such as signal-to-noise ratio (S/N) and the number of injected pulses, and document any bugs or failures that are identified. 
      • Update existing ProTest tests of the SPS pipeline to validate candidates using our recently developed tolerances
      • Investigate how/if pandas can improve ProTest's candidate metadata verification

      Attachments

        Structure

          Activity

            People

              A.Noutsos Noutsos, Aristeidis
              L.Levin-Preston Levin-Preston, Lina
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Feature Progress

                Story Point Burn-up: (100.00%)

                Feature Estimate: 4.0

                IssuesStory Points
                To Do00.0
                In Progress   00.0
                Complete930.0
                Total930.0

                Dates

                  Created:
                  Updated:
                  Resolved:

                  Structure Helper Panel