Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4273

CI/CD infrastructure for compiled components of SDP pipelines

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Must have
    • PI23
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      We are able to deploy SDP pipelines in a repeatable way from know source code. This is essential for:

      • Making useful performance tests (because we know exactly what we are testing)
      • Improving the speed of development (because the build process is more automated and easier ot repeat)
      • Getting Sci/Ops involved in testing of pipelines (it is absolutely essential that we know what software users are testing!)

       

      Show
      We are able to deploy SDP pipelines in a repeatable way from know source code. This is essential for: Making useful performance tests (because we know exactly what we are testing) Improving the speed of development (because the build process is more automated and easier ot repeat) Getting Sci/Ops involved in testing of pipelines ( it is absolutely essential that we know what software users are testing! )  
    • Hide
      1. Spack recipes in git repositor(y/ies):
        1. For all components of Low/Mid ICal
        2. Also for compiled components used in pointing etc
      2. CI triggered build and deploy to CAR:
        1. Containers suitable for CSD3 & Cloud HPC
        2. Spack build cache
      3. CI triggered tests:
        1. Smoke
        2. Regression on small data
      Show
      Spack recipes in git repositor(y/ies): For all components of Low/Mid ICal Also for compiled components used in pointing etc CI triggered build and deploy to CAR: Containers suitable for CSD3 & Cloud HPC Spack build cache CI triggered tests: Smoke Regression on small data
    • Intra Program
    • -1
    • -1

    • PI22 - UNCOVERED

    Description

      Continuous build and test infrastructure of the SDP processing pipelines

      Builds & test binaries to be used for development, testing and deployment of SDP processing pipelines, in accordance with decisions of ADR-100.

       

      • Use Spack as build system
      • use CAR as Spack binary artefact cache
      • Build against ubuntu base images containers and upload containers to CAR 
      • Multiple builds to support the relevant instruction sets we use (currently we regularly use from Xeons form '14 onward + some AMD)

      The scope should include:

      • all astronomy specific software (casacore, fitsio, etc)
      • also, performance critical general software which can benefit from maximum optimisation / instruction set.
      • to include all compiled software e.g. what we would use a binary python wheels.

      Automated run of :

      • Basic smoke tests 
      • Regression tests

      Make sure deployable to:

      • CSD3 
      • New Cloud based HPC platform

       

      https://confluence.skatelescope.org/display/SWSI/ADR-100+Continuous+Integration+and+Deployment+of+HPC+software

       

      Attachments

        Issue Links

          clones

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-3666 CI/CD infrastructure for testing pipelines

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Implementing

          Structure

            Activity

              People

                b.nikolic Nikolic, Bojan
                D.Fenech Fenech, Danielle
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: -1.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel