Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3022

CAR Caching proxies for upstream dependent artefacts for oci images

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Services
    • Hide

      In order to have a reliable, secured supply chain any artefact required by SKAO should be hosted within SKAO where we can access it safely, quickly and maintain its lifecycle. To achieve this we need to host both our own artefacts and any upstream artefacts we depend on. This is also in line with 7 R's of a supply chain (in the logistics world: Getting the Right product, in the Right quantity, in the Right condition, at the Right place, at the Right time, to the Right customer, at the Right price).

       

      To achieve a complete supply chain, it's important we first start with CAR as a first step and trickle down to each location as proxies and caches.

      Show
      In order to have a reliable, secured supply chain any artefact required by SKAO should be hosted within SKAO where we can access it safely, quickly and maintain its lifecycle. To achieve this we need to host both our own artefacts and any upstream artefacts we depend on. This is also in line with 7 R's of a supply chain (in the logistics world: Getting the Right product, in the Right quantity, in the Right condition, at the Right place, at the Right time, to the Right customer, at the Right price) .   To achieve a complete supply chain, it's important we first start with CAR as a first step and trickle down to each location as proxies and caches.
    • Hide
      • Upstream  OCI images are cached in noCAR by setting up proxies
        • noCAR: Seperate Nexus instance for OCI Proxy images into separate Nexus instances (behind oci-cache.artefact.skao.int or something similar)
      • CAR and proxies is connected with the appliances. i.e. 
        • playbooks are updated to use CAR as a proxy 
        • Servers are redeployed with the above config
      • Investigate Harbor(Harbor (goharbor.io) for sharding CAR for OCI Registry
        • What features it has for vetting
        • How does the distribution look like
      • Investigate how to secure the proxies (just an investigation on how it can be done, no implementation is needed at this point)
        • Purging packages
        • notary for OCI images
      Show
      Upstream  OCI images are cached in noCAR by setting up proxies noCAR: Seperate Nexus instance for OCI Proxy images into separate Nexus instances (behind oci-cache.artefact.skao.int or something similar) CAR and proxies is connected with the appliances. i.e.  playbooks are updated to use CAR as a proxy  Servers are redeployed with the above config Investigate Harbor( Harbor (goharbor.io) for sharding CAR for OCI Registry What features it has for vetting How does the distribution look like Investigate how to secure the proxies (just an investigation on how it can be done, no implementation is needed at this point) Purging packages notary for OCI images
    • 3
    • 3
    • 0
    • Team_IT, Team_SYSTEM
    • Sprint 4
    • Hide

      Ansible collections are updated to support caching proxies: 

      * ST-1457: [nexus] Clear variable hierarchy and update dependency on readme (!69) · Merge requests · SKAO / Software Defined Infrastructure / SKA Ansible Collections · GitLab

      Tests are added for the upstream pull scenario: https://gitlab.com/ska-telescope/sdi/ska-ser-ansible-collections/-/merge_requests/78 

      The relevant configs are updated in infra machinery for STFC clusters but the changes were not applied due to docker volume prune cronjob issue

       

      Show
      Ansible collections are updated to support caching proxies:  *  ST-1457: [nexus] Clear variable hierarchy and update dependency on readme (!69) · Merge requests · SKAO / Software Defined Infrastructure / SKA Ansible Collections · GitLab https://gitlab.com/ska-telescope/sdi/ska-ser-ansible-collections/-/merge_requests/73 https://gitlab.com/ska-telescope/sdi/ska-ser-ansible-collections/-/merge_requests/77   Tests are added for the upstream pull scenario: https://gitlab.com/ska-telescope/sdi/ska-ser-ansible-collections/-/merge_requests/78   The relevant configs are updated in infra machinery for STFC clusters but the changes were not applied due to docker volume prune cronjob issue  
    • 17.6
    • Stories Completed, Outcomes Reviewed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI24 - UNCOVERED

    • Team_IT Team_SYSTEM

    Description

      The SKAO provides a Central Artefact Repository based on Nexus Repository Manager3.  This has built in capabilities for caching from upstream sources for most artefact types (formats) that it can support.  The key SKAO published artefacts are OCI Images, Python libraries, Conan packages, Helm charts, and Raw artefact types. 

      The SKAO will also provide mirroring of upstream official artefact repositories - these include OCI Images (docker.io, quay.io, gcr.io), Apt (Ubuntu), Helm (public, Ceph, GitHub, GitLab), PyPi.

      These two caches will provide an authoritative source and logistics support for SKAO related artefacts.  The PoP Nexus instance points to these to provide local caching services for any software deployments and will be integrated with deployment and software building processes in the PoP location - eg: Kubernetes, containers, Docker, Podman etc.

      The caching facility will help with the speed of deployment issues in geographically dispersed locations, and the cache can be primed to support use cases where it is necessary to avoid the first-hit latency problem.

      Attachments

        Issue Links

          is cloned by

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3099 CAR Caching proxies for upstream dependent artefacts for python packages

          • Low - Minor problem or easily worked around.
          • Analyzing
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                P.Harding Harding, Piers
                U.Yilmaz Yilmaz, Ugur
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 3.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1123.0
                  Total1123.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel