Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4060

Inialise Higher Performance Networking Inside OpenStack Environments

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      We believe that, for workloads that require low latency networking to help them scale out between multiple nodes on sites running Kubernetes VMs in OpenStack, a more performant solution exists in the form of the latest SR-IOV option, using OVS offloads.

      To test this, we will apply the latest SR-IOV option, using OVS offloads, on the Cambridge hardware. (This has already been tested with Azimuth at non-SRCNet sites).

      Other OpenStack sites can review the write up of this work, and collaborate to help document a repeatable way to apply this at other OpenStack sites.

      Show
      We believe that, for workloads that require low latency networking to help them scale out between multiple nodes on sites running Kubernetes VMs in OpenStack, a more performant solution exists in the form of the latest SR-IOV option, using OVS offloads. To test this, we will apply the latest SR-IOV option, using OVS offloads, on the Cambridge hardware. (This has already been tested with Azimuth at non-SRCNet sites). Other OpenStack sites can review the write up of this work, and collaborate to help document a repeatable way to apply this at other OpenStack sites.
    • Hide

      AC1: It can be demonstrated that a workload requiring low latency networking is more performant when running SR-IOV / OVS.

      AC2: Results will be peer-reviewed by another OpenStack site

      AC3: Documentation describing how this change can be made at other OpenStack sites will be accepted by the HPC/Cloud COP.

      Show
      AC1: It can be demonstrated that a workload requiring low latency networking is more performant when running SR-IOV / OVS. AC2: Results will be peer-reviewed by another OpenStack site AC3: Documentation describing how this change can be made at other OpenStack sites will be accepted by the HPC/Cloud COP.
    • 2
    • 3
    • 0
    • Team_TEAL
    • Sprint 5
    • Hide

      Feature carried over to PI23 - cloned into SP-4393 .
      (To credit that the majority of this feature has been done).

      See SP-4393 for the final outcome.

      Show
      Feature carried over to PI23 - cloned into SP-4393 . (To credit that the majority of this feature has been done). See SP-4393 for the final outcome.
    • 22.6
    • Accepted by FO

    • PI23 - UNCOVERED

    • Teal-D site-provisioning team_DAAC

    Description

      This is urgent, because it will impact the hardware that needs to be precured for SRCNet v0.1 sites that want to run Kubernetes VMs in OpenStack.

      Some workloads require low latency networking to help them scale out between multiple nodes e.g. Dask UCX, MPI, Nvidia cuFFTMp, etc. Some storage systems need RDMA, to work efficiently.

      While it is possible to enable ethernet RDMA networking within OpenStack VMs, and K8s clusters running in both OpenStack VMs and OpenStack baremetal nodes, this is not well understood, and it is hardware dependent.

      Cambridge Arcus OpenStack cloud (on which CSD3 and Dawn are both running), currently has limited support for RDMA network. We aim to apply the latest SR-IOV option, using OVS offloads, on the hardware where that is possible. Hopefully other OpenStack sites can help review the write up of this work, and collaborate to help document a repeatable way to apply this at other OpenStack sites. It would be helpful to engage others, likely through the HPC and Cloud CoP, to review this write up.

      To help measure the success of the setup within K8s VMs, StackHPC have already worked worth Mellanox to test RDMA within Kubernetes using these tests, which can be reused for this work:
      https://github.com/stackhpc/kube-perftest
      https://www.stackhpc.com/k8s-rdma-openstack.html

      Note there is a follow-up piece of work planned, where we look at evacuating the current state of Infini-band within OpenStack VMs. There have been many changes to UFM since this work was first done as part of the AlaSKA effort. Much of the infrastructure targeting HPC and AI at Cambridge is focused on InfiniBand, so this is critical to get access to that hardware without having to re-wire those nodes with some high speed ethernet, when resources are moved between OpenStack VMs and Dawn/CSD3. This all assumes that we do not want to give untrusted users access to baremetal hardware, due to the risks around securing the firmware on those servers. VMs have been shown to have limited overheads, and be much easier to operate, when correctly configured.

      Attachments

        Issue Links

          is cloned by

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4393 Conclude Higher Performance Networking Inside OpenStack Environments

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Program Backlog
          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                D.Watson Watson, Duncan
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete12.0
                  Total12.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel