Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3232

An baseline implementation for SKA FaaS / Serverless approach

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      Having implemented clusters with Kubernetes, container orchestration and a federation of services at Kubernetes level with KubeFed/Karmada, we thought that the next logical step would be to provide a platform for FaaS, such as OpenFaaS or OpenWisk, on top of Kubernetes/KubeFed that allows to include a subset of functions (from radioastronomy pipelines, such as cleaning, flagging, . ...) that can be used from a python package (or from any other language, for example ska-science-tools) and used in a JupyterNotebook, from source code or from another service (code agnostic). These functions, as they run inside Kubernetes (or KubFed) can be (virtually) infinitely scalable, provide HA, can be distributed or geolocated, run on slurm, and can have brokering capabilities. With it, we can provide users with scientific tools at scale.

      Show
      Having implemented clusters with Kubernetes, container orchestration and a federation of services at Kubernetes level with KubeFed/Karmada, we thought that the next logical step would be to provide a platform for FaaS, such as OpenFaaS or OpenWisk, on top of Kubernetes/KubeFed that allows to include a subset of functions (from radioastronomy pipelines, such as cleaning, flagging, . ...) that can be used from a python package (or from any other language, for example ska-science-tools) and used in a JupyterNotebook, from source code or from another service (code agnostic). These functions, as they run inside Kubernetes (or KubFed) can be (virtually) infinitely scalable, provide HA, can be distributed or geolocated, run on slurm, and can have brokering capabilities. With it, we can provide users with scientific tools at scale.
    • Hide

      AC1: To have a serverless platform ready on top of k8s.
      AC2: To provide a minimum catalogue of functions (one per calibrarion, cleaning, flagging) to interact with serverless (i.e. with from JupyterNotebooks (or code/other service))

      Show
      AC1: To have a serverless platform ready on top of k8s. AC2: To provide a minimum catalogue of functions (one per calibrarion, cleaning, flagging) to interact with serverless (i.e. with from JupyterNotebooks (or code/other service))
    • PI24 - UNCOVERED

    • SRC-CompPlat

    Description

      Our idea with this approach is to unify the knowledge and development done in previous PIs on containers, orchestration and federation at K8s level to test a FaaS/Serverless platform that allows to run scalable functions through Kubernetes (local to the SRC or distributed in other locations). The following diagram shows the components of the implementation:

      Applications and function code of the pipeline can be easily containerised, as well as adding a proxy-sidecar for monitoring etc.

      Attachments

        Issue Links

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                M.Parra Parra, Manuel
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel