Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4241

Prototype an Execution broker within CANFAR Science platform

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      The implementation of the IVOA Execution Broker within the SRCNet offers several potential benefits. Firstly, it provides users with an interface for querying and accessing computational resources, streamlining the resource management and allocation process. This approach is expected to enhance efficiency and reduce downtime by optimizing resource utilization across the SRCNet. Additionally, the prototype serves as a proof of concept for integrating IVOA Execution Broker standards within the SRCNet, demonstrating the feasibility of adopting standardized protocols to interoperate services. Furthermore, the APIs exposed by the Execution Broker and Execution Worker components enable seamless integration with existing tools and workflows, enhancing the overall data-computing mesh of the SRCNet environment.

      Show
      The implementation of the IVOA Execution Broker within the SRCNet offers several potential benefits. Firstly, it provides users with an interface for querying and accessing computational resources, streamlining the resource management and allocation process. This approach is expected to enhance efficiency and reduce downtime by optimizing resource utilization across the SRCNet. Additionally, the prototype serves as a proof of concept for integrating IVOA Execution Broker standards within the SRCNet, demonstrating the feasibility of adopting standardized protocols to interoperate services. Furthermore, the APIs exposed by the Execution Broker and Execution Worker components enable seamless integration with existing tools and workflows, enhancing the overall data-computing mesh of the SRCNet environment.
    • Hide

      AC1: A repository with the source code, documentation and deployment process of a web-service to implement the data model and methods that are described by the IVOA Execution Broker standard. This web-service will answer to a client if it can run a specific container, and if a client can run a specific container with a specific data. CANFAR will be used as backend.

      AC2:  A repository with an OpenAPI machine readable specification based on a JSON file describing the web-service.

      AC3: Update the repository/documentation of the  IVOA Execution Broker standard with the changes or improvement learned from the development of AC1 and AC2. 

      AC4: A demo presenting how the broker interoperate. This demo will include the work with the repository, the web-services implementation, the data-model, JSON specification and the changes applied to the  IVOA Execution Broker.

      Show
      AC1: A repository with the source code, documentation and deployment process of a web-service to implement the data model and methods that are described by the IVOA Execution Broker standard. This web-service will answer to a client if it can run a specific container, and if a client can run a specific container with a specific data. CANFAR will be used as backend. AC2:   A repository with an OpenAPI machine readable specification based on a JSON file describing the web-service. AC3:  Update the repository/documentation of the  IVOA Execution Broker standard with the changes or improvement learned from the development of AC1 and AC2.  AC4: A demo presenting how the broker interoperate. This demo will include the work with the repository, the web-services implementation, the data-model, JSON specification and the changes applied to the   IVOA Execution Broker .
    • Intra Program
    • Team_CORAL, Team_DAAC
    • Sprint 5

    • PI23 - UNCOVERED

    • SRCNet0.x science-platform-services service-discovery-service

    Description

      Context

      This prototype focuses on the intermediation of services of the SKA Regional Centres Network (SRCNet), which encompasses multiple computing resources and distributed services. This distributed environment requires efficient resource management and allocation strategies to meet the complex computing needs of SRCNet users. Within this context, the objective is to deploy a prototype for IVOA (International Virtual Observatory Alliance) Execution Broker that is partially completed (see link). This Execution Broker will serve as a component to be considered within the SRCNet ecosystem, facilitating the communication between client services and the available computational resources, even enabling in future stages, a resource intelligence layer.

      Clients can submit queries to the Execution Broker, requesting information on resource availability or specific computational tasks. The Execution Broker will then evaluate these queries, determining their feasibility and providing options if the requested tasks are feasible. 

       

      Description

      The aim of this feature is to prototype an IVOA Execution Broker by using one of the backend-framework, currently deployed in several SRCs, with CADC CANFAR Science platform (deployed on kubernetes). This Execution Broker will act as an intermediary layer, enabling clients (services) within the SRCNet to interact with the available resources seamlessly. Clients can submit queries to the Execution Broker, requesting information on resource availability or specific computational tasks. The Execution Broker will then evaluate these queries, determining their feasibility and providing options if the requested tasks are feasible.  In this feature, we will create a web service to implement the data model and web service methods that are described by the IVOA Execution Broker standard. This web service will sit in front of the execution platform ( which could be CANFAR or slurm cluster or etc., but for this feature we will start with CANFAR). So we will need to enable the execution broker to understand the "capabilities" of CANFAR. The web service will receive the request from the user ( I need to run X) and the web service will check the "capabilities" of the available execution platforms (only CANFAR in this feature), so it will be able to map the user request with the available platforms.  The deliverable from this  will be a repository with the code of the web service and documentation. In this feature we will implement the first two versions of the Execution broker as described below:

      Version 1

      UR: "Can you run this container?" --> The EB will check if the container is in the computing platform capabilities

      Version 2

      UR: "Can you run this container with this DATA?" --> The EB will check if the data is in the local Rucio node. If the data is local the EB will answer  "yes we can do this now". If the data is no local, the answer will be "NO".  --> For this feature.

      Version 3

      UR: "Can you run this container with this DATA?" --> The EB will check if the data is in the local rucio node. If the data is local the EB will answer  "yes we can do this now". If the data is no local, the answer will be  "We can do this in X minutes" ( This implies an interaction with Rucio to stage a copy of the data, etc. This version is not for this feature)

      The code for the ExecutionPlanner will be developed using a modular design, building the service from a set of plug-in components that can be replaced or extended to enable ExecutionBroker to interact with different types of storage and compute platforms. In the first case, the storage component will interact with the local Rucio node and the compute component will interact with a local CANFAR instance. The plug-in architecture will enable future implementations to replace one or other of these components with a different one, e.g.  a local S3 storage system or a Kubernetes execution platform. The exact shape of the plugin-interfaces will evolve to fit the relevant parts of the data model as we develop the main body of the ExecutionPlanner service implementation.

      Attachments

        Issue Links

          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3884 CANFAR Science Platform User Container CI/CD

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Program Backlog
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                Jesus.Salgado Salgado, Jesus
                D.Morris Morris, Dave
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (29.41%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do24.0
                  In Progress   18.0
                  Complete15.0
                  Total417.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel