Prototype an Execution broker within CANFAR Science platform

Context

This prototype focuses on the intermediation of services of the SKA Regional Centres Network (SRCNet), which encompasses multiple computing resources and distributed services. This distributed environment requires efficient resource management and allocation strategies to meet the complex computing needs of SRCNet users. Within this context, the objective is to deploy a prototype for IVOA (International Virtual Observatory Alliance) Execution Broker that is partially completed (see link). This Execution Broker will serve as a component to be considered within the SRCNet ecosystem, facilitating the communication between client services and the available computational resources, even enabling in future stages, a resource intelligence layer.

Clients can submit queries to the Execution Broker, requesting information on resource availability or specific computational tasks. The Execution Broker will then evaluate these queries, determining their feasibility and providing options if the requested tasks are feasible. 

Description

The aim of this feature is to prototype an IVOA Execution Broker by using one of the backend-framework, currently deployed in several SRCs, with CADC CANFAR Science platform (deployed on kubernetes). This Execution Broker will act as an intermediary layer, enabling clients (services) within the SRCNet to interact with the available resources seamlessly. Clients can submit queries to the Execution Broker, requesting information on resource availability or specific computational tasks. The Execution Broker will then evaluate these queries, determining their feasibility and providing options if the requested tasks are feasible.  In this feature, we will create a web service to implement the data model and web service methods that are described by the IVOA Execution Broker standard. This web service will sit in front of the execution platform ( which could be CANFAR or slurm cluster or etc., but for this feature we will start with CANFAR). So we will need to enable the execution broker to understand the "capabilities" of CANFAR. The web service will receive the request from the user ( I need to run X) and the web service will check the "capabilities" of the available execution platforms (only CANFAR in this feature), so it will be able to map the user request with the available platforms.  The deliverable from this  will be a repository with the code of the web service and documentation. In this feature we will implement the first two versions of the Execution broker as described below:

Version 1

UR: "Can you run this container?" --> The EB will check if the container is in the computing platform capabilities

Version 2

UR: "Can you run this container with this DATA?" --> The EB will check if the data is in the local Rucio node. If the data is local the EB will answer  "yes we can do this now". If the data is no local, the answer will be "NO".  --> For this feature.

Version 3

UR: "Can you run this container with this DATA?" --> The EB will check if the data is in the local rucio node. If the data is local the EB will answer  "yes we can do this now". If the data is no local, the answer will be  "We can do this in X minutes" ( This implies an interaction with Rucio to stage a copy of the data, etc. This version is not for this feature)

The code for the ExecutionPlanner will be developed using a modular design, building the service from a set of plug-in components that can be replaced or extended to enable ExecutionBroker to interact with different types of storage and compute platforms. In the first case, the storage component will interact with the local Rucio node and the compute component will interact with a local CANFAR instance. The plug-in architecture will enable future implementations to replace one or other of these components with a different one, e.g.  a local S3 storage system or a Kubernetes execution platform. The exact shape of the plugin-interfaces will evolve to fit the relevant parts of the data model as we develop the main body of the ExecutionPlanner service implementation.