Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4514

DataStaging broker

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      Without a cost estimate for the data transfers it is difficult to give a reliable quote on how long it will take to setup the environment ready to execute a task on a given set of input data.

      The task itself may take less than an hour to execute, but staging the data to the compute platform could be instant if a copy of the data is already there, or take several hours if the system needs to transfer a large quantity of data across a long distance before the task can start.

      Offering an interactive notebook session starting "now +/- 6hrs" is not provide a good user experience.

      If we can generate reliable estimates of transfer costs, then we can provide better information to the user enabling them to evaluate which offer best meets their requirements.

      "You can use this execution platform now, with 2 cores and 2G of memory, or wait 2 hours and use a larger platform with 200 cores and 200G of memory."

      Show
      Without a cost estimate for the data transfers it is difficult to give a reliable quote on how long it will take to setup the environment ready to execute a task on a given set of input data. The task itself may take less than an hour to execute, but staging the data to the compute platform could be instant if a copy of the data is already there, or take several hours if the system needs to transfer a large quantity of data across a long distance before the task can start. Offering an interactive notebook session starting "now +/- 6hrs" is not provide a good user experience. If we can generate reliable estimates of transfer costs, then we can provide better information to the user enabling them to evaluate which offer best meets their requirements. "You can use this execution platform now, with 2 cores and 2G of memory, or wait 2 hours and use a larger platform with 200 cores and 200G of memory."
    • Hide

      ToDo - New ACs - outcomes regarding design and what requirements the execution broker / data management system have i.e. what data they need to collect and learn from to arrive at a sensible decision how to move the data around...

       

      AC1: A demo showing a prototype service that can estimate the time cost of transferring a data product to a compute platform. 

       

       

      Show
      ToDo - New ACs - outcomes regarding design and what requirements the execution broker / data management system have i.e. what data they need to collect and learn from to arrive at a sensible decision how to move the data around...   AC1: A demo showing a prototype service that can estimate the time cost of transferring a data product to a compute platform.      

    Description

      As a result of our work on ExecutionBroker we have identified a critical part of offering time slots for execution tasks will involve predicting how long it will take to stage the required input data at the target platform, and staging the output results back into permanent storage. Our experience suggests that this functionality could be  best implemented as a separate component, 'used by' rather than 'contained by' ExecutionBroker.

      The primary function of the interface would be to predict the cost of moving data to and from a compute platform, without having to move the data.

      In order to make an offer for executing a given task with a given set of input data, Executionbroker needs to find out how long it will take to stage the data from its current location to the target compute platform in preparation for executing the task.

      In simple terms, "how long would it take to move <this> data to <this> site".

      Attachments

        Issue Links

          Child Of

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-4724 Data Management v0.2 - Roadmap

          • Not Assigned - Priority is not assigned yet.
          • Funnel
          mentioned in

          Page Loading...

          Structure

            Activity

              People

                b.mort Mort, Ben
                S.SanchezExposito Sanchez Exposito, Susana
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel