Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-759

SRC (ESCAPE Project) "Datalake"

    XporterXMLWordPrintable

Details

    • Epic
    • None
    • None
    • None
    • SRC Datalake
    • Data Processing
    • 0

    Description

       

      Epic Hypothesis Statement (https://www.scaledagileframework.com/epic/)

      For EC and global SRC stakeholders
      who want to develop or use a science analysis platform sharing data across global links
      the ESCAPE project "datalake"
      is a federated data management service prototype
      that manages data objects within incorporated storage systems within Europe and inter-continental locations
      Unlike Dropbox
      our solution
      • Gives us access to a system of data storage resources which have been (or are being) included into the ESCAPE Rucio instance.
      • Have enabled the ESCAPE Virtual Organisation which operates using the ESCAPE identity management system and which will participate in network mesh monitoring and point-to-point data transfer tests.
      • The ESCAPE prototypes the use of free and open-source software (FOSS) across these domains and enable us to benefit from the expertise of the team (at ESCAPE partner institutes) who have developed or are expert users of these tools.
         
      Outcome hypothesis
      • Tested datalake (or "datamesh") of storage systems across Europe, and at least Australia & South Africa
      • EC Deliverable documents: Datalake performance results, protocol recommendations
      • In-house expertise, documentation and software to enable SKA to continue this work after the ESCAPE project
      Leading Indicators Is the ESCAPE datalake up-and-running? - i.e. can we:
          1. See and manage a common data object catalogue across the instance.
          2. Manage transfers of files that can scale to appropriate sizes across links within Europe and intercontinentally.
          3. See what development work would be required to make this scale to SKA full scale.
          4. Are the sites stable - and what does this tell us about the requirements on individual SRCs
          5. Are the network links reliable and fast enough / as fast as expected?
      Do we have the engagement of the global SRC community in this endeavour?
      NFRs  

       

      Attachments

        Issue Links

          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-758 SRC (ESCAPE project) Science Analysis Platform

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Discarded

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-613 SKA Regional Centre Science Analysis Platform and Data-lake Prototypes

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Structure

            Activity

              People

                n.rees Rees, Nick
                r.brederode Brederode, Ray
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  Structure Helper Panel