Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4291

Upload intermediate/derived data products back into Rucio for convolution task

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      The SRC workloads Gitlab repository hosts workloads that are representative of what users will be running at SRC sites. Demonstrating that tasks can upload data back into Rucio during or after the completion of a task will show further capabilities of Rucio and how workloads interact with databases. 

      Show
      The SRC workloads Gitlab repository hosts workloads that are representative of what users will be running at SRC sites. Demonstrating that tasks can upload data back into Rucio during or after the completion of a task will show further capabilities of Rucio and how workloads interact with databases. 
    • Hide
      • Run the convolution task on LOTSS data to create images at different resolutions
      • Upload those images back into Rucio automatically after the task completes (ideally automated auth-less)
      • Include an option in the code that allows users to decide if output data should be re-uploaded.
      • Merge request the update of the convolution task and video demo. 
      Show
      Run the convolution task on LOTSS data to create images at different resolutions Upload those images back into Rucio automatically after the task completes (ideally automated auth-less) Include an option in the code that allows users to decide if output data should be re-uploaded. Merge request the update of the convolution task and video demo. 
    • 1
    • 1
    • 0
    • PI23 - UNCOVERED

    • SRCNet0.x example-workflows-and-benchmarks tests-compilation

    Description

      Demonstrating tasks that interact with the Rucio database will be important. Currently there aren't any tasks that create intermediate data products, so this would have to be only uploading final data products back to Rucio. 

      The convolution task would be an obvious choice. The benefit of this is that we could upload images back to Rucio at different resolutions after convolution which would be very useful.

      An option is to upload catalogues from source finding or mosaics from mosaicking. However, there is current no use for catalogues or mosaics in our task, so that limits further processing needs. 

      Uploading data back to Rucio will require authentication, meaning this is not an automated step. It would also be beneficial to do this once we have auth-less upload options, particularly in prep for longer automated workflows where data is moved back and forth.

      Attachments

        Issue Links

          Structure

            Activity

              People

                A.Clarke Clarke, Alex
                A.Clarke Clarke, Alex
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 1.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel