Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4764

Determine the checksum algorithm for Datalake integrity verfication

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      By assessing and selecting an optimal checksum algorithm for SRCNet Data storage, Rucio will enhance file integrity verification through the adoption of an algorithm.
      This will improve performance, reduce computational overhead, and provide easier interoperability with other systems that are required to verify file integrity.
      Timely evaluation and implementation will ensure a smooth transition and avoid potential complications or delays in future adoption.

      Show
      By assessing and selecting an optimal checksum algorithm for SRCNet Data storage, Rucio will enhance file integrity verification through the adoption of an algorithm. This will improve performance, reduce computational overhead, and provide easier interoperability with other systems that are required to verify file integrity. Timely evaluation and implementation will ensure a smooth transition and avoid potential complications or delays in future adoption.
    • Hide

      AC1: Review of existing algorithms with a comparative analysis

      AC2: Identify algorithm of choice

      AC3: Define the work required to implement algorithm choice in rucio

      AC4: Provide a recommendation document for Architecture team and other Stakeholders. 

      Show
      AC1: Review of existing algorithms with a comparative analysis AC2: Identify algorithm of choice AC3: Define the work required to implement algorithm choice in rucio AC4: Provide a recommendation document for Architecture team and other Stakeholders. 
    • 0.5
    • 0
    • Team_PURPLE

    Description

      The Rucio software by default uses the ADLER32 checksum algorithm to verify the integrity of files within the Rucio 'datalake'. While this algorithm has been successful, it is typically not as well supported as other algorithms (e.g. CRC32C) which can benefit from Hardware-based computation, or already be available from within storage systems (e.g. Ceph).
      As assessment of the optimal checksum algorithm for SRCNet Data storage should be performed and a recommendation be presented to be implemented / adopted within Rucio.
      Delaying this for too long may prevent an easy transition (should one be needed). 

      Attachments

        Issue Links

          Child Of

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-4724 Data Management v0.2 - Roadmap

          • Not Assigned - Priority is not assigned yet.
          • Funnel

          Structure

            Activity

              People

                b.mort Mort, Ben
                J.Walder Walder, James
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.5

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel