Details
-
Feature
-
Must have
-
SRCnet
-
-
-
1
-
2
-
0
-
Team_MAGENTA
-
Sprint 5
-
-
-
-
22.6
-
Stories Completed, Outcomes Reviewed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
-
-
data-ingestion-dissemination-and-replication
Description
By default Rucio uses "deterministic" Rucio Storage Elements (RSEs). All but one of the RSEs in the prototype Rucio datalake are deterministic, but there exists another option, "non-deterministic", which may be more a appropriate type to use in some SRCNet context, especially data ingest.
For deterministic RSEs, Rucio handles translation between a logical file name (LFN) to physical file name (PFN). This is the mapping between the internal data identifier used by Rucio and the replica path on storage. In this mode, to ingest data one would have to both upload the data (so that Rucio can generate this mapping) and register it (flag it as belonging to the datalake). The upload step means that the data is duplicated.
For "non-deterministic" RSEs, the operator tells Rucio the mapping between the LFN and the PFN. This way, data that already exists can be ingested by simply registering it; no upload (and corresponding data duplication) is required.
Currently the ingestion service (for Rucio) use deterministic RSEs.