Data Management Storage Infrastructure

The DLM system will need to be integrated with several levels of distributed and heterogeneous storage. Potential technologies include flash based burst buffers, SSDs, HDs, and tapes as well as different access and redundancy mechanisms such as RAID, MAID, local and global block based storage with error encoding, (distributed) object storage as well as various tape management systems. In particular with tape management systems there is a significant overlap with the potential use of hierarchical storage management systems (HSM), such as Spectra Logic , Tivoli, DMF, Oracle SAM-QFS and Alluxio. Whatever the solution for the actual DLM system is, it will need to support the anticipated storage technologies and scale with the projected growth in data holdings. This feature addresses the storage infrastructure point of view, but in practice both of these sides have to be tackled in a co-design manner, while keeping the required roll-out of additional capacity on track with the demands of the two telescopes. During the construction period we will also see at least one significant step in capacity increase of available technologies and thus we will also need to start thinking about data migration. The planning needs to cover at least the whole construction period, since procurement and installation time-spans can be quite lengthy. It is also required to plan the whole supporting infrastructure for the storage as well, including network and processing connectivity, racks, power, redundancy storage and operational interfaces and procedures to ensure that incoming data can be ingested and the data is safe and available as required.