Simulation and precursor data sets for SDP AA2 pipeline testing

The SDP Pipeline development throughout the construction phase will need a consistent, easily discoverable and available set of test data for regularly/continually assessing scaling and science performance as the pipelines evolve. 

This needs to be a solution that can be built upon and extended alongside pipeline development and can evolve to meet the needs of the current state of pipelines which are anticipated to evolve in the complexity of the algorithm and size of the data sets being tested with over time.

To get to such a solution we should start with already existing data sets from simulations and precursors and extend that as new data sets are produced or identified.

1. Identification of data sets to provide access to by the implemented solution further specified below and assessment of the need for future test data sets for AA2 SDP pipeline development.
2. Download capability using basic tools, like e.g. rsync or rclone.
3. Documented capability of how to add additional data sets. 

Partial agreement on the necessary requirements, policies and interface capabilities was achieved with stakeholders during Processing CoP meetings and demoes on  2022-10-03 and 2022-10-10. The limiting implications on outcomes are discussed in the Resolution column of the first table here :

• AC1 Initial Test Data Sets
• AC2 Download Capability
• AC3 Upload Capability