Establish AAVS3 Pawsey storage data transfer

AAVS3 data needs to be stored offsite at Pawsey.  

The work carried on as part of SP-3407 should provide the basis for the organisation of metadata collected by the MCCS , and also for the setup of the Ceph storage for AAVS3. 

As part of this feature it should be analysed how data can programmatically be transferred to the Pawsey storage as part of the observation lifecycle. 

Data transfer should happen so that: 

• There is a mechanism in place to trigger transfer of data (could be based on an interface with MCCS or on some other convention)
• Data is transferred to the Pawsey storage storage automatically, together with metadata. Transfer is monitored and data integrity is verified.
• Data is purged from local MCCS storage upon successful transfer, according to a policy to be agreed with MCCS team. 
• A monitoring dashboard shows transfers in progress, and generates alerts upon errors. 

Desirable behaviours: 

• alerts are triggered upon data transfer error
• a mechanism is in place to trigger data indexing on the target platform upon ingest 
• a data dashboard can be used to browse data in the collection
• data transfer is robust to network disconnections and retries 

A baseline assumption and starting point for investigating an easy solution that solves AAVS3 needs, is to base this on Rclone , but more detailed investigation is needed. 

• There is a mechanism in place to trigger transfer of data (could be based on an interface with MCCS or on some other convention)
• Data is transferred to the Pawsey storage storage automatically, together with metadata. Transfer is monitored and data integrity is verified.
• Data is purged from local MCCS storage upon successful transfer, according to a policy to be agreed with MCCS team.
• A monitoring dashboard shows transfers in progress, and generates alerts upon errors.

• There is a mechanism in place to trigger transfer of data (could be based on an interface with MCCS or on some other convention)
• Data is transferred to the Pawsey storage storage automatically, together with metadata. Transfer is monitored and data integrity is verified.
• Data is purged from local MCCS storage upon successful transfer, according to a policy to be agreed with MCCS team.
• A monitoring dashboard shows transfers in progress, and generates alerts upon errors.