Workshop: Building reproducible workflows for earth sciences

In 2012 the New Zealand (NZ) MetService started on a journey to move its NWP modelling and other automated processing to Amazon Cloud computing. Sitting on multiple earthquake faults with a data centre that has limited abilities to increase capacity and resilience against external outages, it left the organisation to make a revolutionary change compared with the past: not to own a data centre any more for weather modelling. Although mainly driven by the requirement of a resilient computing environment, the preparation for the Cloud made many more benefits apparent if only a Cloud infrastructure solution would be designed appropriately.

The main benefits we aimed for were
- high resilience of infrastructure by combining multiple AWS regions in a seamless environment.
- change towards scientists adopting professional software development practices and establishing an extremely robust release process.
- cost effectiveness by using spot market instance prices for operations and research.
- "self-healing" workflows that can guarantee automatic completion against all hardware failures.
- scientists don't wait for data any more, they analyse data,
- much clearer cost attribution towards applications and services.

This presentation will touch on a number of aspects that were encountered on this journey of change which impacted on people, financials, accountabilities, and the mindset of solving science and infrastructure problems in the Cloud. It has been an interesting time with a few surprises along the way but the general consensus it: let's not go back to the way it was.

Having said all this, not all systems and data can be moved to the Cloud leaving the NZ MetService to operate in a hybrid environment. The requirements for delivering data to NZ aviation with an availability beyond what the Southern Cross cable can provide as well as what the NZ privacy law asks for means that some data needs to be hosted within NZ redundantly. Some of the challenges that this poses will be presented as well.