Workshop: Building reproducible workflows for earth sciences

ECMWF operational forecast generates massive amounts of I/O in short bursts, accumulating to tens of TiB in hourly windows. From this output, millions of user-defined daily products are generated and disseminated to member states and commercial clients all over the world. These products are processed from the raw output of the IFS model, within the time critical path and under strict delivery schedule. Upcoming rise in resolution and growing popularity will increase both the size and number of these products.
In 2019, ECMWF has brought into operations a completely redesigned Product Generation software stack that computes the requested user products. To introduce the new software swiftly into operations, minimal user impact was a design criteria, as well as increased robustness and reliability.
We will present the lessons learned, technical decisions and the steps taken to bring this massive operational change successfully into completion.