19th Workshop on high performance computing in meteorology

Particle filters are a major tool used for data assimilation (DA) in climate modeling. The ability to handle a very large number of particles, is critical for high dimensional climate models. The presented approach introduces a novel way of running such DA studies for the example of a particle filter using sequential important resampling (SIR). The new approach executes efficient on latest high performance computing platforms. It is resilient to numerical and hardware faults while minimizing data movements and enabling dynamic load balancing. Multiple particle propagations are performed per running simulation instance, i.e., runner, each assimilation cycle. Particle weights are computed locally on each of these runners and transmitted to a central server that normalizes them, resamples new particles based on their weight, and redistributes the work to runners one by one to react to load imbalance. Our approach leverages the multi-level checkpointing library FTI, permitting particles to move from one runner to another in the background while particle propagation goes on. This also enables the number of runners to vary during the execution either in reaction to failures and restarts, or to adapt to changing resource availability dictated by external decision processes. The approach is experimented with the Weather Research and Forecasting (WRF) model, to assess its performance for probabilistic weather forecasting. Multiple thousand particles on more than 20000 compute cores are used to assimilate cloud cover observations into short-range weather forecasts over Europe.