18th Workshop on high performance computing in meteorology

CIMA Research Foundation, a non-profit organization committed to promote the study, research and development in engineering and environmental sciences, has developed a workflow manager for the execution of high-resolution NWP on a regional scale and the exploitation of meteorological products for further simulations (hydro, fire, energy production) supporting civil protection, and energy trading.

The system has to provide results within a strict time constraint, exploiting HPC resources granted on a best-effort basis, and cloud resources. Users include regional meteo services, civil protection, SMEs active on energy trading market, insurance and re-insurance companies.

The workflow manager, developed in go and fully controlled by REST APIs, is able to exploit concurrency to minimize the latency in all steps of the workflow and to minimize the uncertainties due to exploitation of best effort resources.
In particular, it fetches Global NWP products in parallel; as soon as a timeframe of global data is available, it is pre-processed by WPS with a custom namelist. Then the WRF simulation is started and continuously monitored. As soon as output timestep is available, we further process it on HPC and cloud resources, and we deliver the time steps of final products to service subscribers.
First output timestep is available in less than 5 minutes from the start of our NWP simulation.

We plan to enrich the system with: the possibility to exploit dedicated resources to further reduce the input-to-output latency; more flexible namelist generation, to automate on-demand simulations on a new domain; a web interface to simplify workflow monitoring and management.