19th Workshop on high performance computing in meteorology

Spherical Multiple-Cell (SMC) grid is an unstructured grid, supporting flexible domain shapes and multi-resolutions. It retains the quadrilateral cells as in the latitude-longitude grid so that simple finite difference schemes can be used. Sub-timesteps are applied on refined cells and grid cells are merged at high latitudes to relax the CFL restriction. A fixed reference direction is used to solve the vector polar problem so that the whole Arctic can be included. The SMC grid has been implemented in the WAVEWATCH III (WW3) wave model since 2012 and updated in the latest WW3 V6.07. A 4-level resolution (3, 6,12, 25 km) global wave forecasting model has been used in the Met Office since October 2016, leading to great reduction of model errors and removal of our European regional wave model. WW3 model parallelization is by wave spectral component decomposition (CD) in MPI mode, which has a limit on number of MPI ranks. Hybrid (MPI-OpenMP) parallelization may extend the node usage but the OpenMP scalability flattens out beyond a few threads. Another parallelization method to combine CD with domain decomposition (DD) is enabled in WW3 model by a multi-grid framework for further extension of node usage. The SMC grid is recently added into the multi-grid framework and its flexible domain shape allows optimized domain splitting and minimized boundary exchanges. The combined CD-DD method is tested with SMC sub-grids with various hybrid node-thread combinations. Results indicate that switching from pure MPI to hybrid MPI-OpenMP mode can halve the global model elapsed time. Using hybrid CD-DD on 3 SMC sub-grids may reduce the time further by 30%. Node usage is extended from 10 to 180 nodes and elapsed time for one model day run is reduced from 3.5 min on 10 nodes to 1 min on 180 nodes. Besides, the hybrid mode reduces memory demand on one computing node and allows future model updates to higher resolutions.