Workshop on data assimilation: initial conditions and beyond

In the first part of this talk, we will review what has been done at Météo-France regarding the links between Data Assimilation and Machine Learning.
We will then also introduce what we are eager to explore in a near future, including the emulation of the analysis step.

The poster will showcase observation influence results of the convective-scale data assimilation system of the German Weather Service (ICON-KENDA), which uses a Local Ensemble Transform Kalman Filter (LETKF). Partial analysis increments (PAI) are used as a new diagnostic tool to assess the three-dimensional observation influence in the analysis. PAI enables the evaluation of the influence of various subsets of observations or observation types, such as, e.g. visible satellite images, in the presence of other conventionally assimilated observations. The recently proposed method uses a formalism similar to EFSOI. However, it focuses primarily on the analysis. A further aim of the poster is to explain the PAI diagnostic and its similarities and differences to EFSOI.
PAI allows for computationally inexpensive sensitivity studies on certain parameters of the LETKF system, such as observation error or localization length scale. The poster will show an application of PAI where it was used to compute optimal vertical localization length scales of visible satellite observations with respect to radiosonde observations used for verification.
Further, it will be shown how the method can be extended to study the transition from the analysis influence into the short-range forecast. This may be relevant to gain insights about the nonlinearity of the model and the spin-up of the model state after data assimilation.

In 2020 ECMWF started assimilation of European Mode-S winds. In 2023 the aircraft thinning was revised to prevent locally very dense data causing convergence problems. Since late 2024 we have been trialling use of the winds form the Met Office EMADDC Global Mode-S product - with very encouraging results. The coverage depends on the air traffic control systems in place. The data provide a huge increase in aircraft winds in the tropics and we expect to use the data operationally in 2025.

Numerical weather prediction (NWP) models are essential tools for forecasting meteorological conditions. Data assimilation techniques further improve these models by incorporating observational data. However, traditional data assimilation methods often rely on fixed observational networks, which may be insufficient, especially during extreme weather events. Crowdsourcing offers a promising solution by leveraging local residents with mobile devices and vehicles to collect real-time weather observations. This approach can increase the spatial and temporal density of observations, leading to potentially better data assimilation performance. However, the diversity of crowdsourced data sources presents challenges in terms of data quality, format, and integration into data assimilation systems. To address these challenges, we propose a novel crowd-aided pipeline. This pipeline incorporates crowdsourcing, information inference, and data assimilation techniques to effectively utilize observations. We demonstrate the effectiveness of our pipeline through real-world case studies involving extreme weather events.

Recent advances in generalized variational inference and closely related generalized posteriors via generalized Bayeisan inference offer curious opportunities for Bayesian data assimilation. First advances deriving equivalents to the celebrated Kalman filter with novel, desiarble properties while maintaining conjugacy in the analysis step and stability under mild assumptions. This encourages investiagting generalizations to more sophisticated and state-of-the-art approaches in data assimilation, e.g. akin to the popular local ensemble transform Kalman filter (LETKF). We present first results of one such apdatation of the LETKF utilizing diffusion score matching (DSM) as a discrepancy between measures of observations with an appropriate choice of diffusion matrix to assimialte observations online under suspicion of mis-specification of the observation likelihood. Additionally, we showcase consistency of the approach with regard to assumption expanding the KF to the LETKF for the DSM KF in relation to the DSM LETKF. The resulting algorithm maintains the desirable properties of the LETKF while obtaining robustness to outliers at a quantified cost of uncertainty. Moreover, the resulting formular suggests an insightful interpretations of novel dynamics of this new filter.

We will first provide an overview of the various atmospheric, SST and sea-ice data assimilation systems currently operational or under development at Environment and Climate Change Canada (ECCC) as well as our unified framework: the Modular and Integrated Data Assimilation System (MIDAS). Several ongoing research activities will then be highlighted, including increasing our usage of ensemble forecasts, moving towards coupled data assimilation and the integration of machine learning approaches to better leverage the existing observing networks.

Recent and ongoing evolutions at Météo-France will be reviewed regarding the representation of error covariances and associated DA formulations, benefitting from the object-oriented OOPS framework. This includes updated versions of EDA approaches, which allow the propagation of error contributions to be simulated in DA cycling. Enhanced representation of flow-dependent 3D and 4D background error covariances will be then presented, corresponding to evolutions towards 4DEnVar formulations for the mesoscale AROME model and for the global ARPEGE system. This also leads to EnVar developments for surface DA and for coupled ocean-atmosphere DA. Finally, ongoing work on the representation of horizontal correlations in observation errors will be illustrated.

The Copernicus Atmosphere Monitoring Service (CAMS), operated by the European Centre for Medium-Range Weather Forecasts (ECMWF) on behalf of the European Commission, provides daily analyses in near-real time (NRT) and 5-day forecasts of atmospheric composition as well as a reanalysis of atmospheric composition going back to 2003. This is accomplished by assimilating air quality and greenhouse gas retrievals from a range of satellite sensors into ECMWF’s Integrated Forecasting System (IFS) which contains on-line chemistry routines.

While the assimilation of atmospheric composition data is beneficial in its own right to provide the best possible initial conditions for the 5-day CAMS composition forecasts, there are exciting and challenging potential feedbacks between composition and NWP assimilation that are worth exploring. For example, using ozone, aerosols or greenhouse gases interactively in the model’s radiation scheme or radiative transfer observation operator could improve NWP forecasts and might allow us to make better use of the assimilated radiances. There is a beneficial impact of ozone-wind tracing in the ECMWF 4D-Var system on ozone analyses and forecasts in the NWP version of the IFS, and NWP forecast benefits are seen when extending the humidity analysis into the stratosphere.

In this talk we will explore some of these feedbacks and highlight the benefits as well as the growing complexity and challenges that composition forecasts can bring to the Earth system model and data assimilation system.

Balances among analysed meteorological variables in the data assimilation process can be effectively introduced by means of variational constraints (VC) [1]. The ALADIN-NH dynamics (the dynamics of an operational, state-of-the-art, convection permitting NWP system known as HARMONIE-AROME), contains a semi-implicit linear system for the non-hydrostatic fully compressible Euler equations (SI) that can be used to give a precise definition to these constraints. An interesting feature of this method is the integration in the analysis algorithm of the vertical velocity field, which clearly is important in convection permitting NWP. Another point is that SI is a time-step forward operator, and this property gives to this algorithm a nudging-like functionality making it well suited for DA continuous-in-time, also an indispensable feature for NWP of intrinsically short-time predictability weather.

This VC problem is solved using Green Functions (GF), a method that has a number of advantages and that also reveals an interesting correspondence between these GF kernels and covariance matrices. This correspondence can be pursued further and, drawing on ideas of field theory, leads to a technique for modelling flow-dependency in the covariance structure of the model error (see plot attached). These methods are being implemented in the HARMONIE-AROME system, and tested in an operational-like environment.

Data assimilation for Numerical Weather Prediction has traditionally been concerned with estimating initial conditions for forecasting the weather and, increasingly, the future evolution of the Earth system on scales ranging from days to months. This is still DA core business and there are numerous directions where near and medium-term DA developments hold promise to significantly extend current predictive capabilities. We will briefly review these DA research directions from an ECMWF perspective and discuss some of the opportunities and challenges ahead.

At the same time the recent advent of Machine Learning has changed the landscape of NWP and Earth system prediction and offered new opportunities but also challenges for the sustainability of current R&D and operational practices in our domain. In the second part of the talk, we will give an overview of the activities going on at ECMWF aimed at hybrising current NWP workflow with Machine Learning technologies and make the case that these activities offer a more effective and sustainable development path than the acritical adoption of fully data-driven emulators.

The project "Earth System Modelling at the Weather scale" (ESM-W) by DWD in cooperation with GeoInfoDienst BW aims to develop a coupled ocean-atmosphere forecasting system based on ICON-O for the ocean model and ICON-NWP for the atmosphere. The system employs a weakly-coupled assimilation approach that integrates DWD's operational data assimilation (DA) system for the atmosphere with a newly developed 3DVar(-FGAT) system for the ocean. The atmospheric assimilation operates on a three-hour window, consistent with the operational setup, while ocean assimilation is performed once per day.

In the ocean, vertical profiles of temperature and salinity from ARGO floats, along with satellite products such as sea surface temperature (OSTIA) and sea surface salinity (SMOS mission), are assimilated. This work presents the current status of the coupled system, alongside recent results concerning weakly-coupled DA experiments and associated daily forecasts with lead times of up to 10 days. The ocean analysis and forecast data are evaluated against in-situ observations, including non-assimilated datasets such as fixed buoys as well as satellite-derived products. Lastly, ongoing advancements are explored and plans for future system development are outlined.

Numerical weather prediction (NWP) has undergone a profound revolution in recent decades. At the same time, the volume and diversity of atmospheric observations have expanded dramatically, including enhanced measurements of atmospheric water. The advent of deep learning in geophysical modeling has further transformed the field, leading to the development of hybrid NWP models that combine physical modeling with machine learning techniques. In some experimental cases, machine learning models have even replaced traditional physical models, trained using reanalysis data produced through the assimilation of observations into physical models.

Building on these exciting advancements in numerical modeling, we examine perspectives of data assimilation and its integration with machine learning. Our discussion highlights the critical importance of respecting physical constraints and accurately representing uncertainties, particularly in high-resolution models. Such models must capture physical processes across scales ranging from a single kilometer to thousands of kilometers, including the complex dynamics of water phase transitions.

The German Federal Maritime and Hydrographic Agency (BSH) has been providing forecasts of water level, currents, temperature, salinity, sea ice, and biogeochemical parameters for the North and Baltic Seas, with a focus on German coastal waters, for over 30 years. The service is based on its ocean-biogeochemical model HBM-ERGOM, which is further coupled to the parallel data assimilation framework (PDAF, https://pdaf.awi.de). The data assimilation component uses the Local Error Subspace Kalman Transform Filter (LESKTF) algorithm. All ensemble members are dynamically generated through a second-order exact sampling method using the trajectories from the HBM-ERGOM model.

Currently, the system has various modules to assimilate different satellite-derived products, including sea surface temperature (SST), sea ice concentration (SIC), sea ice thickness (SIT), and chlorophyll. Further specific phytoplankton subgroups, such as cryptophytes, cyanobacteria, and diatoms can be assimilated from novel satellite derived plankton data. Additionally, we can assimilate in situ observations, such as temperature and salinity. Different data assimilation coupling regimes and satellite products lead to different results.

This poster will compare the influences of assimilating different satellite products on model performance and discuss the role of data assimilation in enhancing the operational forecasting system.

Starting circa 2022, global data-driven numerical weather prediction (NWP) models have become competitive with leading physics-based systems such as ECMWF's Integrated Forecast System (IFS) across a number of headline skill scores. Currently, these AI-powered models invariably rely on conventional weather (re)analyses for initialization and training. The question then arises: can machine learning be used to build an end-to-end forecasting system, thus bypassing the need for a traditional analysis?

Recent attempts to emulate 4D-Var have shown some promise, but have proven unable to calculate a weather analysis at comparable resolution and quality to what is routinely produced by traditional data assimilation methods such as 4D-Var.

Over the past year, the ECMWF has been exploring a radically different data-driven approach to learning a medium-range weather forecast exclusively from Earth System observations, called Artificial Intelligence Direct Observation Prediction. AI-DOP seeks to learn physical dynamics and processes in the Earth System by leveraging the relationships between observed quantities (e.g., microwave satellite brightness temperatures, GPS-RO bending angles, or altimeter wave height data) and geophysical variables such as temperature and winds, using only historical time series of satellite and conventional observations - i.e., with no climatology and/or NWP (re)analysis inputs or feedbacks.

An in-depth look into AI-DOP forecasts offers good indications that the system is able to build a coherent internal representation of the Earth System state. The relationships that the network learns between different observed variables generalize to areas where no observations exist - for example, forecasts of upper-level winds compare well with ERA5 even in areas where there is no radiosonde or aircraft coverage.

This talk will give an overview of AI-DOP and its current development status, and discuss possible ways forward towards a data-driven, end-to-end system that can compete with the IFS at the medium-range.

The operational NWP system at DWD uses for its global forecasts the ICON model at 13km resolution including a two-way nesting at 6.5km for the wider European area. An EnVar assimilation with a 40 member ensemble provides the flow-dependent part of the background errors as well as the analyses for ensemble-based forecasts for 7 days. It ingests a wide range of conventional and satellite data in a 3-hourly cycling. On the convection-resolving scale, the ICON is used in a regional 2km setup, with an ensemble assimilation LETKF approach. Additionally, it is run in a rapid-update cycle for very short-range forecasts using a more enhanced 2-moment cloud physics scheme setup which is beneficial especially for the direct assimilation of the full 3-dimensional radar reflectivity data as well as all-sky radiances from the infrared and visible range.

Recent upgrades of the global system include the addition of an ensemble for dust aerosol forecasts with ICON aimed at improving forecasting to support the transition to renewable energies. Additionally, marked improvements in 2m forecasts have been achieved through a new method exploiting 2m temperature increments to adjust parameters relevant for boundary layer physics in an adaptive parameter tuning approach.

Current work focuses on further evolving the ICON to a coupled atmosphere and ocean analysis and modelling system as well as an extension for the modelling and assimilation of trace gases. In order to enable improved analysis and forecasts of clouds, especially also including ice clouds, experimentation is ongoing with a 2-moment ice microphysics scheme on the global scale with a view to also add more satellite data sensitive to ice clouds. Additionally, AI-based methods are being tested and included into different parts of the NWP system. To answer the need for increasingly detailed forecasts, very high-resolution runs at 500m are being studied as well as an addition of data assimilation for the European nest within the global runs.

The presentation will present DWD’s current operational forecasting system along with a discussion of specific challenges linked e.g. to increasing resolution, the integration of additional data sources, as well as new focus areas of forecasting, like renewable energies and upper tropospheric humidity and contrail formation for aviation applications.

• Overview DWD DA system global, regional; mention time constraints/cut-offs (evolution: aerosol,
• Role of ensemble approaches -
• Developments in data use (time diagram of satellite data usage & cake diagrams global/regional)
• Extensions to coupled modelling / ESM: ocean and aerosol, trace gases
• Exploitation of new observations (VIS, MODE-S, aircraft humidity?)
• Challenges
  o New requirements/: upper tropospheric humidity, incoming radiance, wind at rotor height
  o Need for new and specific obs - and immense quantity of available OBS
  o Drive to higher resolution – challenges like DA window, linearity assumption? gaussian assumptions?
  o Exploitation of OBS in coupled modelling
  o Exploitation of OBS for improvements in physics, e.g. APT, parameter estimation -> ‘model of the day’? prospects for guiding improved physical modelling?

While most earth system assimilation research exists within the purely physical or purely empirical domains, a hybrid of the two may ultimately provide the best results. From a Bayesian perspective, it is only by combining prior knowledge (physical equations) with observations that the most accurate analysis (or posterior) is obtained. More practically, only physical equations are capable of accessing unobserved physical variables. A granular hybrid of data assimilation and machine learning would retain the better known physical components to provide strong constraints around which more poorly known or empirical model components, such as physical parametrizations, can be learned or improved from observations. This level of hybridization ultimately requires a complete overhaul of our data assimilation approaches and a new machinery to support it. To adapt to evolving model components and new observations, as well as to cope with the changing climate, it is unclear whether the hybrid should evolve continuously, or whether the model should stay fixed apart from during defined training phases. Despite all this, pioneer hybrid approaches are already becoming part of the operational forecasting system, where they allow rapid expansion to new domains, particularly the land, snow, sea ice and ocean surfaces. One example is the assimilation of microwave observations over sea ice. There are few "ground truth" observations in polar regions with which to train any conventional machine learning method, so the sea ice framework needs the hybrid approach to deal with the otherwise poor knowledge of both the physical state and the required physical models. In this time of rapid evolution of earth system assimilation and forecasting in response to machine learning and empirical methods, the granular hybrid approach is probably the least well explored, but perhaps the most promising, avenue for future progress.

Machine learning (ML) and more specifically deep learning (DL) is increasingly used in geophysical data assimilation (DA),
whether serving as a tool to improve classical DA, to be combined with DA, or to offer a substitute to DA. I will give an overview of the recent achievements and promising routes offered by ML into DA. For instance, ML can be leveraged in the regularisation of ensemble-based DA, in the solvers of variational DA methods, for generating or augmenting ensembles in DA, for building surrogates of the tangent linear and adjoint models to be used within DA, to learn a model error correction within a weak-constraint 4D-Var framework, or, ultimately, as a replacement for the DA analysis. I will also present an example where DL unveils a potentially efficient analysis scheme that the DA community completely overlooked so far.

Satellite remote sensing observations cover most of the ocean and play a vital role in operational forecasting systems. However, satellite observations cannot provide information below the sea surface, so it is important to effectively transmit satellite surface observations to the subsurface ocean in the assimilation system. Indirect assimilation of satellite remote sensing data means that the satellite remote sensing data is first inverted into a T/S "pseudo" observation and then assimilated into ocean models, so as to improve the simulation effect of the subsurface thermohaline structure. The U.S. Naval Research Institute has successively established the MODAS (Modular Ocean Data Assimilation System) and ISOP (Improved Synthetic Ocean Profiles) systems, both of which adopt indirect assimilation schemes. In this study, a global three-dimensional thermohaline field reconstruction system was established based on the deep learning thermohaline joint intelligent reconstruction model, which provided global thermohaline "pseudo" observations for the MaCOM (Mass Conservation Ocean Model) data assimilation system. A comparative experiment of reconstructed data assimilation was designed to explore the role of indirect assimilation in the MaCOM data assimilation system.

Machine learning (ML) models trained on the reanalysis record are now the most computationally efficient way to provide highly accurate forecasts for the medium range weather given the initial state generated by the traditional (highly computationally expensive) data assimilation methods. Can we use the power of the ML methods to enable faster, more accurate data assimilation? This presentation reviews several directions that NOAA is pursuing to incorporate ML into the data assimilation pipeline. This includes using the ML model emulators to propagate the ensemble forecast for the Ensemble Kalman Filter; use of the ML-based tangent linear and adjoint models in the context of the 4DVAR, ML-based balance and observation operators, ML-based observational bias correction, and early exploration of the direct ML-based data assimilation.

In recent decades, steady improvements in numerical weather prediction (NWP) skill have been driven by enhancements to data assimilation (DA) methods and increasing volumes of observations assimilated. Nevertheless, only 5-10 % of available satellite data is assimilated in operational systems, in part due to a lack of 1) knowledge of observation uncertainty structures and 2) appropriate fast numerical techniques. We will discuss recent progress in estimating and accounting for observation-error correlations, potentially allowing for the use of denser observations in operations. We will give examples for different observation types, including Doppler radar, geostationary and polar orbiting satellite data.
Given this progress, it is important to ask how prior and observation-error correlations interact and how this affects the value of the observations in analyses produced via conventional data assimilation techniques. In an optimal system, the reduction in the analysis-error variance and spread of information is shown to be greatest when the observation and prior errors have complementary statistics. This can be explained in terms of the relative uncertainty of the observations and prior on different spatial scales.
In the new paradigm of machine learning weather prediction (MLWP), uncertainty and information content remain important concepts. Overfitting and underfitting are well known issues that can arise. Overfitting occurs when a ML model learns the noise as well as the signal in the training data, resulting in poor performance when confronted with new independent data. Underfitting occurs when the training data does not contain enough information, and the ML model cannot make satisfactory predictions.

Future observing systems may be required to provide data driving both NWP and MLWP forecasts. Thus, a better understanding of the relationships between observation data information content and information encoded in the weights of a neural network is a key open question for future research.

Since 2020, the Met Office has worked on the reformulation of both its observation processing and its data assimilation system to ensure that the expected capability of the next generation HPC are fully exploited. To achieve this task, the MO has adopted the JEDI code framework (Joint Effort for Data assimilation Integration) and initiated a strong collaboration with JCSDA (Joint Centre for Satellite Data Assimilation). This collaboration has allowed rapid development of new data assimilation methods and tools and observation quality control procedures.

In this presentation, we will give an overview of both our new JEDI-based observation processing application (JOPA) and our new JEDI-based Application for Data Assimilation (JADA) as well as their key components such as: the hybrid tangent-linear. Scientific and technical challenges and achievements will also be presented.

At time scales of one week and beyond, predictability of day-to-day weather is a global problem, as the quality of midlatitude forecasts is influenced by large-scale tropical processes and underlyling strength of tropics-extratropical coupling (TEC). Advancements in extratropical practical predictability have been argued as coupled to improvements in tropical initial states and a better representation of TEC-related processes in NWP models. Additionally, equatorially-trapped waves such as Kelvin and mixed Rossby-gravity waves have been hypothesized to contribute to longer predictability in the tropics relative to the extratropics, potentially benefiting midlatitude predictability. However, the underlying mechanisms remain unclear, and no evidence from ECMWF data supports extended predictability of these waves.
A well-established approach to studying the impact of tropical forecast errors on extratropical forecast skill involves nudging tropical forecasts toward analyses. While this method suppresses tropical error growth and isolates extratropical forecast errors, it prevents the study of TEC processes in subtropical regions, where both midlatitude circulation and tropical forcing play roles.
I will present a novel framework for investigating the role of the tropics in global predictability, implemented across a hierarchy of models. This approach applies observing system experiments (OSEs) or observing system simulation experiments (OSSEs) that assimilate observations exclusively within the tropics or extratropics. It can be seen as observation-denial experiments but instead of excluding specific observation types, these OSSEs confine or deprive observations to the tropical belt. I will discuss latitude- and altitude-dependent analyses of forecast error growth in balanced and unbalanced wave circulation focusing on subtropical regions and TEC processes associated with poleward-propagating wave signals from tropical initial conditions.

In recent years, data assimilation (DA), and more generally the climate science modelling enterprise have been influenced by the rapid advent of artificial intelligence, in particular machine learning (ML), opening the path to various form of ML-based methodology.
In this talk we will schematically show how ML can be included in the prediction and DA workflow in different ways with various degrees of integration within each other. In a so-called “non-intrusive” ML, we will show how ML can be used to supplement a chaotic system and help predicting the local instabilities and/or abrupt regime’s changes. DA and ML can also be placed side by side in an iterative approach alternating a DA step that assimilate sparse and noisy data, and a ML step whereby the data-driven model is further optimised against the analyses outputted from the DA. In a further level of fusion ML can finally be used to within hybrid ML-DA methods in which ML is used to cope with some limitations in DA approaches. In particular we shall show an innovative formulation of the EnKF that embodies a variational autoencoder enabling the EnKF to (i) handle non-Gaussian observations, and, (ii) respecting physical balances.
Using a set of idealised model and observational scenarios, we will show numerical results for all of the above-mentioned possibilities. We will focus on, and will be motivated by, problems originated in diverse areas of climate science, namely chaotic systems such as the atmosphere and the highly nonlinear and non-Gaussian DA for Arctic Sea ice and ocean biogeochemistry.

At RIKEN, the Japan’s national flagship research institute for all sciences, we have been exploring several attempts to integrate data assimilation (DA) and AI/ML. DA integrates the (usually process-driven) model and data, while AI/ML is purely data driven and is proven to be very powerful in many applications. An example is to integrate data-driven AI/ML-based precipitation nowcasting with process-driven numerical weather prediction (NWP). We developed a nowcasting system based on a convolutional long short term memory (LSTM) which takes several time steps of 2-D precipitation image data to predict future images. NWP with radar DA produces future precipitation images, which can be input to the data-driven LSTM to further improve the predicted images. Another example is to develop ML’ed observation operators for satellite radiances. We obtained an improvement by purely ML’ed observation operators without any information from a physically based radiative transfer model. The third example is to use DA with an ML’ed surrogate model for producing more accurate analyses for further training the ML’ed surrogate model. We found that DA with flow-independent background error covariance could produce more accurate ML’ed surrogate model, but ensemble-based DA resulted in a mixed situation probably because the ensemble forecasts by the ML’ed surrogate model may not produce proper error covariance. We also explored developing a limited-area ML’ed surrogate NWP model in collaboration with IMT-Atlantique. In this presentation, we will share the most recent activities of integrating DA and AI/ML at RIKEN.

Enhancing the spatial resolution and increasing the update frequency of global analyses and forecasts enables weather prediction systems to better capture rapidly evolving atmospheric conditions. The Integrated Forecast System (IFS) of ECMWF comprises an Earth-system model coupled with an advanced 4D-Var data assimilation (DA) system, which was recently experimentally upgraded to higher spatial resolution and to provide more frequent updates.
We developed and tested an updated ECMWF 4D-Var DA system, called Extending Window (Ext-Win) DA, which increases the frequency of analysis updates from the current 6-hour interval to as frequent as hourly. The Ext-Win DA system employs assimilation windows of varying lengths, ranging from 5 to 14 hours, to incorporate the most recently available observations and provide optimal estimates of the Earth's system state at any time of day. Considering computational constraints and product dissemination requirements, we propose a practical implementation of the Ext-Win framework that upgrades the ECMWF DA system to provide analysis and forecast updates every 3 hours. The impact of this new framework is evaluated in terms of forecast skill, convergence rate, and computational footprint.
The higher resolution initial conditions were achieved through higher resolution (4.4 km) 4D-Var trajectory and higher resolution (20 km) 4D-Var minimizations utilizing tangent linear model and its adjoint. We demonstrate significant improvements in large-scale forecast skill, an improved forecasting of extreme events, and better use of high-resolution observations. The impact of the higher resolution DA system is demonstrated using a specific test case of the tropical cyclone Otis, which made landfall as a Category 5 tropical cyclone, and was only predicted to reach the tropical storm intensity by most of the global (and regional) NWP models. Employing higher resolution 4D-Var DA, we were able to emulate the observed rapid intensification of TC Otis.

Satellite observations are critical for numerical weather prediction (NWP), yet the full potential of visible spectral data remains to be explored. In recent years, ECMWF has been advancing efforts to incorporate visible satellite observations to enhance the analysis and forecasts of clouds and aerosols in the Integrated Forecasting System (IFS). We discuss results from the CLOVIS-2 and CERTAINTY projects, presenting visible reflectance (655 nm) monitoring and assimilation experiments using observations from the Ocean and Land Color Instrument (OLCI) onboard ESA's Sentinel 3A and 3B satellite missions. Our study assesses the first-ever successful experimental assimilation of visible all-sky satellite observations in the IFS using model CY49R1, which became operational in November 2024. A comprehensive evaluation of these experiments demonstrates that visible reflectance assimilation can improve the model analysis of clouds by better fitting the model trajectory to observations in reflectance space. We will also discuss the remaining challenges related to the future operational assimilation of visible observations, including observation operator refinements, observation error modelling, and the role of model errors and biases. Our findings underscore the vast potential of visible spectral data for operational NWP and future re-analysis products.

The Deutscher Wetterdienst (DWD) is dedicated to utilize the potential of artificial intelligence (AI) across various components of the data assimilation (DA) process such as quality control, forward operators, bias correction, and error covariance estimation. DWD aims to employ a multi-faceted approach that leverages AI to address the complex challenges. We envision the AI-Var approach described by Keller and Potthast (2024) to be a cornerstone of DWD's future DA system, reimagining DA for Numerical Weather Prediction (NWP) as a complete data-driven learning problem. By embedding the variational DA cost function within a neural network, AI-Var enables the direct assimilation of observations without requiring a pre-existing analysis dataset.

Building on AI-Var, DWD is developing AIDA, a generalized framework that extends AI-var into a comprehensive DA system. While AIDA is designed to handle arbitrary grids and diverse observational datasets, it is also developed in a modular way, ensuring compatibility and adaptability to seamlessly integrate with DWD’s operational workflow. By leveraging scalable AI architectures, AIDA combines computational efficiency and flexibility, making it a robust foundation for a fully data-driven NWP system as well as other heavily DA-related tasks such as reanalysis. Furthermore, the system also supports the blending and merging of different sources of data, enabling its use also in other contexts such as nowcasting and post-processing.

DWD envisions its future DA system to rely extensively on AI, integrating it with classical physics- and statistics-based approaches while leveraging its transformative potential. This hybrid approach ensures that the strengths of traditional methods, such as physical interpretability and established theoretical foundations, complement flexibility and computational power of AI. Additionally, DWD aims to foster community collaboration by contributing AIDA to Anemoi as a data assimilation component. This enables the community to benefit from and build upon AIDA’s capabilities, thus enabling further advancement of data-driven NWP.

This presentation provides an overview of the current status, advancements, and future vision of data assimilation within the ACCORD-HIRLAM consortium. Focused on the HARMONIE-AROME framework, it highlights ongoing efforts in enhancing weather prediction accuracy through advanced data assimilation methods, including 3D-Var, 4D-Var, and ensemble-based techniques. Key areas of progress include enhanced data assimilation methods, the integration of more observational data types, improvements in radar and satellite data usage, optimisation of computational performance, and integration of Artificial Intelligence and machine learning in various parts of the data assimilation process.

The work also emphasizes pre-operational implementations and diagnostic and verification tools to refine assimilation strategies, demonstrate the benefits of high-resolution data assimilation and ensure robust operational readiness. We also show some experiences from km-scale re-analysis projects using HARMONIE-AROME.

Current challenges include combining operational needs such as short observation cut-off times and the need for frequent launches of forecasts with state-of-the-art developments regarding data assimilation algorithms and observation usage. Towards sub-hourly observation cycling and continuous data assimilation implications for model spin-up are being addressed. Plans also include expanding system capabilities, such as the assimilation of newly launched and planned satellite instruments, application of all-sky and dynamical emissivities for low-peaking satellite channels and work towards coupling the atmospheric and surface data assimilation. There is also a need to focus on developments for the prediction of extreme weather, such as tuned and adaptive observation quality control, and measures to verify these improvements.

This collaborative effort represents a significant contribution to leveraging innovative technologies to improve high-resolution weather forecasting across the Nordic and European operational centres.

The relevance of observed information for numerical weather prediction has never been higher now that data-driven models based on machine learning have made the headlines with their performance and accuracy. While direct prediction from observations is attractive as it dispenses with the need for a complex analysis model, there are some limitations since only observed quantities can be the target of the learning. Therefore, there is still a reliance on physically based models which can provide analysis of present-day observations or reanalysis of past observations which can provide information on meteorologically relevant variables that are not directly observed. The balance between physics based versus ML techniques in forward modelling and traditional data assimilation may evolve but the need for a comprehensive global observing system will remain.

All traditional NWP physical models, as well as most data driven models, still rely on accurate analyses for their initialisation. These analyses are usually obtained with various assimilation techniques which vary from variational approached (3D-Var, 4D-Var, etc) to ensemble methods (various flavours of Ensemble Kalman Filters, etc). What all these systems have in common is the need for observations which are well calibrated and accurate. Satellite observations from microwave and infrared sensors as well as GPS radio occultation (RO) have the largest impact in the analysis, particular in areas where ground-based observations are scarce, such as over the open oceans. Ground based observations are however fundamental as they provide a very accurate anchor for the satellite observations. Emerging low-cost technologies are an interesting area to explore, but their level of accuracy is still to be understood. Continued reliance on the commercial sector for satellite observations, for example small satellite constellations for radio occultation measurements, is, however, expected.

This talk will cover the present observing system usage for NWP analysis at ECMWF and the future landscape of the next decade. This will include highlights from preparatory studies for EUMETSAT’s new hyperspectral sensors, IRS on MTG-S and IASI_NG on EPS_SG, the proposed microwave constellation (STERNA) and the Doppler wind lidar (Aeolus-2) as well as studies to exploit the ESA’s Copernicus evolution missions (CIMR, CRYSTAL, LSTM, etc). Results from the RO meteorology experiment (ROMEX) with increased observation availability will be presented. Advances towards all-sky IR and visible radiance assimilation for operational NWP will also be presented along with the exploitation of actively sensed (radar and lidar) observations from EarthCARE.

The weather we experience is a result of the atmosphere interacting with all the other parts of the Earth system.
In particular, the atmosphere interacts with the ocean and waves, rivers and lakes, the land surface and vegetation, and sea ice and snow.

Many of these subsystems evolve with substantially longer timescales than the atmosphere itself, so can be a source of predictability but also a brake on the speed at which we update the initial conditions for the Earth system.

In this talk we will discuss why coupled data assimilation exists as a topic and why the analysis of the different components is often not combined with the atmospheric analysis.
But more optimistically, coupled data assimilation can bring benefits to both the analysis of the Earth system state, and on the coupled forecasts themselves. We will discuss examples where this has already been demonstrated, what the community is currently trying to achieve, as well as the challenges which we foresee.

The predictability of weather is intrinsically limited by rapid growth of small-scale errors. But current prediction systems are not yet at the limit, and improvements are possible in the forecast model, the data assimilation system and in the observations. In this presentation I will present estimates of the potential for improvement from each of these components, which suggests that the largest gains will come from improved data assimilation. Some implications will be discussed for the potential of machine learning methods to accelerate these improvements, including new results on how ai weather models are constrained by the choice of loss function versus model architecture and training data.

Systematic model errors significantly limit the predictability horizon and practical utility of the current state-of-the-art forecasting systems. Even though accounting for these systematic model errors is increasingly viewed as a fundamental challenge in the field of numerical weather prediction, estimation and correction of the predictable component of the model error has received relatively little attention. Modern implementations of weak-constraint 4D-Var are an exception here and a promising avenue within the variational data assimilation framework, showing encouraging results. Weak-constraint 4D-Var can be viewed as an online hybrid data assimilation and machine learning approach which gradually learns about model errors from partial and imperfect observations, allowing to improve the state estimation. We propose a natural extension of this approach by applying deep learning techniques to further develop the concept of online model error estimation and correction.

In this talk, we will present recent progress in developing a hybrid model for the ECMWF Integrated Forecasting System (IFS). This system augments the state-of-the-art physics-based model with a statistical model implemented via a neural network, providing flow dependent model error corrections. While the statistical model can be pre-trained offline, we demonstrate that by extending the 4D-Var control vector to include the parameters of the neural network, i.e. the model of model error, we can further improve its predictive capability. We will discuss the impact of applying the flow dependent model error corrections in the medium range forecasts on the forecast quality.

In recent years, machine learning has revolutionized weather forecasting. However, its application to data assimilation has not yet seen the same progress. This disparity is not due to the simplicity of data assimilation but likely stems from the absence of accessible benchmark datasets. Effective data assimilation for earth system modeling is hindered by high-dimensional state spaces, nonlinearities, and complex error structures, especially when approaching the hectometric scale. This talk explores the potential of machine learning to address these issues within consortia focusing on very high resolution models like HIRLAM (ACCORD). Representational learning like autoencoders can reduce dimensionality and offer latent space representations where the system is kept in balance. Transport methods, such as diffusion or normalizing flows, can transform non-Gaussian error distributions into Gaussian ones, facilitating the use of classical approaches. Self-supervised learning holds promise for capturing interactions within and between the land, ocean, and atmosphere. Additionally, generative models can efficiently produce ensembles for uncertainty quantification and extend data assimilation from state to probabilistic estimation. Ultimately, machine learning holds the potential to revolutionize the entire weather prediction chain by enabling direct forecasts from observations, potentially rendering traditional data assimilation obsolete.

Computational efficiency improvements are central to the development of the Ensemble of Data Assimilations (EDA). Over the past years we have achieved more accurate and reliable EDA to match the resolution and accuracy improvements in the assimilation system through a combination of better model and observation uncertainty representations and increase in number and resolution of members enabled through computational efficiency gains. Taken together, the computational optimizations of the EDA over the last few years resulted in EDA about 1/4 of the cost for same performance and enables improved performance at manageable costs.
The last development is soft re-centring which enables to run EDA members at same 9km resolution as the deterministic 4D-Var analysis. In soft re-centring the control member of the EDA runs in a 4D-Var configuration closer to the high-resolution 4D-Var, which is used to re-centre the background and warm-start the minimization of the perturbed members. The soft re-centring warm-start shifts the distribution of perturbed members first guess towards the unperturbed observations by adding the analysis increment from the first minimization of the control member. This effectively replaces one minimization so that same accuracy and reliability can now be achieved with a single minimization in the perturbed members compared with two minimizations required without soft re-centring. As each perturbed analysis is the result of a full 4D-Var analysis update, it is a valid model trajectory and this allows to minimise the initial shock of re-centring.
To further improve the computational efficiency of the EDA, we have started to explore the application of probabilistic data-driven techniques in 4D-Var. We developed a Variational Encoder-Decoder (VED) machine learning model for emulating EDA statistics. Specifically, we define a VED that parameterizes the sampling distribution of the diagonal part of the flow dependent 4D-Var B matrix, conditioned on a subset of EDA analyses. We show the approach is theoretically justified and explore the sensitivity of ECMWF 4D-Var setup to the use of such methods. Next steps are to show to what degree the EDA can be augmented with emulated EDA members, and in particular explore the trade-off between the number of perturbed EDA members and emulated members, with attention on the minimum number of EDA members required to maintain accuracy and reliability.

Instruments on geostationary and polar orbiting satellites generate high resolution images providing information on clouds and aerosols, which is highly relevant for numerical weather prediction (NWP). While using thermal infrared images for data assimilation and model evaluation is well-established, and the operational use of visible channels is emerging, near-infrared channels are not yet directly assimilated, mostly because fast and accurate forward operators have become available only very recently. In particular the 1.6 micron channel available on many satellites is interesting for NWP, as this channel is not only much more sensitive to particle radii than visible channels but allows also for distinguishing water from ice clouds. Solar channels sensitive to low-level water vapor like the 0.9 micron channel on the flexible combined imager onboard Meteosat Third Generation could complement the infrared water vapor channels that cover only the upper half of the troposphere. Here we will present the latest developments related to MFASIS-NN, a fast, neural network-based forward operator that allows us to exploit solar satellite channels for data assimilation and model evaluation. We will discuss the information content of the solar channels, review the current state of MFASIS and present new approaches that allow for simulating also water vapor sensitive channels and aerosol-affected reflectances. Moreover, the sensitivity of the simulated images to subgrid cloud assumptions will be discussed.

In variational data assimilation, the assumption of uncorrelated observation errors is commonly made to simplify the inverse correlation operator. However, this assumption becomes problematic when dealing with certain observation types, notably high-resolution satellite data. Neglecting observation error correlations during assimilation often results in suboptimal analyses, where observations tend to be overfit at large spatial scales and underfit at small scales. Conventional mitigation strategies include thinning — assimilating only a subset of spatially separated observations — and inflating variances to mitigate overfitting at large scales. Both methods inhibit the extraction of small-scale features from observations and thus limit the potential of high-resolution data. In this poster, we describe developments for modelling correlated error in altimeter data for the ocean data assimilation system NEMOVAR. The approach is based on a diffusion operator which provides a cost-effective and flexible framework for modelling the inverse observation-error correlation operator with unstructured data. While accounting for observation-error correlations should improve the accuracy of the analysis, it also affects the convergence rate of the minimisation algorithm. In operational applications, where the minimisation process is usually truncated before achieving full convergence, even correctly accounted for observation-error correlations might therefore be detrimental to the accuracy of the analysis. We explore the influence of the observation-error correlations on both the sensitivity and convergence rate of the minimization. We provide insights into how the choice of an observation-error correlation model must reflect a balance between computational efficiency and accuracy.

Satellite brightness temperatures have a significant positive impact on atmospheric analyses and ensuing numerical weather predictions (NWP). However, much science potential remains untapped due to certain limitations. In particular, data assimilation algorithms often assume spatially uncorrelated observation errors, thus it is common practice to perform spatial thinning of observations to minimize correlations between remaining observations. Additionally, multiple spectral channels are currently rejected in the presence of clouds and over land/sea-ice. Finally, only a relatively small subset of all available channels is assimilated for NWP, especially for hyperspectral infrared sounders.

ECMWF recently introduced a new Deep Learning (DL) approach to predict future weather parameters (e.g., surface temperature, wind) directly from raw radiance observations. This proof-of-concept suggests that there may be predictive potential in an increased usage of satellite radiance observations and there is interest to build on this hypothesis from the perspective of NWP.

This poster outlines the methodology being explored at ECCC to leverage DL to retrieve thermodynamic profiles of the atmosphere directly from raw radiance observations, without knowledge of the prior state. In other words, this study aims to generate a DL inverse observation operator that would enable the assimilation of additional radiances in the form of thermodynamic profiles. This preliminary work is based on an idealized framework using modeled temperature and humidity profiles in conjunction with a radiative transfer model (i.e. RTTOV) to generate a “known truth” and synthetic radiances. It is currently planned to use part of the dataset to learn mapping between co-located synthetic radiances and modeled thermodynamic profiles. Finally, using an independent evaluation dataset, resulting profile retrievals from the DL approach will be compared with those from a 1D-VAR scheme.

Earth-scale weather modeling represents one of the most computationally intensive challenges in scientific computing. Data assimilation, the process of integrating observational data with numerical models, is particularly demanding - operational methods either require expensive adjoint models or face limitations in uncertainty quantification. Following our introduction of score-based data assimilation (SDA) and its validation on idealized systems and small-scale ocean models, we present its extension to global weather modeling at 0.25-degree resolution, covering six surface variables and six atmospheric variables across 37 pressure levels at hourly intervals. Our approach combines SDA with a latent diffusion model trained on ERA5 and operating in a 100x compressed space, substantially reducing computational costs while preserving physical realism at the original resolution. The framework provides access to the full posterior distribution over state trajectories of arbitrary lengths and can be conditioned on diverse types of observations without retraining. Initial results demonstrate physically consistent state trajectories at the global scale and well-calibrated uncertainty estimates. Finally, while SDA can operate independently for both data assimilation and forecasting, we show that it can also initialize numerical or machine learning-based forecast systems with uncertainty quantification. Overall, our initial results demonstrate the potential of latent SDA as a flexible and efficient data assimilation framework.

The localized weighted ensemble Kalman filter(LWEnKF) is a particle filter (PF) that uses the ensemble Kalman filter(EnKF) as the proposal density. This method can provide weights for each ensemble member abtained by the EnKF to indicate the importance of each ensemble member in simulating the the full posterior probability density function.

A global ocean assimilation and forecast system is developed using the LWEnKF and the mass conservation ocean model(MaCOM). For the mass conservation physical framework, we chose temperature, salinity, sea surface height and bottom pressure as control variables. A fast observation thinning method based on the K-Dimension tree was developed, which increase the computational effciency for the disordered grid of the MaCOM. The global ocean model has a horizontal resolution of 1/12 degree with 75 vertical levels, whereas the data assimilation employs a finer horizontal resolution of 1/6 degree, maintaining the same 75 vertical levels as the model.

Since August 15, 2024, the system has been in operational trial, assimilating observations including Sea Surface Temperature (SST), Sea Level Anomaly (SLA), Argo profiles, and temperature and salinity profiles reconstructed by an AI method. It provides 10 days of global ocean forecasts, including daily mean temperature, salinity, currents, and sea surface height. We have recently compared our forecast results from the past four months with observations, as well as with the widely recognized CMEMS forecast data. The results indicate that our forecasts are stable and reliable. We are committed to continuously validating and enhancing our system to ensure ongoing improvement.

Author: Josef Schroettle
Co-authors: Crisina Lupu, Chris Burrows

A refined 4D-Var assimilation system within DestinE allows us to assimilate the Meteosat-10/SEVIRI clear-sky radiances over Europe, as well as globally at a spatial scale of 75 km instead of the previous 125 km in the ECMWF Integrated Forecasting System (IFS). Higher resolution observations can potentially improve the analysis and therefore the prediction of extreme weather events over Europe, as well as globally. The effects of using higher resolution observations have been investigated with a detailed set of experiments and the impact on wind, temperature, and humidity has been evaluated. A broad range of experiments indicate that exploiting the higher spatial density clear-sky radiances leads to an improvement of humidity sensitive fields in short- range forecasts with the IFS as independently measured for example by instruments on low-earth- orbiting satellites (IASI, CrIS, SSMIS, or ATMS). Due to a reduced displacement and representativeness error, these changes could further lead to improvements in longer range forecasts as these errors propagate upscale nonlinearly. First experiments show an upscale propagation of initially very localised increments in the analysis fields of vertical wind, as well as humidity above the Pacific or the North Atlantic. Over time, these incremental improvements from the 4D-Var system lead to an improvement in forecast scores of the IFS up to 5 days ahead.

In addition, pre-processed GOES-16/ABI and GOES-18/ABI observations by NOAA have been assimilated with 10 min sampling rates at 75 km spatial density. Exploring how to best assimilate relatively small spatial and temporal scales for one geostationary satellite, will allow us to approach these smaller scales with other satellites such as HIMAWARI/AHI above the Pacific or MTG-I/FCI above Europe. Data from both satellites will be available for us early in 2024. Preliminary experiments demonstrate the ability of IFS to assimilate observations at the highest available temporal resolution for the GOES-16 and GOES-18 satellites. Higher resolution radiances observed at these shorter time intervals naturally capture smaller scale atmospheric features such as mesoscale convective systems. In our experiments, simultaneously assimilating observations at a higher spatial and temporal resolution leads to an impact that is only marginally better than assimilating higher density observations alone, suggesting a combined investigation of optimal time-assignment, as well as assessment of the observation error are needed to optimise the integration of rapid update measurements in 4D-Var.

Visible channels offer valuable information on clouds, their presence, location and nature and are complementary to the widely assimilated IR and MW data. Additionally, these channels are widely available both on geostationary and polar orbiting satellites. Their use promises improved cloud and near-surface analyses and forecasts, particularly in conjunction with all-sky assimilation of corresponding infrared channels. Here, the visible channel information is complementary especially for the analysis of the representation of low clouds and has a higher sensitivity with respect to some model physics aspects like sub-grid scale cloud representation.
In preparation of an assimilation of visible satellite images in the global NWP systems of DWD and ECMWF, we perform a joint evaluation and intercomparison of global ICON and IFS model cloud fields using one month of visible reflectance observations by SEVIRI, ABI and AHI on board the geostationary satellites Meteosat-9, Meteosat-10, Goes-16, Goes-18 and Himawari-9. These data offer a unique test bed as they cover a wealth of atmospheric situations and different local times. Model equivalents are computed using the Neural Network based forward operator MFASIS available in RTTOV-13.2 and we analyze the difference to the observed values. For an optimal comparability of the ICON and IFS evaluations, we ensure that the setup in both NWP systems is as consistent as possible, particularly in terms of model resolution, superobbing and quality control.
The possibility to use two global models in the evaluations is ideally suited to disentangle model and observation errors. Furthermore, we take advantage of our evaluation setup to study the results in view of different parameterizations of the effective cloud water and cloud ice particle radii from RTTOV or derived within the NWP models to provide feedback on the model cloud physics and also to compare the performance of the two models in predicting cloud cover and liquid water and ice water contents. These evaluations help understand the cloud-related biases and are important steps for developing an all-sky assimilation of visible satellite channels in the global systems of DWD and ECMWF, and is also being extended to the newly available FCI instrument onboard MTG.

Microwave Humidity Sounder (MHS) and Micro-Wave Humidity Sounder-2 (MWHS-2) are high-frequency sensors with a wavelength of 183 GHz, which are more sensitive to frozen particles than to water droplets. In the all-sky MHS/MWHS2 data assimilation experiment using the Korean Integrated Model (KIM) forecast system, the performance of the analysis and forecast fields for major atmospheric variables is evaluated by the European Centre for Medium-Range Weather Forecast (ECMWF) Integrated Forecast System (IFS) analysis. Cloud and precipitation are not directly assimilated in our hybrid four-dimensional ensemble-variational (4D-EnVar) data assimilation system - only temperature and humidity profiles are adjusted to fit all-sky observations. All-sky MHS/MWHS2 data assimilation shows an improvement of initial specific humidity, particularly in the equatorial middle and upper layers (700-200 hPa). This study aims to investigate how the improvement in the model initial conditions through all-sky radiance assimilation affects the precipitation prediction over East-Asia. Cycled analysis and forecast experiments will examine how mass components, in company with dynamic variables such as wind and pressure, are adjusted in response to the all-sky radiance assimilation.