Training course: Data assimilation

24 Feb 2020, 08:45 → 28 Feb 2020, 17:00 Europe/London

ECMWF

Overview

This five-day module focuses on describing data assimilation methods and general aspects of assimilating observations. Aspects of the implementation of the assimilation techniques for real-size numerical weather prediction (NWP) systems will also be described.

As well as lectures there will be discussion and hands-on sessions - please see the draft timetable for more details.

Main topics:

• The fundamental data assimilation concepts
• Optimal Interpolation, 3D-Var, 4D-Var and the Kalman filter
• Ensemble Kalman Filter methods; Ensemble of Data Assimilations and uncertainty estimation; Hybrid variational/ensemble based methods
• Modelling of error covariances; handling of non-Gaussian errors
• The global observing system, with emphasis on how to use satellite observations
• Bias correction, quality control and diagnostics
• Applications of data assimilation methods for the land surface, ocean, atmospheric composition and reanalysis

Requirements

Participants should have a good meteorological and mathematical background, and in particular a good understanding of linear algebra. They are expected to be familiar with the contents of standard meteorological and mathematical textbooks.

If you are less familiar with data assimilation concepts, such as Bayes Theorem, you may wish to consider attending the University of Reading Introductory course, which runs the week before our course; for details see the corresponding tab on this page.

Introductory material not covered by the course can be found in our lecture note series.

Some practical experience in numerical weather prediction is an advantage.

All lectures will be given in English.

Please note that no funds are available from ECMWF to support participants' attendance at training courses.

A course fee is payable by applicants who do not reside in an ECMWF Member or Co-operating State.

Monday, 24 February

08:45 → 09:15 Registration

09:15 → 10:15 Introduction

In this session we will sort out: general house keeping for the course, such as computing accounts; have an overview of ECMWF research and introduce ourselves to one another.

Speakers: Andy Brown (ECMWF), Dr Sarah Keeley (ECMWF)

09:40 → 10:10 Computer hall tour

10:15 → 10:45 Coffee break

10:45 → 11:45 Overview of assimilation methods

The goal of the ECMWF Earth System data assimilation is to provide an accurate and physically coherent description of the state of the atmosphere, ocean, sea ice and land surface as an initial point for our forecasts.

This requires blending in a statistically optimal way information from a huge variety of observations and our prior knowledge about the physical laws of the Earth system, which is encapsulated in our models.

In this lecture we will lay the general conceptual framework on how to achieve this from a Bayesian perspective. We will then highlight the approximations and hypotheses which are required to make the assimilation problem computationally tractable and which underlie the practical data assimilation algorithms which will be described in detail in this training course.

By the end of lecture you should be able to:

• understand the basics of how a geophysical data assimilation system works;
• understand the main approximations and hypotheses which are required to build practical data assimilation algorithms for large geophysical systems

Speaker: Massimo Bonavita (ECMWF)

11:45 → 11:55 Comfort break

11:55 → 12:55 Assimilation Algorithms: (1) Basic concepts

This lecture will explain the basic concepts of the assimilation algorithms. The terminology used in the next lectures will be introduced. Simple examples will conduce towards the formulation of the optimal minimum-variance analysis. The optimal interpolation method will finally be presented.

By the end of the lecture the participants should be able to:

• Recognize the notations used for the rest of the week
• Solve the optimal minimum-variance analysis problem
• Apply the optimal interpolation method

Speaker: Sebastien Massart (ECMWF)

13:00 → 14:15 Lunch break

14:15 → 15:15 Conventional and actively sensed observations

This lecture will introduce how observations are an essential part of the data assimilation system.

It will focus on in situ (also called conventional) observations, from surface stations, drifters, aircraft and radiosondes. They are important both for direct use in the data assimilation system and for diagnostics. Radiosonde and surface observations also help to control the biases in the assimilation system. However, they are diverse and hey can be complex, so close attention to quality control, observation uncertainty and (in some cases) bias correction is needed to optimise their use. The use of new BUFR format high resolution radiosonde data will also be presented.

The lecture will also introduce the actively sensed satellite observations used for data assimilation at ECMWF: radio occultation data, scatterometer winds, and altimeter wind/significant wave height.

By the end of the lecture the student should be able to:

• understand how in situ and actively sensed observations are used in data assimilation, including bias aspects and observation uncertainty aspects.

• appreciate the diverse and complex range of in situ observations used in modern NWP.

• understand how radio occultation data, scatterometer winds and altimeter data are used in data assimilation.

Speaker: Lars Isaksen

15:15 → 15:45 Coffee break

15:45 → 16:45 Assimilation Algorithms: (2) 3D-Var

This lecture will present the 3D-Var assimilation algorithm. This algorithm is based in the formulation of a cost function to minimize. Minimization methods will be presented together with some information on how to improve their efficiency.

By the end of the lecture the participants should be able to:

• Recognize the 3D-Var cost function
• Explain the various terms of the cost function
• Question the efficiency of methods designed to find the minimum of the cost function

Speaker: Sebastien Massart (ECMWF)

17:00 → 18:00 Ice breaker

Tuesday, 25 February

09:15 → 10:15 Analysis of radiance observations

The primary purpose of this lecture is to explore the implications of the fact that satellites can only measure radiation at the top of the atmosphere and do not measure the geophysical variables we require for NWP (e.g. temperature, humidity and wind). The link between the atmospheric variables and the measured radiances is the radiative transfer equation - the key elements of which are discussed. It is shown how - with careful frequency selection - satellite measurements can be made for which the relationship to geophysical variables is greatly simplified. Despite these simplifications, it is shown that the extraction of detailed profile information from downward looking radiance measurements is a formally ill posed inverse problem.

Data assimilation is introduced as the solution to this inverse problem, where background information and satellite observations are combined to produce a best or optimal estimate of the atmospheric state. The main elements of the assimilation scheme (such as the chain of observation operators for radiances) and its key statistical inputs are examined. In particular, it is shown that incorrect specification of observation errors (R) and background errors (B) can severely limit the successful exploitation of satellite data.

By the end of this lecture you will:

• understand exactly what a satellite actually measures (radiance)
• appreciate the complex relationship between what is measured and what we wish to know for NWP
• how information is extracted from satellite measurements in data assimilation

Speaker: Tony McNally (ECMWF)

10:15 → 10:45 Coffee break

10:45 → 11:45 Assimilation Algorithms: (3) 4D-Var

Speaker: Sebastien Massart (ECMWF)

11:45 → 11:55 Comfort break

11:55 → 12:55 Assimilation Algorithms: (4) Ensemble Kalman filters

The aim of this lecture is to introduce the concept of the EnKF in the context of atmospheric data assimilation. Strengths and weaknesses of the algorithm will be discussed and results of the ECMWF implementation will be presented.

By the end of the lecture the participants should be able to:

• Describe the basic EnKF algorithm and its connections with the Kalman Filter;
• Discuss some of the advantages and the limitations of EnKF algorithms with respect to more established variational algorithms;
• Be aware of recent developments in hybrid variational-EnKF data assimilation

Speaker: Massimo Bonavita (ECMWF)

13:00 → 14:15 Lunch break

14:15 → 15:15 Bias correction methods

In this lecture, the variational bias correction scheme (VarBC) as used at ECMWF is explained. VarBC replaced the tedious job of estimating observation bias off-line for each satellite instrument or in-situ network by an automatic self-adaptive system. This is achieved by making the bias estimation an integral part of the ECMWF variational data assimilation system, where now both the initial model state and observation bias estimates are updated simultaneously.

By the end of the session you should be able to realize that:

• many observations are biased, and that the characteristics of bias varies widely between types of instruments
• separation between model bias and observation bias is often difficult
• the success of an adaptive system implicitly relies on a redundancy in the underlying observing system.

Speaker: Niels Bormann (ECMWF)

15:15 → 15:45 Coffee break

15:45 → 16:45 Tangent linear and adjoints

The goal of this lecture is to familiarise the student with the notion of tangent linear and adjoint models, and their use in variational data assimilation. A general overview of the current use of tangent linear and adjoint models in the ECMWF system will also be provided. Theoretical definitions and practical examples of tangent liner and adjoint models will be given. The student will be invited to work some simple tangent linear and adjoint derivations together with the instructor. A brief introduction to automatic differentiation software will also be given.

By the end of the session you should be able to:

• define what tangent linear and adjoint models are
• derive tangent linear and adjoint equations for a simple nonlinear equation
• describe the use of tangent linear and adjoint codes within the ECMWF's 4D-VAR system.

Speaker: Marcin Chrust (ECMWF)

16:45 → 17:15 Practical session: Tangent linear and adjoints

Speakers: Marcin Chrust (ECMWF), Sebastien Massart (ECMWF)

Wednesday, 26 February

09:15 → 10:15 Assimilation Algorithms: (5) Hybrid data assimilation methds

Speaker: Massimo Bonavita (ECMWF)

10:15 → 10:45 Coffee break

10:45 → 11:45 Quality control of observations

A single observation can under some conditions undermine the quality of a global analyses. The lecture will go through methods used to make the analysis more robust against oulier or wrong observations, with focus on variational quality control.

Speaker: Elias Holm (ECMWF)

11:45 → 11:55 Comfort break

11:55 → 12:55 Model error in data assimilation

In this lecture, the impact of model error on variational data assimilation will be presented. This lecture will introduce weak-constraint 4D-Var as a way to account for model error in the data assimilation process. Several examples of results from simplified implementations in the IFS will be shown.

By the end of the lecture the participants should be able to:

• describe the impact of model error on the data assimilation process,
• explain the difficulties in properly accounting for model error in data assimilation.

Speaker: Patrick Laloyaux (ECMWF)

13:00 → 14:15 Lunch break

14:15 → 15:15 Background error modelling in data assimilation

The background error is central to the performance of the analysis system and tells how much confidence to put in the best available forecast which is to be updated with new observations. The lecture will review how background errors are estimated and represented for current variational algorithms.

Speaker: Elias Holm (ECMWF)

15:15 → 15:45 Coffee break

15:45 → 17:15 Practical session: DA experiments with OOPS

Speakers: Marcin Chrust (ECMWF), Massimo Bonavita (ECMWF), Patrick Laloyaux (ECMWF), Sebastien Massart (ECMWF)

19:00 → 21:00 Social dinner at Côte Brasserie (self-funded)

Thursday, 27 February

09:15 → 10:15 Data assimilation diagnostics - forecast sensitivity

Speaker: Cristina Lupu (ECMWF)

Recording

10:15 → 10:45 Coffee break

10:45 → 11:45 Parametrization and data assimilation

This one-hour lecture will identify the challenges associated with the use of physical parametrizations in the context of four-dimensional variational data assimilation (4D-Var). The importance of the linearity constraint in 4D-Var and the methods to address it will be detailed. The set of linearized physical parametrizations used at ECMWF will be briefly presented. Examples of the use of physical parametrizations in variational data assimilation and its impact on forecast quality will be given.

By the end of the lecture, the students should be able:

• to tell why physical parametrizations are needed in data assimilation.
• to recognize the importance of the regularization of the linearized code

Speaker: Philippe Lopez (ECMWF)

11:45 → 11:55 Comfort break

11:55 → 12:55 Land data assimilation

The aim of these sessions is to understand the role of land surface data assimilation on medium range weather forecasts.

We will give an overview of the different approaches used to assimilate land surface data and to initialise model variables in NWP. We will present the current observing systems and describe the land data assimilation structure within ECMWF system.

By the end of the session you should be able to:

• identify the different observations used for snow and soil moisture data assimilation
• define land surface data assimilation approaches used for NWP
• describe the role of land surface data assimilation on medium-range weather forecasts

Speaker: Patricia de Rosnay (ECMWF)

13:00 → 14:15 Lunch break

14:15 → 15:15 DA experiments with OOPS

Speakers: Marcin Chrust (ECMWF), Massimo Bonavita (ECMWF), Patrick Laloyaux (ECMWF)

15:15 → 15:45 Coffee break

15:45 → 16:45 DA experiments with OOPS

Speakers: Marcin Chrust (ECMWF), Massimo Bonavita (ECMWF), Patrick Laloyaux (ECMWF), Sebastien Massart (ECMWF)

Friday, 28 February

09:15 → 10:15 Data assimilation of atmospheric composition

At ECMWF, atmospheric composition data are assimilated into the IFS as part of the Copernicus Atmospheric Monitoring Service. On a global scale, atmospheric composition represents the full state of the global atmosphere covering phenomena such as desert dust plumes, long-range transport of atmospheric pollutants or ash plumes from volcanic eruptions, but also variations and long-term changes in the background concentrations of greenhouse gases.

The aim of this lecture is to give an overview of the work that is carried out at ECMWF regarding the assimilation of atmospheric composition data, and to address why this is of interest and which special challenges are faced when assimilating atmospheric composition data.

By the end of the session you should:

• have some understanding of the work carried out at ECMWF to assimilate data of atmospheric composition

Speaker: Nicolas Bousserez (ECMWF)

10:15 → 10:45 Coffee break

10:45 → 11:45 Ocean data assimilation

This lecture provides an overview of a typical ocean data assimilation system for initialization and re-analyses application. The lecture uses as an example the ECMWF ocean data assimilation system, which is based the NEMOVAR (3Dvar FGAT). This will be used to discuss design of the assimilation cycle, formulation of error covariances, observations assimilated and evaluation procedure, among others.

By the end of the lecture students should be able to:

• describe the different components involved in an ocean data assimilation system
• list the commonalities and differences between ocean and atmosphere data assimilation
• describe the basics of the physical ocean observing system
• explain the essential multivariate relationships between ocean variables
• identify the limitations of the existing systems.

Speaker: Hao Zuo (ECMWF)

11:45 → 11:55 Comfort break

11:55 → 12:55 Reanalysis methods

The aim of this session is to understand how data assimilation can improve our knowledge of past weather over long time-scales. We will present recent advances that help capture changes over time in observing system networks, and project this variation in information content into uncertainty estimates of the reanalysis products. We will also discuss the applications of reanalysis, which generally put weather events into the climate context.

By the end of the session you should be able to:

• explain what are the goals of data assimilation in a reanalysis data assimilation system
• list the key aspects that require particular attention in reanalysis, as compared to numerical weather prediction
• describe the most common problems in reanalysis products

Speaker: Dinand Schepers (ecmwf)

13:00 → 14:15 Lunch break

14:15 → 15:15 Coupled data assimilation opportunities and challenges

At ECMWF, we are striving to move towards an Earth System approach to our data assimilation techniques. We currently have models not only of the atmosphere, but of the ocean, the land surface, sea ice, waves, and atmospheric composition. These systems interact with each other in different ways and all need to be initialised through the incorporation of observational data.

The aim of this lecture is to recognise the benefits and challenges associated with data assimilation in coupled models.

By the end of the lecture the participants should be able to:

• Recall the challenges associated with variational data assimilation in systems with different timescales and computer codes.
• Describe the benefits of having more consistently balanced coupled systems from coupled data assimilation.
• Explain the differences between weakly and strongly coupled data assimilation approaches.
• Discuss the various methods that are in use at ECWMF and explain the planned developments of the systems.

Speaker: Phil Browne (ECMWF)

15:15 → 15:45 Coffee break

15:45 → 16:45 Final discussion and Q&A

Speakers: Elias Holm, Lars Isaksen, Massimo Bonavita (ECMWF), Patrick Laloyaux (ECMWF)