Virtual Event: ECMWF-ESA Workshop on Machine Learning for Earth System Observation and Prediction

Europe/London
ECMWF

ECMWF

Virtual
Description

#AIforEOWS

Workshop motivation and description

Machine Learning/Deep Learning (ML/DL) techniques have made remarkable advances in recent years in a large and ever-growing number of disparate application areas, e.g. natural language processing, computer vision, autonomous vehicles, healthcare, finance and many others. These advances have been driven by the huge increase in available data, the increase in computing power and the emergence of more effective and efficient algorithms.

Earth System Observation and Prediction (ESOP) have arguably been latecomers to the ML/DL party, but interest is rapidly growing, and innovative applications of ML/DL tools are also becoming increasingly common in ESOP.

The interest of ESOP scientists in ML/DL techniques stems from different perspectives. From the observation side, the current and future availability of satellite-based Earth System measurements at high temporal and spatial resolutions and the emergence of entirely new observing systems made possible by ubiquitous internet connectivity (so called “Internet Of Things”) pose new challenges to established processing techniques and ultimately to our ability to make effective use of these new sources of information. ML/DL tools can potentially be useful to overcome some of these problems, for example in the areas of observation quality control, observation bias correction and the development of efficient observation operators and observation-based retrievals.

From a data assimilation perspective, ML/DL approaches are interesting because they can be typically framed as Bayesian inference problems using a similar methodological toolbox as the one used e.g. in variational data assimilation. It can be argued that some of the techniques already common in the data assimilation community (e.g. model error estimation, model parameter estimation) are effectively a type of ML/DL. The question is then, what lessons can the ESOP community learn from the methodologies and practices of the ML/DL community? Can we seamlessly integrate these new ideas into current data assimilation practices?

ML/DL solutions are also being explored for model identification, either in terms of the full forecast model or for specific model parametrizations which are computationally expensive and/or physically uncertain. How to best combine physical knowledge with the statistical knowledge provided by ML/DL approaches is an important and open question. Various types of machine learning technologies have also a rather long history of application in model interpretation and post-processing. The question of how ML/DL can help us extract more value from environmental forecasts is thus a relevant and current one to pose.

An important issue are the uncertainty characteristics of the ML results, and to understand better what physical relations they have been trained on. Many methodologies for both uncertainty quantification and for back-tracing ML output to input features have been proposed, but there is not yet a consensus view. Progress here is needed to improve and better understand reliability of ML results, which is crucial in an operational context.             

Workshop aims

In the application of ML/DL techniques to ESOP there are still many unanswered questions. The aim of the workshop was to appraise the state of the art of the application of ML/DL techniques to ESOP, to identify the main issues that need to be solved for further progress, and to make a start on charting ways forward. Presenters of the longer talks covered not just their own work but also to gave a general overview of the subject. Discussions were facilitated by parallel working groups where the main issues were discussed in more detail. The output of the workshop is in the form of working group reports, to be summarised in a technical memorandum or paper.

 

Events team
    • 11:00 11:15
      Welcome and introduction - ECMWF 15m
      Speaker: Andy Brown (ECMWF)
    • 11:15 11:30
      Welcome and introduction - ESA 15m
      Speaker: Pierre-Philippe Mathieu (ESA)
    • 11:30 13:00
      Session 1: ML for the Earth System - Setting the scene
      Convener: Massimo Bonavita (ECMWF)
      • 11:30
        Accelerating and Explaining Earth System Process Models with Machine Learning 45m

        Earth scientists have developed complex models to represent the interactions among small particles in the atmosphere, such as liquid water droplets and volatile organic compounds. These complex models produce significantly different results from their simplified counterparts but are too computationally expensive to run directly within weather and climate models. Machine learning emulators trained on a limited set of runs from the complex models have the potential to approximate the results of these complex models at a far smaller computational cost. However, there are many open questions about the best approaches to generating emulation data, training ML emulators, evaluating them both offline and within Earth System Models, and explaining the sensitivities of the emulator.

        In this talk, we will discuss the development of machine learning emulators for warm rain bin and superdroplet microphysics as well as the Generator of Explicit Chemistry and Kinetics of Organics in the Atmosphere (GECKO-A) model. For microphysics, we ran CAM6 for 2 years with the cloud-to-rain processes handled by either the bin or superdroplet scheme and saved the inputs and outputs of the scheme globally at selected time steps. We used this information to train a set of neural network emulators. The neural network emulators are able to approximate the behavior of the bin and superdroplet schemes while running within CAM 6 but at a computational cost close to the bulk MG2 scheme. Machine learning interpretation techniques also reveal the relative contributions and sensitivities of the different inputs to the emulator. We will discuss lessons learned about both the training process and the resulting model climate.

        For GECKO-A, we ran the model forward in time with multiple sets of fixed atmospheric conditions and different precursor compounds. Then we trained fully connected and recurrent neural networks to emulate GECKO-A's. We tested the different machine learning methods by running them forward in time with both fixed and varying atmospheric conditions. We plan to incorporate the GECKO-A emulator into a full 3D atmospheric model to evaluate how the transitions between precursors, gases, and aerosols evolve spatio-temporally. We will also discuss lessons learned from working with this dataset and the challenges of problem formulation and evaluation. Finally, we are releasing data from both of these domains as machine learning challenge problems to encourage further innovation in this area by the broader Earth Science and Machine Learning communities.

        Speaker: David Gagne (National Center for Atmospheric Research)
      • 12:15
        Learning from earth system observations: machine learning or data assimilation? 45m
        Speaker: Alan Geer (ECMWF)
    • 13:00 14:00
      Lunch break 1h
    • 14:00 15:30
      Session 2: ML for Earth System Observations
      Convener: Bertrand Le Saux (ESA)
      • 14:00
        The Rise of AI for EO 30m

        The Rise of AI for EO

        AI and EO is a marriage made in heaven!,
        Today, AI is in the midst of a true “renaissance”, driven by Moore’s Law* and now super-fed by Big Data. We believe AI has a huge, but still largely untapped potential for EO technology. In a sense, we are now at an inflection point, at a kind of crossroads of opportunities, whereby on the one hand AI is becoming one of the most transformative technologies of the century, while on the other hand European EO capability is delivering a totally unique and comprehensive and dynamic picture of the planet, thereby generating big open data sets to be explored by AI.
        Due to the rapid increase in the volume and variety of EO data sources, AI techniques become increasingly needed to analyse the data in an automatic, flexible and scalable way. Today, EO data remotely sensed from space are particularly suited - but at the same time challenging - for AI processing as they are:

        • Big in size and volume with Terrabytes of data routinely streamed daily from space, which at the end needs to be turned into “small” actionable information.

        • Diverse including a variety of sensors from optical (multispectral and hyperspectral) to radar data. Up to now, AI has been applied mainly to optical imagery, in particular at very high resolution by use of traditional Computer Vision techniques (using mainly RGB bands). More work is needed to make full use of all available spatial, temporal and spectral information of EO data at the global scale. e.g. exploiting the full information of the “complex nature” of radar data within AI schemes, including information on the amplitude, frequency, phase or polarization of the collected radar echoes,

        • Complex and physically-based capturing dynamic features of a highly non-linear coupled Earth System. This goes well beyond merely recognising cats and dogs, in images where a wide variety of training datasets are available (such as ImageNet) and also calls to integrate physical principles into the statistical approach.

        Machines algorithms powered by AI are therefore critically needed to accelerate “insight” into the data but always in combination with domain experts vital to properly interpret the statistical correlations and data. The intersection of AI and EO remains an emergent field, but a rapidly growing one. There has been a lot of work on ML applied to EO over the past decade, but with the rapid emergence of Deep Learning, the field has been growing rapidly, as illustrated by the increase in the number of publications. However, although being very powerful, DL techniques of course suffer from their own inherent limitations such as being data hungry while lacking transparency and not being able to distinguish causation.

        In this talk, we will present some of the ESA activities on AI4EO, and in particular the Φ-lab, in order to better understand how to harness the full power of Artificial Intelligence for Earth Observation (AI4EO).

        Speaker: Pierre-Philippe Mathieu (ESA)
      • 14:30
        Leveraging Modern AI/ML Techniques in NWP Including Data Fusion/Assimilation 30m

        In this presentation we will attempt to demonstrate that adopting modern AI techniques, including ML, has the potential to optimize the Numerical Weather Prediction (NWP) process and ultimately, the Earth System Model of the future, when all components of the Environment become coupled at the assimilation and forecasting levels. In this presentation, we will highlight some of the results of an incubator effort done in NOAA’s center for satellite applications and research, where the initial work has focused on the aspect of satellite data exploitation. The study covers (1) instrument calibration error correction, (2) pre-processing of satellite data and quality control, (3) parameterization of radiative transfer modeling, (4) data fusion and data assimilation, as well as (5) post-processing of NWP outputs to correct for systematic and geophysically-varying errors and finally (6) Spatial and Temporal resolutions enhancement.
        We will assess the quality of the analyses obtained using an entirely AI-based system, by checking the inter parameters correlation matrix, the spatial variability spectrum and the mass conservation. A first step shown in this study is to ensure that we can perform similar steps currently done in NWP without loss of accuracy and without introducing artefacts, but with significant efficiency increase. This will allow us to assimilate and exploit a higher volume of data and to begin exploiting other sources of environmental data such as IoT, smallsats, near-space platforms, etc.
        We will also discuss the potential of AI/ML beyond the efficiency aspect, and the limitations that should be circumvented in order to achieve the full potential of AI/ML in NWP.

        Speaker: Sid Boukabara (NOAA)
      • 15:00
        Exploring the Frontiers of Deep Learning for Earth System Observation and Prediction 30m

        In the last few years, the Earth System Science community has rapidly come to adopt machine learning as a viable and useful approach for doing science, and it has been applied to a surprising diverse array or problems, with an ever-increasing degree of success. However, despite our progress, there remains much to learn. In its current form, ML may be best viewed as an alternative and complimentary approach to traditional software development. However, we have a great deal to learn about how to best employ and deploy these tools, which require a new and more ‘orgnanic’ process. Furthermore, there are many challenges specific to science that need to be further explored including: massive data labelling, enforcing physical constraints, uncertainty quantification, explainability, reliability, AI safety, data movement problems, and the need for benchmarks. Finally, it is my opinion that ML has the potential to grow far beyond its current limits, as a wide range of possibilities open up when both the software and the software-engineer and composed of code. In this presentation, we will explore these issues and the survey cutting-edge research that is taking place on the frontiers of ML sampling from topics like: self-supervision, continual learning, online-learning, human in the loop, AutoML, neural architecture search, expanded use of GANs, loss-function learning, spatio-temporal prediction, equation identification, ODE learning, differential programming, amongst others.

        Speaker: David Hall (NVIDIA)
    • 15:30 16:00
      Coffee break 30m
    • 16:00 17:00
      Session 2 (cont.): ML for Earth System Observations
      Convener: Peter Dueben (ECMWF)
      • 16:00
        On the Interpretation of Neural Networks Trained for Meteorological Applications 30m

        Neural networks (NNs) have emerged as a promising tool in many meteorological applications. While they perform amazingly well at many complex tasks neural networks are generally treated as a black box, i.e. it is typically considered too difficult a task to understand how they work. However, a better understanding of neural networks would have many advantages. A better understanding could provide important information for the design and improvement of NNs, increase trust in NN methods especially for operational use, and even enable us to gain new scientific knowledge from NN models. Fortunately, progress in the computer science field of Explainable AI (XAI) is yielding many new methods that can help scientists gain a better understanding of a NN's inner workings. For example, neural network visualization methods, such as Layer-Wise Relevance propagation (LRP), can help meteorologists extract strategies the neural network uses to make its decisions. Furthermore, viewing the problem more from a meteorologist’s perspective, another important tool is synthetic experiments, where we design synthetic inputs that represent specific meteorological scenarios and test the response of the neural network to those inputs. We present some of these techniques and demonstrate their utility for sample applications. For example, we show how these methods can be used to identify strategies used by a neural network trained to emulate radar imagery from GOES satellite imagery. Finally, we look at the process of gaining insights into neural networks for meteorological applications as a whole – and highlight that it is an iterative, scientist-driven discovery process, that incorporates old fashioned methods of hypothesis generation and testing and experimental design. In this context NN visualization tools simply provide additional tools to assist the meteorologist in this endeavor, and are comparable to a biologist's use of a microscope as tool for scientific analysis.

        Speaker: Dr Imme Ebert-Uphoff (Colorado State University)
      • 16:30
        Deep Hashing for Scalable Remote Sensing Image Retrieval in Large Archives 30m

        With the unprecedented advances in the satellite technology, recent years have witnessed a significant increase in the volume of remote sensing (RS) image archives (Demir and Bruzzone 2016). Thus, the development of efficient and accurate content based image retrieval (CBIR) systems in massive archives of RS images is a growing research interest in RS. CBIR aims to search for RS images of the similar information content within a large archive with respect to a query image. To this end, CBIR systems are defined based on two main steps: i) image description step (which characterizes the spatial and spectral information content of RS images); and ii) image retrieval step (which evaluates the similarity among the considered descriptors and then retrieve images similar to a query image in the order of similarity).

        Due to the significant growth of RS image archives, an image search and retrieval through linear scan (which exhaustively compares the query image with each image in the archive) is computationally expensive and thus impractical. This problem is also known as large-scale CBIR problem. In large-scale CBIR, the storage of the data is also challenging as RS image contents are often represented in high-dimensional features. Accordingly, in addition to the scalability problem, the storage of the image descriptors also becomes a critical bottleneck. To address these problems, approximate nearest neighbour (ANN) search has attracted extensive research attention in RS. In particular, hashing based ANN search schemes have become a cutting-edge research topic for large-scale RS image retrieval due to their high efficiency in both storage cost and search /retrieval speed. Hashing methods encode highdimensional image descriptors into a low-dimensional Hamming space where the image descriptors are represented by binary hash codes. By this way, the (approximate) nearest neighbours among the images can be efficiently identified based on the Hamming distance with simple bit-wise operations. In addition, the binary codes can significantly reduce the amount of memory required for storing the content of images. Traditional hashing-based RS CBIR systems initially extract hand-crafted image descriptors and then generate hash functions that map the original high-dimensional representations into low-dimensional binary codes, such that the similarity to the original space can be well preserved (Fernandez-Beltran et al. 2020). Thus, descriptor generation and hashing processes are independently applied, resulting in sub-optimal hash codes. Success of DNNs in image feature learning has inspired research on developing DL based hashing methods, which can simultaneously learn the image representation and the hash function with proper loss functions.

        This paper aims at presenting recent advances in CBIR systems in RS for fast and accurate information discovery from massive data archives. Initially, we analyse the limitations of the traditional CBIR systems that rely on the hand-crafted RS image descriptors applied to exhaustive search and retrieval problems. Then, we focus our attention on the advances in RS CBIR systems for which the DL models are at the forefront. In particular, we present the theoretical properties of the deep hashing based CBIR systems that have high time-efficient search capability within huge data archives (Roy et al. 2020). A particular attention is given to the metric learning and the graph structure driven deep hashing networks for scalable and accurate content-based indexing and retrieval of RS images. Finally, the most promising research directions in RS CBIR are discussed together with the description of the BigEarthNet (which is a new large-scale Sentinel-2 multispectral benchmark archive introduced to advance deep learning studies in RS) (Sumbul et al. 2019).

        REFERENCES

        Demir, B., Bruzzone, L., 2016. Hashing Based Scalable Remote Sensing Image Search and Retrieval in Large Archives. IEEE Transactions on Geoscience and Remote Sensing 54 (2):892-904.

        Fernandez-Beltran, R., Demir, B., Pla, F. and Plaza, A., 2020. Unsupervised Remote Sensing Image Retrieval using Probabilistic Latent Semantic Hashing. IEEE Geoscience and Remote Sensing Letters, doi: 10.1109/LGRS.2020.2969491.

        Roy, S., Sangineto, E., Demir, B., Sebe, N., 2020. Metric-Learning based Deep Hashing Network for Content Based Retrieval of Remote Sensing Images. IEEE Geoscience and Remote Sensing Letters, in press.

        Sumbul, G., Charfuelan, M., Demir, B., Markl, V., 2019. BIGEARTHNET: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding. IEEE International Conference on Geoscience and Remote Sensing Symposium, pp. 5901-5904, Yokohama, Japan.

        Speaker: Begüm Demir (TU Berlin)
    • 17:00 18:30
      Poster session
    • 08:55 10:30
      Session 2 (cont.): ML for Earth System Observations
      Convener: Susanne Mecklenburg (ESA)
      • 09:00
        Using Machine Learning to advance hour-scale heavy rain forecast with high resolution ECMWF Global Model and Local Meso-scale Model Forecasts 30m

        Numerical weather predictions (NWP) and observational systems have improved greatly both on quantity and quality in recent years. Combining expertise knowledge and machine learning (ML) to extract and synthesize valuable information from these "big data" is expected to one of present major challenges and opportunities to improve the severe weather forecast. A ML correction method based on the physical ingredients was developed in this study to improve the hour-scale flash heavy rain forecast, which contains a key factors extraction technique by blending ML feature engineering and expert knowledge , and several ML models.
        The summer flash heavy rain events in the Beijing-Tianjin-Hebei region of China are studied. Firstly, the hourly features were analyzed based on ten years observations. Then a comprehensive verification was conducted with the operational forecasts including the high resolution ECMWF global model and mulitple local forecasts (SMS 9 km and 3km, GRAPES 9km and 3km, BJ-RUC 9km and 3km) , and those with better performances are chosen as ML Components. Data in 2018 and 2019 summer are used for training and test the ML models. Here, we report on initial results.
        More than 200 thermo-dynamical features were selected by expertise. Multiple ML feature engineering techniques including the correlation coefficient, the mutual information, and the embedded methods are combined and used to furtherly select features and get rid of the redundancy. The results show that the finer the scale is the more important the physical features are. There is a good correlation between 6hr (or 12hr) accumulated precipitation and the estimated precipitation. However, the direct rainfall forecasts become less important when it comes to the 3hr (or 1hr) accumulations, and the contributions of thermo-dynamical features are significantly enhanced, in particular the moisture at low level and the wind field near surface. This means that the uncertainty of model rain is increasing with the finer of the forecast, which is consistent with the professional cognition. Therefore, by using forecasted physical ingredients with more reliability and the rapidly updating observations may be potential to improve the hour-scale flash heavy rain forecast. Multiple ML models including ET (Extra Tree)/RF (Radom Forest)/Catboost were trained, and then a secondary ensemble was conducted. The results show that the ML correction based on physical features can significantly improve the forecast on 3-hour cumulative precipitation compared to the ECMWF and local meso-scale model. Taking the threshold of 10mm rainfall as an example, the Ts score of ET model is 0.27, which is about 35% higher than those of the numerical forecasts (0.2 or less); the missing alarm rate is about 26% lower than ECMWF forecast, and the false alarm rate is about 28% lower than the local SMS model. A case study of flash heavy rain in August 2019 shows that both the intensity and the heavy rain area are significantly improved in the ML correction system.

        Speaker: Prof. Qi Zhong (China Meteorological Administration Training Centre)
      • 09:30
        Artificial Intelligence in ESA Contribution to the United Nations Framework Convention on Climate Change (UNFCCC) : An Overview 30m

        Climate change is arguably the greatest challenge facing humankind in the twenty-first century. The United Nations Framework Convention on Climate Change (UNFCCC) provides the vehicle for multilateral action to combat climate change and its impacts on humanity and ecosystems. In order to make decisions on climate change mitigation and adaptation, the UNFCCC requires a systematic monitoring of the global climate system.

        The objective of the ESA Climate Change Initiative (CCI) programme is to realise the full potential of the long term global EO archive that ESA together with its Member States have established over the last 35 years, as a significant and timely contribution to the climate data record required by the UNFCCC.

        Since 2010 the programme has contributed to a rapidly expanding body of scientific knowledge on 22 Essential Climate Variables (ECVs), through the production of Climate Data Records. Although varying across geophysical parameters, ESA Climate Data Records follow a community-driven data standards, so facilitating their blending and application.

        AI has played a pivotal role in the production of these Climate Data Records. Eleven CCI projects - Aerosol CCI, Cloud CCI, Fire CCI, Greenhouse Gases CCI, Ocean Colour CCI, Sea Level CCI, Soil Moisture CCI, High Resolution Landcover CCI, Biomass CCI, Permafrost CCI, and Sea Surface Salinity CCI - have applied AI in their data record production and research, or identified specific AI usage for their research roadmap.

        The use of AI in these CCI projects is varied, for example to detect burned areas in Fire CCI, to retrieve dust Aerosol Optical Depth from thermal infrared spectra in Aerosol CCI, and pixel classification via a custom built AI tool in Ocean Colour CCI. Moreover, the ESA climate community have identified climate science gaps in context to ECVs with the potential for meaningful advancement through AI.

        Speakers: Eduardo Pechorro (ESA Climate Office), Amy Campbell (National Oceanography Centre Graduate School)
      • 10:00
        Neural network products in land surface data assimilation 30m

        Since July 2019, a SMOS neural network soil moisture product has been assimilated into the ECMWF operational soil moisture simplified extended Kalman filter (SEKF) as part of the land data assimilation system. There are two versions of SMOS neural network soil moisture products produced routinely at ECMWF. The first has been trained on the SMOS level 2 soil moisture product and is delivered to ESA. The second is trained on the ECMWF operational model soil moisture values and this is the one that is then assimilated into the SEKF. The neural network products will be compared, with differences due to the different training datasets used and despite the same SMOS observation inputs. Also, assimilation results for the product trained on the ECMWF model soil moisture will be presented.

        In the context of the EUMETSAT HSAF project, recent work to develop neural network methods linking ASCAT backscatter measurements to soil moisture from ERA5, both in a retrieval and as part of a forward model, will also be introduced.

        Speaker: Peter Weston (ECMWF)
    • 10:30 11:00
      Coffee break 30m
    • 11:00 13:00
      Session 3: ML for Data Assimilation
      Convener: Alan Geer (ECMWF)
      • 11:00
        Big Data, Big Computation, and Machine Learning in Numerical Weather Prediction 30m

        At RIKEN, we have been exploring a fusion of big data and big computation, and now with AI techniques and machine learning (ML). The new Japan’s flagship supercomputer “Fugaku”, ranked #1 in the most recent TOP500 list (https://www.top500.org/) in June 2020, is designed to be efficient for both double-precision big simulations and reduced-precision machine learning applications, aiming to play a pivotal role in creating super-smart “Society 5.0.” Our group in RIKEN has been pushing the limits of numerical weather prediction (NWP) through two orders of magnitude bigger computations by taking advantage of the previous Japan’s flagship supercomputer named “K computer”. The efforts include ensemble Kalman filter experiments with 10240 ensemble members and 100-m-mesh, 30-second-update “Big Data Assimilation” by fully exploiting the novel phased array weather radar. Now with the new “Fugaku” in mind, we have been exploring ideas for fusing Big Data Assimilation and AI. The ideas include fusion of data-driven precipitation nowcasting with process-driven NWP, NWP model acceleration using neural networks (NN), applying ML to satellite and radar operators in data assimilation (DA), and NWP model’s systematic error identification and correction by NN. The data produced by NWP models become bigger and moving around the data to other computers for ML may not be feasible. Having a next-generation computer like “Fugaku”, good for both big NWP computation and ML, may bring a breakthrough toward creating a new methodology of fusing data-driven (inductive) and process-driven (deductive) approaches in meteorology. This presentation will provide general perspectives toward future developments and challenges in NWP, with some specific research examples of DA-AI fusion at RIKEN.

        Speaker: Takemasa Miyoshi (RIKEN)
      • 11:30
        Artificial Neural Network at the service of Data Assimilation (and vice versa) 30m

        Can Artificial Neural Network (NN) lean (and/or replace) a Data Assimilation (DA) process? What would be the effect of this approach?

        DA is the Bayesian approximation of the true state of some physical systems at a given time by combining time-distributed observations with a dynamic model in an optimal way. NN models can be used to learn the assimilation process in different ways. In particular, Recurrent Neural Networks can be efficiently applied for this purpose.

        NNs can approximate any non-linear dynamical system. How DA can be used to improve the performance of a NN?

        DA can be used, for example, to improve the accuracy of a NN by including information provided by external data. In general, DA can be used to ingest meaningful data in the training process of a NN.

        We show the effectiveness of these methods by case studies and sensitivities studies.

        Speaker: Rossella Arcucci (Imperial College London)
      • 12:00
        Using machine learning and data assimilation to learn both dynamics and state 30m

        The recent introduction of machine learning techniques in the field of numerical geophysical prediction has expanded the scope so far assigned to data assimilation, in particular through efficient automatic differentiation, optimisation and nonlinear functional representations. Data assimilation together with machine learning techniques, can not only help estimate the state vector but also the physical system dynamics or some of the model parametrisations. This addresses a major issue of numerical weather prediction: model error. I will discuss from a theoretical perspective how to combine data assimilation and deep learning techniques to assimilate noisy and sparse observations with the goal to estimate both the state and dynamics, with, when possible, a proper estimation of residual model error. I will review several ways to accomplish this using for instance offline, variational algorithms and online, sequential filters. The skills of these solutions with be illustrated on low-order chaotic dynamical systems. Finally, I will discuss how such techniques can enhance the successful techniques of data assimilation. Examples will be taken from collaborations with J. Brajard, A. Carrassi, L. Bertino, A. Farchi, M. Bonavita, P. Laloyaux, and Q. Malartic.

        Speaker: Marc Bocquet (Ecole des Ponts ParisTech)
      • 12:30
        Combining data assimilation and machine learning to emulate hidden dynamics and to infer unresolved scale pametrisation. 30m

        A novel method based on the combination of data assimilation and machine learning is introduced. The combined approach is designed for emulating hidden, possibly chaotic, dynamics and/or to devise data-driven parametrisations of unresolved processes in dynamical numerical models.
        The method consists in applying iteratively a data assimilation step, here ensemble Kalman filter or smoother, and a neural network. Data assimilation is used to effectively handle sparse and noisy data. The output analysis is spatially complete and is used as a training set by the neural network. The two steps can then be repeated iteratively.
        We will show the use of this combined DA-ML approach in two set of experiments. In the first one the goal is to infer a full surrogate model. Here we carry experiments using the chaotic 40-variables Lorenz 96 model and show that the surrogate model achieves high forecast skill up to two Lyapunov times, it has the same spectrum of positive Lyapunov exponents as the original dynamics and the same power spectrum of the more energetic frequencies. In this context we will also illustrate the sensitivity of the method to critical setup parameters: the forecast skill decreases smoothly with increased observational noise but drops abruptly if less than half of the model domain is observed.
        In the second set of experiments, the goal is to infer unresolved-scale parametrization. Data assimilation is applied to estimate the full state of the system from a truncated model. The unresolved part of the truncated model is viewed as model errors in the DA system. In a second step, ML is used to emulate the unresolved part, a predictor of model error given the state of the system. Finally, the ML-based parametrisation model is added to the physical core truncated model to produce a hybrid model.
        Experiments are carried out using the two-scale Lorenz model and the reduced-order coupled atmosphere-ocean model MAOOAM. The DA component of the proposed approach relies on an ensemble Kalman filter while the ML parametrisation is represented by a neural network. We will show that in both cases the hybrid model yields better forecast skills than the truncated model, and its attractor resembles much more the original system’s attractor than the truncated model.

        Speaker: Alberto Carrassi (University of Reading and NCEO (UK); University of Utrecht (NL))
    • 13:00 14:00
      Lunch 1h
    • 14:00 15:30
      Session 3 (cont.): ML for Data Assimilation
      Convener: Peter Lean (ECMWF)
      • 14:00
        Data Assimilation and Machine Learning Science at ECMWF 30m

        Model error is one of the main obstacles to improved accuracy and reliability in state-of-the-art analysis and forecasting applications, both in Numerical Weather Prediction and in climate prediction conducted with comprehensive high resolution general circulation models. In a data assimilation framework, recent advances in the context of weak constraint 4D-Var have shown that it is possible to estimate and correct for a large fraction of model error in parts of the atmosphere. This has been demonstrated in the stratosphere where the current global observing system is sufficiently dense and homogeneous.
        The recent explosion of interest in Machine Learning / Deep Learning technologies has been driven by their remarkable success in disparate application areas. This raises the question of whether model error estimation and correction in operational NWP and climate prediction can also benefit from these techniques. Based on recent results (Bonavita and Laloyaux, 2020) we aim to start to provide answers to these questions. Specifically, we show that Artificial Neural Networks (ANN) can reproduce the main results obtained with weak constraint 4D-Var in the operational configuration of the IFS model of ECMWF. More interestingly, we show that the use of ANN models inside the weak-constraint 4D-Var framework has the potential to extend the applicability of the weak constraint methodology for model error correction to the whole atmospheric column. Finally, we discuss the potential and limitations of the Machine Learning / Deep Learning technologies in the core NWP tasks. In particular, we reconsider the fundamental constraints of a purely data driven approach to forecasting and provide a view on how to best integrate Machine Learning technologies within current data assimilation and forecasting methods

        Speaker: Dr Massimo Bonavita (ECMWF)
      • 14:30
        Joint learning of variational data assimilation models and solvers 30m

        This paper addresses variational data assimilation from a learning point of view. Data assimilation aims to reconstruct the time evolution of some state given a series of observations, possibly noisy and irregularly-sampled. Using automatic differentiation tools embedded in deep learning frameworks, we introduce end-to-end neural network (NN) architectures for variational data assimilation. It comprises two key components: a variational model and a gradient-based solver both implemented as neural networks. The latter exploits ideas similar to meta-learning and optimizer learning. A key feature of the proposed end-to-end framework is that we may train the NN models using both supervised and unsupervised strategies. Especially, we may evaluate whether the minimization of the classic definition of variational formulations from ODE-based or PDE-based representations of geophysical dynamics leads to the best reconstruction performance.

        We report numerical experiments on Lorenz-63 and Lorenz-96 systems for a weak constraint 4D-Var setting with noisy and irregularly-sampled/partial observations. The key features of the proposed neural network framework is two-fold: (i) the learning of fast iterative solvers, which can reach the same minimization performance as a fixed-step gradient descent with only a few tens of iterations, (ii) the significant gain in the reconstruction performance (a relative gain greater than 50%) when considering a supervised solver, i.e. a solver trained to optimize the reconstruction error rather than to minimize the considered variational cost. In this supervised setting, we also show that the joint learning of the variational prior and of the solver significantly outperform NN representations. Intriguingly, the trained representations leading to the best reconstruction performance may lead to significantly worse short-term forecast performance. We believe these results may open new research avenues for the specification of assimilation models and solvers in geoscience, including the design of observation settings to implement learning strategies.

        Speaker: Ronan Fablet (IMT Atlantique)
      • 15:00
        Hyperparameter learning in data assimilation systems 30m

        Data assimilation combines different information sources using a quantification of the uncertainty of each source to weight them. Therefore, a proper consideration of the uncertainty of observations and model states is crucial for the performance of the data assimilation system. Expert knowledge and expensive offline fine tuning experiments have been used in the past to determine the set of hyperparameters that define aspects of the prior, model error and observational uncertainties and in particular their covariances. In recent years, there is a paradigm shift with several Bayesian and maximum likelihood methods that attempt to infer these hyperparameters within the data assimilation system. In this talk, I will give the foundational basis of these methods which rely on the assumption of slow variations of the hyperparameters compared to the latent state variables. An overview of maximum likelihood methods including the online and batch expectation-maximization algorithm, gradient-based optimization and a Bayesian hierarchical method based on nested ensemble Kalman filters will be discussed. Finally, some experiments to estimate stochastic parameterizations that compare the methods in a proof-of-concept dynamical system will be shown.

        Speaker: Manuel Pulido (Universidad Nacional del Nordeste)
    • 15:30 16:00
      Coffee break 30m
    • 16:00 18:00
      Session 3 (cont.) and Session 4: ML for Data Assimilation and ML for Product Development
      Convener: Marcin Chrust (ECMWF)
      • 16:00
        Bayesian Deep Learning for Data Assimilation 30m

        Deep Learning has been shown to be efficient for many data-assimilation problems, and many deep learning methods have been used for this purpose. However, these applications typically focus on obtaining a best estimate of the atmospheric state, while providing a proper uncertainty estimate is as least as important. This is even more problematic as deep learning is prove to overfitting as the number of parameters to be estimated is always larger than the output dimension. Ad hoc techniques like weight decay and drop out have been proposed to avoid the overfitting, and indeed they do provide a regularisation of the problem, but the methods cannot be seen consistently as prior information (even though this has been claimed in the literature).

        In this presentation I will discuss the problem, show why standard techniques for uncertainty quantification are not appropriate, and formulate a principled way to treat uncertainty quantification in Deep Learning. Existing ideas for Bayesian Deep Learning have been shown to scale badly with dimension, so special interest will be given to scalability, exploring existing techniques from data assimilation. Since it is unlikely that the full data-assimilation system will be abandoned in favour of deep learning, the incorporation of deep learning with uncertainty quantification into an existing data-assimilation structure will also be discussed.

        Speaker: Prof. Peter Jan van Leeuwen (Colorado State University)
      • 16:30
        Machine Learning for Applied Weather Prediction 30m

        Weather forecasting has progressed from being a very human-intensive effort to now being highly enabled by computation. The first big advance was in terms of numerical weather prediction (NWP), i.e., integrating the equations of motion forward in time with good initial conditions. But the more recent improvements have come from applying machine-learning (ML) techniques to improve forecasting and to enable large quantities of machine-based forecasts.

        One of the early successes of the use of AI in weather forecasting was the Dynamical Integrated foreCast (DICast®) System. DICast builds on several concepts that mimic the human forecasting decision process. It leverages the NWP model output as well as historical observations at the site for the forecast. It begins by correcting the output of each NWP model according to past performance. DICast then optimizes blending of the various model outputs, again building on the past performance record. DICast has been applied to predict the major variables of interest (such as temperature, dew point, wind speed, irradiance, and probability of precipitation) at sites throughout the world. It is typical for DICast to outperform the best individual model by 10-15%. One advantage of DICast is that it can be trained on a relatively limited dataset (as little as 30 to 90 days) and updates dynamically to include the most recent forecast information. The gridded version of this system, the Graphical Atmospheric Forecast System (GRAFS) can interpolate forecasts to data-sparse regions.

        DICast and other machine-learning methods have been applied by the National Center for Atmospheric Research (NCAR) to various needs for targeted weather forecasts. Such applications include hydrometeorological forecasting for agricultural decision support; forecasting road weather to enhance the safety of surface transportation; forecasting movement of wildland fires; and predicting wind, and solar power for utilities and grid operators to facilitate grid integration. NCAR has found AI/ML to be an effective for postprocessing for these and many applications and it has become part of any state-of-the-science forecasting system.

        An example of using multiple AI methods for targeted forecasts is predicting solar power production. AI methods are used in both Nowcasting and in forecasting for Day-Ahead grid integration. DICast is one of the methods that blends input from multiple forecast engines. For the very short ranges, NCAR developed a regime-based solar irradiance forecasting system. This system uses k-means clustering to identify cloud regimes, then applies a neural network to each regime separately. This system was shown to out-predict other methods that did not utilize regime separation. NCAR is currently designing a comprehensive wind and solar forecasting system for desert regions that combines NWP with machine-learning approaches. For the first few hours, ML approaches leverage historical and real-time data. DICast improves on the NWP forecasts. The meteorological variables are converted to power using a model regression tree. The Analog Ensemble ML approach further improves the forecast and provides probabilistic information. This systems approach that leverages the best of NWP with ML shows success at providing a seamless forecast across multiple time scales for use-inspired applications.

        Speaker: Dr Sue Ellen Haupt (National Center for Atmospheric Research)
      • 17:00
        Deep Unsupervised Learning for Climate Informatics 30m

        Despite the scientific consensus on climate change, drastic uncertainties remain. Crucial questions about regional climate trends, changes in extreme events, such as heat waves and mega-storms, and understanding how climate varied in the distant past, must be answered in order to improve predictions, assess impacts and vulnerability, and inform mitigation and sustainable adaptation strategies. Machine learning can help answer such questions and shed light on climate change. I will give an overview of our climate informatics research, focusing on semi- and unsupervised deep learning approaches to studying rare and extreme events, and downscaling temperature and precipitation.

        Speaker: Claire Monteleoni (University of Colorado)
      • 17:30
        Deep Learning for Post-Processing Ensemble Weather Forecasts 30m

        Quantifying uncertainty in weather forecasts is critical, especially for predicting extreme weather events. This is typically accomplished with ensemble prediction systems, which consist of many perturbed numerical weather simulations, or trajectories, run in parallel. These systems are associated with a high computational cost and often involve statistical post-processing steps to inexpensively improve their raw prediction qualities. We propose a mixed model that uses only a subset of the original weather trajectories combined with a post-processing step using deep neural networks. These enable the model to account for non-linear relationships that are not captured by current numerical models or post-processing methods.

        Applied to global data, our mixed models achieve a relative improvement in ensemble forecast skill (CRPS) of over 13%. Further, we demonstrate that this improvement is even more significant for extreme weather events on selected case studies. We also show that our post-processing can use fewer trajectories to achieve comparable results to the full ensemble. This can enable reduced computational costs for ensemble prediction systems or allow higher resolution trajectories to be run within operational deadlines, resulting in more accurate raw ensemble forecasts.

        Speaker: Nikoli Dryden (ETH Zurich)
    • 09:00 10:30
      Session 4 (cont.): ML for Product Development
      Convener: Bertrand Le Saux (ESA/ESRIN)
      • 09:00
        Automatic detection of weather events in high-resolution NWP outputs 30m

        Since the number of available NWP forecasts is rapidly increasing, especially with the development of ensemble prediction systems, there is a need to develop innovative forecast products that provide a synthetic view of the weather situation and potential risks. A promising option is to identify different weather patterns in NWP outputs, that can then be used to delimit areas of interest, to provide a diagnostic of occurrence of a given event, or to issue pattern-based probability maps. The detection of weather objects has been performed for several years, mainly with algorithmic approaches based on a set of simple rules. In recent years, machine learning and deep learning algorithms have provided powerful tools to segment objects, that can overcome some limitations of standard algorithms and allow for the detection of more complex and finer-scale objects. In this presentation we show that the well-known U-Net convolutional neural network, a typical encoder-decoder architecture with skip connections, can be successfully applied to the high-resolution Arome model outputs for the detection of several weather features with different spatial scales, including continuous and intermittent rainfall areas, weather fronts, and two extreme weather events, tropical cyclones and bow echoes. The performance of these detections strongly relies on the availability and accuracy of large training and testing datasets. Since no off-the-shelf data are available, a time-consuming human labelling exercise has been performed for each pattern considered. In order to extend the application of our U-Net to a wider range of outputs, without increasing labelling and training steps, transfer learning has been successfully used. It is shown in particular that a network trained on a specific geographic region can be applied to other domains, and networks trained on NWP outputs can properly detect similar objects in corresponding observations.

        Speaker: Laure Raynaud (Météo-France)
      • 09:30
        Machine learning at ECMWF 30m

        The capability of machine learning to learn complex, non-linear behaviour from data offers many application areas across the numerical weather prediction workflow, including observation processing, data assimilation, the forecast model and the emulation of physical parametrisation schemes, as well as post-processing. This talk provides an overview on the activities at ECMWF to explore the potential of machine learning, and in particular deep learning, to improve weather predictions in the coming years.

        Speaker: Dr Peter Dueben (ECMWF)
      • 10:00
        Probabilistic Deep Learning for Postprocessing Wind Forecasts in Complex Terrain 30m

        The numerical weather predictions (NWPs) approach to forecasting of surface winds and their corresponding uncertainty in complex terrain remains an important challenge. Even for kilometer-scale NWP, many local topographical features remain unaccounted, often resulting in biased forecasts with respect to local weather conditions.

        Through statistical postprocessing of NWP, such systematic biases can be adjusted a posteriori using wind measurements. However, for unobserved locations, these approaches fail to give satisfying results. Indeed, the complex and nonlinear relationship between model error and topography calls for more advanced techniques such neural networks (NN).

        In addition, the prevalence of aleatoric uncertainties in wind forecasts demands the adoption of a probabilistic approach where the statistical model is not only trained to predict an error expectation (the bias), but also its scale (standard deviation). In this context, the model must be trained and evaluated using a proper scoring rule.

        To this end, we developed a machine learning application to efficiently handle very large datasets, train probabilistic NN architectures, and test multiple combinations of predictors. This enabled us to improve the quality of the model direct output not only at the location of reference measurements, but also at any given point in space, and for forecasts up to 5 days. More importantly, the results underline that the combination of physical models with a data-driven approach opens new opportunities to improve weather forecasts.

        Speaker: Mr Daniele Nerini (MeteoSwiss)
    • 10:30 11:00
      Coffee break 30m
    • 11:00 13:00
      Session 4 (cont.) and Session 5: ML for Product Development and ML for Model Identification and development
      Convener: Sveinung Loekken (ESA)
      • 11:00
        Significance-tested and physically constrained interpretation of a deep-learning model for tornadoes 30m

        We have developed a convolutional neural network (CNN) to predict tornadoes at lead times up to one hour in a storm-centered framework. We trained the CNN with data similar to those used in operations – namely, a radar image of the storm and a sounding of the near-storm environment. However, CNNs and other ML methods are often distrusted by users, who view them as opaque “black boxes” whose decisions cannot be explained. To address this problem, the field of interpretable ML has emerged, providing methods for understanding what an ML model has learned. However, interpretation methods can be misleading, often producing artifacts ("noise") rather than illuminating the true physical relationships in the data. To address both of these problems (opaque models and noisy interpretation results), we have applied several interpretation methods to the CNN for tornado prediction, each augmented with either a statistical-significance test or physical constraints.

        Specifically, we use four interpretation methods: the permutation importance test, saliency maps, class-activation maps, and backward optimization. For the permutation test, we use four different versions of the test and apply significance-testing to each, which allows us to identify where the ranking of predictor importance is robust. For saliency and class-activation maps, we use the "sanity checks" proposed by Adebayo et al. (2018; http://papers.nips.cc/paper/8160-sanity-checks-for-saliency-maps), augmented with formal significance tests. These tests ensure that interpretation results cannot be reproduced by a trivial method like an untrained edge-detection filter. For backward optimization, which produces synthetic storms that minimize or maximize tornado probability (prototypical non-tornadic and tornadic storms, respectively), we use physical constraints that force the synthetic storms to be more realistic.

        To our knowledge, this work is one of the few applications of ML interpretation, and the only one with significance-tested or physically constrained ML interpretation, in the geosciences. As ML becomes more integrated into geoscience research and everyday applications, such work will be crucial in building ML systems that are trusted and understood by humans.

        Speaker: Ms Ryan Lagerquist (Cooperative Institute for Research in the Atmosphere (CIRA))
      • 11:30
        Neural Networks for Postprocessing Ensemble Weather Forecasts 30m

        Ensemble weather predictions require statistical postprocessing of systematic errors to obtain reliable and accurate probabilistic forecasts. Traditionally, this is accomplished with distributional regression models in which the parameters of a predictive distribution are estimated from a training period. We propose a flexible alternative based on neural networks that can incorporate nonlinear relationships between arbitrary predictor variables and forecast distribution parameters that are automatically learned in a data-driven way rather than requiring prespecified link functions. In a case study of ECMWF 2-m temperature forecasts at surface stations in Germany, the neural network approach significantly outperforms benchmark postprocessing methods while being computationally more affordable. Key components to this improvement are the use of auxiliary predictor variables and station-specific information with the help of embeddings. Furthermore, the trained neural network can be used to gain insight into the importance of meteorological variables, thereby challenging the notion of neural networks as uninterpretable black boxes. Our approach can easily be extended to other statistical postprocessing and forecasting problems. We anticipate that recent advances in deep learning combined with the ever-increasing amounts of model and observation data will transform the postprocessing of numerical weather forecasts in the coming decade.

        Speaker: Sebastian Lerch (Karlsruher Institut für Technologie (KIT))
      • 12:00
        Towards an end-to-end data-driven weather model 30m

        Weather forecasting systems have not fundamentally changed since they were first operationalised nearly 50 years ago. They use traditional finite-element methods to solve the fluid dynamical flow of the atmosphere and include as much sub-grid physics as they can computationally afford. Given the huge amounts of data currently available from both models and observations new opportunities exist to train data-driven models to produce these forecasts. Traditional weather forecasting models are steadily improving over time, as computational power and other improvements allow for increased spatial resolution, effectively incorporating more physics into our forecasts. However these improvements are best seen in the prognostic variables of weather forecasting: e.g. velocity, temperature, pressure. For other quantities of arguably greater importance, for example precipitation, these improvements come at a slower pace.
        The current boom in machine learning (ML) has inspired several groups to approach the problem of weather forecasting. Here we will provide an overview of the latest attempts at this from local now-casting of precipitation up to global forecasts of atmospheric dynamics. We will then present our latest efforts towards a multi-model system leveraging existing numerical models to incorporate physical understanding within a data-driven machine learning approach for skilful forecasting of global precipitation over several days.

        Speaker: Duncan Watson-Parris (University of Oxford)
      • 12:30
        Emulation of gravity wave parameterisation in weather forecasting 30m

        The rise of machine learning offers many exciting avenues for improving weather forecasting. Possibly the lowest hanging fruit is the acceleration of parameterisation schemes through machine learning emulation. Parameterisation schemes are highly uncertain closure schemes necessitated by the finite grid-spacing of weather forecasting models. Here we assess the challenges and benefits of emulating two parameterisation schemes related to gravity wave drag in the IFS model of ECMWF. Despite the similar structure of these schemes we find that one poses far greater challenge to build a successful emulator. After successful offline testing we present results coupling our emulators to the IFS model. In coupled mode the IFS still produces accurate forecasts and climatologies. Building on this we use our emulator in the data assimilation task, leveraging that tangent-linear and adjoint models of neural networks can be easily derived.

        Speaker: Matthew Chantry (University of Oxford)
    • 13:00 14:00
      Lunch break 1h
    • 14:00 15:30
      Session 5 (cont.): ML for Model Identification and development
      Convener: Massimo Bonavita (ECMWF)
      • 14:00
        Physics Guided Machine Learning: A New Framework for Accelerating Scientific Discovery 30m

        Physics-based models of dynamical systems are often used to study engineering and environmental systems. Despite their extensive use, these models have several well-known limitations due to incomplete or inaccurate representations of the physical processes being modeled. Given rapid data growth due to advances in sensor technologies, there is a tremendous opportunity to systematically advance modeling in these domains by using machine learning (ML) methods. However, capturing this opportunity is contingent on a paradigm shift in data-intensive scientific discovery since the “black box” use of ML often leads to serious false discoveries in scientific applications. Because the hypothesis space of scientific applications is often complex and exponentially large, an uninformed data-driven search can easily select a highly complex model that is neither generalizable nor physically interpretable, resulting in the discovery of spurious relationships, predictors, and patterns. This problem becomes worse when there is a scarcity of labeled samples, which is quite common in science and engineering domains.

        This talk makes a case that in a real-world systems that are governed by physical processes, there is an opportunity to take advantage of fundamental physical principles to inform the search of a physically meaningful and accurate ML model. Even though this will be illustrated for a few problems in the domain of aquatic sciences and hydrology, the paradigm has the potential to greatly advance the pace of discovery in a number of scientific and engineering disciplines where physics-based models are used, e.g., power engineering, climate science, weather forecasting, materials science, and biomedicine.

        Speaker: Vipin Kumar (University of Minnesota)
      • 14:30
        Assessing Machine Learning Approaches for Physical Parameterizations in Atmospheric General Circulation Models 30m

        Atmospheric General Circulation Models (GCMs) contain computationally-demanding physical parameterization schemes, which approximate the unresolved subgrid-scale physics processes. This work explores whether a selection of machine learning (ML) techniques can serve as computationally-efficient emulators of physical parameterizations in GCMs, and what the pros and cons of the different approaches are. We test the ML emulators in a simplified model hierarchy with NCAR’s Community Atmosphere Model version 6 (CAM6), which is part of NCAR’s Community Earth System Model. Dry and idealized-moist CAM6 model configurations are used, which employ simplified physical forcing mechanisms for radiation, boundary layer mixing, surface fluxes, and precipitation (in the moist setup). Several machine learning techniques are developed, trained, and tested offline using CAM6 output data. These include linear regression, random and boosted forests, and multiple deep learning architectures. We show that these methods can reproduce the physical forcing mechanisms. We also show that the growing complexity of the physical forcing in our model hierarchy puts increased demands on the ML algorithms and their training & tuning. We compare the different machine learning techniques and discuss their strengths and weaknesses.

        Speaker: Christiane Jablonowski (University of Michigan)
      • 15:00
        Model optimization with a genetic algorithm 30m

        In an Ensemble Kalman Filter (EnKF), many short-range forecasts are used to propagate error statistics. In the Canadian global EnKF system, different ensemble members use different configurations of the forecast model. The integrations with different versions of the model physics can be used to optimize the probability distributions for the model parameters.

        Continuous parameters accept a continuous range of values. Categorical parameters can serve as switches between different parametrizations. In the genetic algorithm, the best member are duplicated, while adding a small perturbation, and the worst performing configurations are removed. The algorithm is being used in the migration of the global ensemble prediction system to an upgraded version of the model.

        Quality is measured with both a deterministic and an ensemble score, using the observations assimilated in the EnKF system. With the ensemble score, the algorithm can converge to non-Gaussian distributions. Unfortunately, for several model parameters, there is not enough information to improve the distributions. The optimized system has slight reductions in biases for humidity sensitive radiance measurements. Modest improvements are also
        seen in medium-range ensemble forecasts.

        Speaker: Pieter Houtekamer (Environment and Climate Change Canada)
    • 15:30 16:00
      Coffee break 30m
    • 16:00 18:00
      Session 5 (cont.): ML for Model Identification and development
      Convener: Peter Jan van Leeuwen (Colorado State University)
      • 16:00
        Analog Forecasting of Extreme‐Causing Weather Patterns Using Deep Learning 30m

        Numerical weather prediction (NWP) models require ever-growing computing time and resources, but still, have sometimes difficulties with predicting weather extremes. We introduce a data-driven framework that is based on analog forecasting (prediction using past similar patterns), and employs a novel deep learning pattern-recognition technique (capsule neural networks, CapsNets) and an impact-based auto-labeling strategy. Using data from a large-ensemble fully coupled Earth system model, CapsNets are trained on mid-tropospheric large-scale circulation patterns (Z500) labeled $0-4$ depending on the existence and geographical region of surface temperature extremes over North America several days ahead. The trained networks predict the occurrence/region of cold or heat waves, only using Z500, with accuracies (recalls) of $69\%-45\%$ ($77\%-48\%$) or $62\%-41\%$ ($73\%-47\%$) $1-5$ days ahead. Using both surface temperature and Z500, accuracies (recalls) with CapsNets increase to $\sim 80\%$ ($88\%$). In both cases, CapsNets outperform simpler techniques such as convolutional neural networks and logistic regression, and their accuracy is least affected as the size of the training set is reduced. The results show the promises of multi-variate data-driven frameworks for accurate and fast extreme weather predictions, which can potentially augment NWP efforts in providing early warnings.

        Speaker: Mr Ashesh Chattopadhyay (Rice University)
      • 16:30
        What If The Easiest Part of the Global Atmospheric System For Machines To Learn Is The Dynamics? 30m

        We present a deep convolutional neural network (CNN) to forecast four variables on spherical shells characterizing the dry global atmosphere: 1000-hPa height, 500-hPa height, 2-m surface temperature and 700-300 hPa thickness. The variables are carried on a cubed sphere, which is a natural architecture on which to evaluate CNN stencils. In addition to the forecast fields, three external fields are specified: a land-sea mask, topographic height, and top-of-atmosphere insolation. The model is recursively stepped forward in 12-hour time steps while representing the atmospheric fields with 6-hour temporal and roughly 1.9 x 1.9 degree spatial resolution. It produces skillful forecasts at lead times up to about 7 days. The model remains stable out to arbitrarily long forecast lead times.

        As an example of its climatological behavior, panel (a) in the figure shows the 1000-hPa and 500-hPa height fields from a free running forecast 195 days after a July initialization. The model correctly develops active wintertime weather systems in response to the seasonal changes in top-of-atmosphere insolation. As a qualitative comparison, panels (b) and (c) show the verification and the climatology for the same January 15th.

        While our model certainly does not provide a complete state-of-the-art weather forecast, its skill is less than 2 days of lead time behind the approximately equivalent horizontal resolution T63 137L IFS. It is difficult to make a rigorous timing comparison between our model, which runs on a GPU, and the T63 IFS which was run on a multi-core CPU, but reasonable wall-clock estimates suggest our model is three orders of magnitude faster. It remains to be seen how more advanced deep-learning weather prediction models will compare to current NWP models with respect to both speed and accuracy, but these results suggest they could be an attractive alternative for large-ensemble weather and sub-seasonal forecasting.

        Speaker: Prof. Dale Durran (University of Washington)
      • 17:00
        S2S forecasting using large ensembles of data-driven global weather prediction models 30m

        We develop an ensemble prediction system (EPS) based on a purely data-driven global atmospheric model that uses convolutional neural networks (CNNs) on the cubed sphere. Mirroring practices in operational EPSs, we incorporate both initial-condition and model-physics perturbations; the former are sub-optimally drawn from the perturbed ECMWF ReAnalysis 5 members, while the latter are produced by randomizing the training process for the CNNs. Our grand ensemble consists of 320 perturbed members, each of 32 CNNs run with 10 perturbed initial conditions. At lead times up to two weeks, our EPS lags the state-of-the-art 50-member ECMWF EPS by about 2-3 days of forecast lead time, and is modestly under-dispersive, with a spread-skill ratio of about 0.75.

        For weekly-averaged forecasts in the sub-seasonal-to-seasonal forecast range (2-6 weeks ahead), a particularly challenging window for weather forecasting, our data-driven EPS consistently outperforms persistence forecasts of 850-hPa temperature and 2-meter temperature, with useful skill relative to climatology as measured by the ranked probability skill score and the continuous ranked probability score (CRPS). Over twice-weekly forecasts in the period 2017-18, the CRPS of our model matches that of the ECMWF EPS to within 95% statistical confidence bounds for T850 at week 4 and weeks 5-6. While our model performs similarly for T2 compared to T850, the ECMWF EPS includes a coupled ocean model, which results in much better T2 forecasts, especially over tropical oceans. Our EPS is closest to parity with the ECMWF EPS in the extra-tropics, especially during spring and summer months, where the ECMWF ensemble is weakest. Notably, our model, while only predicting six 2-dimensional atmospheric variables, runs extremely efficiently, computing all 6-week forecasts in under four minutes on a single GPU. Nevertheless, further work remains to incorporate ocean and sea-ice information into our data-driven model to improve its representations of large-scale climate dynamics.

        Speaker: Jonathan Weyn (University of Washington)
      • 17:30
        Description of Working Groups 30m
        Speaker: Massimo Bonavita (ECMWF)
    • 09:00 10:30
      Working groups
      • 09:00
        WG1 (Observations) chaired by Alan Geer (ECMWF) and Bertrand Le Saux (ESA/ESRIN) 1h 30m
      • 09:00
        WG2 (Data Assimilation) chaired by Alberto Carrassi (Univ. of Reading) and Rosella Arcucci (Imperial College) 1h 30m
      • 09:00
        WG3 (Models) chaired by Massimo Bonavita (ECMWF) and Peter Dueben (ECMWF) 1h 30m
      • 09:00
        WG4 (Ensembles, Product Generation) chaired by Laure Raynaud (Météo-France) and Nicolas Longepe (ESA) 1h 30m
    • 10:30 11:00
      Coffee break 30m
    • 11:00 12:30
      Working groups
      • 11:00
        WG1 (Observations) chaired by Alan Geer (ECMWF) and Bertrand Le Saux (ESA/ESRIN) 1h 30m
      • 11:00
        WG2 (Data Assimilation) chaired by Alberto Carrassi (Univ. of Reading) and Rosella Arcucci (Imperial College) 1h 30m
      • 11:00
        WG3 (Models) chaired by Massimo Bonavita (ECMWF) and Peter Dueben (ECMWF) 1h 30m
      • 11:00
        WG4 (Ensembles, Product Generation) chaired by Laure Raynaud (Météo-France) and Nicolas Longepe (ESA) 1h 30m
    • 12:30 13:00
      WG chairs finalise reports
    • 13:00 14:00
      Lunch break 1h
    • 14:00 15:30
      Working Groups plenary discussion and close
      Convener: Massimo Bonavita (ECMWF)