Workshop on Predictability, dynamics and applications research using the TIGGE and S2S ensembles

Receiver Operating Characteristic (ROC) curves

Tilmann Gneiting, Peter Vogel, and Eva-Maria Walz

Heidelberg Institute for Theoretical Studies and Karlsruhe Institute of Technology, Germany

Relative or Receiver Operating Characteristic (ROC) curves are used in a very wide range of settings where covariates, features, markers, or probability forecasts for binary events are to be evaluated. In meteorology, the WMO mandates the use of the Area Under the ROC curve (AUC) measure for evaluating skill in long-range forecasts, and an ECMWF supplementary headline score uses the AUC to assess extreme forecast index (EFI) skill for wind speed over Europe. All these uses require the conversion of a continuous, real-valued weather quantity to a binary event.

I will review and examine the construction and usage of ROC curves from a general scientific perspective, with emphasis on current and future applications to ensemble weather forecasts, and illustrated on probabilistic quantitative precipitation forecasts over the tropics. Technically, a ROC curve is simply a plot of the hit rate vs. the false alarm rate as the cut-off value of a real-valued predictor variable for a binary event ranges over all possible thresholds. We distinguish raw ROC diagnostics and ROC curves, elucidate the special role of concavity in interpreting and modelling ROC curves, and establish an equivalence between ROC curves and cumulative distribution functions (CDFs).

In the final part of the talk, I will hint at very recent developments, in which we seek generalizations of ROC curves that apply to predictors of real-valued quantities directly, thereby eliminating the need for the conversion to a binary event.