Forecast Calibration and Regression Diagnostics via Isotonicity

• Applicant: Professor Dr. Timo Dimitriadis
• Subject Area: Statistics and Econometrics
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 502572912

Evaluating and comparing predictions is of fundamental interest in many disciplines, particularly reinforced by the current surge of machine learning tools. One crucial property of the predictions is calibration, which means that the predictions can be taken at face value: E.g., calibration of a binary event (let's say rainfall tomorrow) means that if it is predicted to occur with 80% probability, it should indeed materialize with that frequency. Calibration is evaluated in many academic disciplines, often under different names and with varying methodologies. E.g., the so-called backtests used in the regulatory frameworks for banking and insurance essentially test forecast calibration for the risk measures Value at Risk and Expected Shortfall. Furthermore, regression diagnostics and goodness of fit tests assess whether a parametric model fits the data sufficiently well, which, in a nutshell, evaluates whether the model predictions are calibrated. The statistical tools employed in these field are however subject to multiple criticisms. This project tackles some of these criticisms by advancing the recently developed CORP methodology of Dimitriadis et al. (2021, PNAS). The authors propose the use of a nonparametric isotonic regression which results in a stable and reproducible graphical display evaluating calibration (the reliability diagram), and an improved decomposition of proper scoring rules. Their methodology is however limited to the setting of binary events and the isotonic regression complicates valid inference for formal statistical tests. This project proposes generalizations along both these dimensions: First, we will develop universally valid statistical inference methods which allow to employ the CORP methodology to testing goodness of fit for (binary) regressions. This can drastically improve upon the current benchmark method, the Hosmer and Lemeshow test in terms of stability, reproducibility and interpretability. Second, we will generalize the CORP methodology to point forecasts and risk measures such as Value at Risk and Expected Shortfall. This will facilitate a better understanding of the existing backtests, and allows for the development of new, robust ones. The results of this project will be of interest for the theoretical literature on forecast evaluation and isotonic regressions as well as for practitioners working with binary regressions in general, and on financial risk management and backtests in particular. We anticipate that similar adaptations and generalizations allow for improvements of the state of the art methodology in other academic fields where calibrated predictions are of interest.

• DFG Programme: Research Grants
• International Connection: Switzerland
• Cooperation Partner: Professorin Dr. Johanna Ziegel