Recalibrating a quantile regressor

The problem

A quantile regressor claims that, given X, its output q_0.95 is the value (in MW) below which the load will fall 95% of the time. If that claim is true the model is calibrated: over many predictions, about 95% of the observed values end up below q_0.95. In practice models are often miscalibrated, typically too confident (intervals too narrow), and the error may depend on the predicted level (here, the peaks are the worst).

The correction (calibration.Recalibrator)

The recalibrator wraps a quantile regressor. For each predicted quantile level τ (each column q_τ) independently:

1. Out-of-fold predictions. Using a walk-forward time-series CV on the training data (with a gap between train and test), the wrapped model is fit on the past and predicts the future of each fold. We concatenate the predictions p with the true values y. These predictions are honest, i.e. not made on data the model was trained on.
2. Bin by predicted value. The out-of-fold predictions are split into 10 bins with the same number of points. Within a bin the model claims that p is (about) the τ-quantile of y.
3. Measure the actual quantile in each bin. We compute the actual τ-quantile of the residuals y − p over the points in the bin. This is a number in MW: the shift that would make the bin calibrated. If the model is calibrated the shift is about 0; if the intervals are too narrow, the shift for q_0.95 is positive (the prediction should be higher) and the shift for q_0.05 is negative.
4. Smooth. The per-bin shifts are noisy, especially for tail quantiles. We fit a small model (SplineTransformer with 5 basis functions followed by RidgeCV, weighted by the number of points in each bin) that maps a predicted value to a shift. The figure below shows the bin measurements and the fitted function.
5. Apply. The wrapped model is refit on all the training data. At prediction time, each predicted quantile is corrected by adding the shift predicted for its value: q_τ_corrected = q_τ + shift_τ(q_τ). The quantiles are finally sorted so that they cannot cross.

The correction can only fix a systematic, level-dependent miscalibration. It does not make the model sharper: if intervals have to be wider to be honest, they will be wider. It also assumes the calibration measured in the past still holds in the future.

The toy example

Hourly synthetic load over 1460 days: a daily cycle, a yearly cycle, an observable autocorrelated "weather" signal and Gaussian noise whose standard deviation is 8% of the load level. The first 70% is used for training (and for the recalibrator's internal CV), the rest as the test set. The base model is a gradient boosting quantile regressor (one HGB per quantile in 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95) whose intervals are then deliberately shrunk towards the median, more so for high predicted loads (factor 0.95 at 40 MW down to 0.4 at 90 MW), to simulate a model that under-estimates uncertainty, mostly at peaks.

The recalibrator uses a walk-forward CV with 4 test folds of 90 days and a 7-day gap between train and test. Its out-of-fold models must be trained on enough data to be representative of the final model (here, at least about 1.8 years, so that the yearly cycle is seen). If the first folds are trained on very little data, their errors are larger than those of the final model, and the correction over-shoots (we observed 94% coverage for a 90% interval with a too-short initial training window).

Learned corrections

For each bin of predicted values: the measured shift (dots, hover for details) and the smooth function used for the correction (line). The upper quantile needs to be raised and the lower one lowered, increasingly so for high loads.

Predictions over time

Raw model on top, recalibrated model below (same y-axis range). The band is the 0.05–0.95 interval and the line is the median. The raw band is too narrow and true values often fall outside, especially at the daily peaks; after correction the band is wider where needed.

Reliability diagrams

Both plots below are called reliability diagrams, but they check different properties.

Coverage curve (marginal reliability)

For each nominal level τ, the empirical coverage: the fraction of observed values at or below the τ-quantile forecast, pooled over all the test cases and plotted against τ. The dashed diagonal is perfect calibration. This is the marginal check: it averages over all forecast situations, so a model can be on the diagonal on average while being miscalibrated in particular situations (e.g. at high load), with errors cancelling out. Points below the diagonal above τ = 0.5 (and above it below 0.5) mean intervals that are too narrow.

Conditional coverage of the interval (by predicted load)

The marginal coverage can hide over-confidence in some regions and under-confidence in others. Here the test cases are split into 10 bins of equal size by the predicted median, and in each bin we compute the observed coverage of the 0.05–0.95 interval. The dashed line is the nominal level; a curve below it means the interval is too narrow at that predicted load, above it too wide.

Conditional reliability curves, binned (one panel per quantile)

One panel per quantile level τ. The test cases are split into 10 bins of equal size by predicted value (each model's own predictions); in each bin, x is the mean prediction and y is the mean prediction plus the τ-quantile of the residuals y − p in the bin. Using the residuals (rather than the plain quantile of y in the bin) removes an artefact of the predictions varying inside a bin, so a perfectly calibrated model lies exactly on the dashed 45° diagonal (actual = predicted); the axes are on the same scale (1:1).

Conditional reliability curves, isotonic / PAV (one panel per quantile)

The same check, with the conditional τ-quantile estimated by isotonic regression: sort the cases by prediction, then repeatedly pool adjacent blocks whenever their τ-quantile would decrease. The result is a monotone step function whose blocks are placed by the data, with no bin width to choose. Cases with the same prediction are pooled (a step function gives them the same value). Again, a perfectly calibrated model lies on the dashed 45° diagonal.

Metrics on the test set

Coverage and width are computed with electricity_load_forecasting.coverage and mean_width. The pinball loss (lower is better) is averaged over all the quantile levels, and the MAPE is that of the median prediction (mean_absolute_percentage_error). Recalibration restores the nominal coverage at the cost of wider intervals (the honest width for this model); the pinball loss, which rewards both calibration and sharpness, also improves, while the MAPE of the median is essentially unchanged.