Recalibration of the quantile regressor on the real data

Walk-forward CV of the data op with quantile_strategy="multiple_regressors" (HGB, max_iter=2000), horizons 1h, 12h, 24h, quantiles 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 8 CV splits, with (recalibrate=True, 10 bins, calibrated on the last 4 inner folds of each training set) and without calibration.Recalibrator. See calibration_demo.html for how the correction works. Scores are those reported by the data op, averaged over the CV splits. Coverage and width are for the 0.05–0.95 interval (nominal coverage 0.90).

Scores

Interval coverage and width

MAPE of the median prediction

mean_absolute_percentage_error of each model's median prediction, averaged over the CV splits. The last column is the change in percentage points (negative is better).

MAPE of the median for each CV split (%)

d2 pinball score (higher is better), averaged over horizons

Coverage of the 0.05–0.95 interval for each CV split

Reading the results

max_iter=2000, new splitter (8 outer test folds of 13 weeks, aligned to the end, 2023-06-03 to 2025-05-31), calibration on the last 4 inner folds of each training set (4 x 13 weeks, about 8.7k out-of-fold rows per split), 10 bins. Runtime was 1h45 (raw 27 min, recalibrated 78 min).

• Coverage of the 90% interval (raw → recalibrated): 0.815 → 0.904 (1h), 0.764 → 0.910 (12h), 0.761 → 0.904 (24h). Width: 2.2 → 2.7 GW (1h, +26%), 3.3 → 4.8 GW (12h, +44%), 3.3 → 4.8 GW (24h, +45%). These are very close to the earlier 5-fold/`max_iter=1000` run (0.909 / 0.901 / 0.898 with +25% / +42% / +45%).
• d2 pinball averaged over the 7 quantiles: 0.8213 → 0.8284. It improves in the tails (q_0.05 +0.015, q_0.1 +0.010, q_0.9 +0.017, q_0.95 +0.017) and slightly degrades in the centre (q_0.25 −0.004, q_0.5 −0.007). MAPE of the median: 1.11% → 1.12% (1h), 1.96% → 2.05% (12h), 1.97% → 2.06% (24h).
• Per split, coverage improves everywhere (24h, raw → recalibrated): 0.653 → 0.889, 0.785 → 0.884, 0.773 → 0.909, 0.788 → 0.928, 0.758 → 0.957, 0.763 → 0.904, 0.759 → 0.856, 0.807 → 0.907. Unlike the earlier 5-fold runs, the first splits now improve too: with 8 shorter folds and a calibration window of the last 4 inner folds (about 1 year), the out-of-fold data is much closer in time to each test fold. Split 4 (test from 2024-06-01) over-corrects (0.957 at 24h, 0.974 at 1h).
• The centre is still pushed down. Pooled share of observations below the predicted quantile at 24h: q_0.05 0.034, q_0.25 0.178, q_0.5 0.409, q_0.75 0.691, q_0.95 0.938 (targets 0.05 / 0.25 / 0.5 / 0.75 / 0.95; raw 0.115 / 0.319 / 0.458 / 0.611 / 0.876). The correction fixes the tails more than the centre, which is why the interval coverage is right while the median accuracy is slightly lower.
• Iterations. At `max_iter=2000` the central quantiles still hit the cap (q_0.25, q_0.5, q_0.75: 100% of the 24 models per horizon), while q_0.05 (mean 1380) and q_0.95 (mean 1146) stop earlier. More iterations would probably still change the centre.

Predictions over time

Ground truth (black), median (line) and the 0.05–0.95 quantile band, for the raw (top) and recalibrated (bottom) models, concatenating the test folds of all the CV splits. The x axis is the target time (prediction time plus horizon). Use the dropdown to change the horizon. Drag a rectangle to zoom in both x and y; double-click to reset. The initial window is two weeks of the last split. The two panels look nearly identical because the correction is small: over these test folds the recalibrated median differs from the raw one by at most 1374 MW, and the interval bounds by at most 2689 MW (about 5.5% of the load). Zoom in to see it (for example around 2023-12-03 18:00).

Reliability diagrams

Both plots below are reliability diagrams, but they check different properties. All the CV test folds are pooled.

Coverage curve (marginal reliability)

For each nominal level τ, the empirical coverage: the fraction of observed values at or below the τ-quantile forecast, plotted against τ (one panel per horizon, dashed line = perfect calibration). It is the marginal check, averaged over all situations, so a model can be on the diagonal on average while being miscalibrated in particular situations, with errors cancelling out.

Conditional coverage of the interval (by predicted load)

The marginal coverage above can hide over-confidence in some regions and under-confidence in others. Here the test cases are split into 10 bins of equal size by each model's predicted median, and in each bin we compute the observed coverage of the 0.05–0.95 interval (the fraction of observations inside it). The dashed line is the nominal level. A curve below it means the interval is too narrow at that predicted load, above it too wide.

Conditional reliability curves, binned (one panel per quantile)

One panel per quantile level τ (use the dropdown to change the horizon). The test cases are split into 10 bins of equal size by predicted value (each model's own predictions); in each bin, x is the mean prediction and y is the mean prediction plus the τ-quantile of the residuals y − p in the bin. Using the residuals (rather than the plain quantile of y in the bin) removes an artefact of the predictions varying inside a bin, so a perfectly calibrated model lies exactly on the dashed 45° diagonal (actual = predicted); the axes are on the same scale (1:1). This is the conditional check: it asks whether the claimed level holds for the cases with that prediction, and reveals miscalibration that depends on the predicted value, which the coverage curve can hide.

Conditional reliability curves, isotonic / PAV (one panel per quantile)

The same check, with the conditional τ-quantile estimated by isotonic regression: sort the cases by prediction, then repeatedly pool adjacent blocks whenever their τ-quantile would decrease. The result is a monotone step function whose blocks are placed by the data, with no bin width to choose. Again, a perfectly calibrated model lies on the dashed 45° diagonal, and the axes are on the same scale (1:1).

Learned corrections

For q_0.95, q_0.5 and q_0.05, one colour per CV split. Lines: the correction (shift in MW added to the predicted quantile) learned by the recalibrator on the training set of the split, as a function of the predicted value. Points: the same measured on the outer test fold with the raw predictions (equal-count bins of the predicted value, shift = quantile of y − p in the bin), i.e. what the correction should have been on the test data. If the calibration transfers from the training set to the test fold, the lines go through the points. Bottom: the correction itself; top: the same as predicted quantile → actual quantile (predicted + shift), where the dashed diagonal is a calibrated model (the corrections are small compared to the range of loads, so the bottom row is easier to read).