Both comparisons use the same 8 walk-forward CV folds and the same model settings. The oracle comparison uses the Open-Meteo Historical Forecast files as weather features; the forecast comparison uses the safe-lead historical forecasts (day1 for 1h/12h, day2 for 24h). Positive deltas mean the forecast-weather run is worse, i.e. the oracle score was optimistic.
KumoForecast only accepts past covariates. Its run uses oracle/analysis past weather in both comparisons, so it is shown in the table but excluded from the weather-impact charts. Its near-zero delta is expected.
| Model | MAE oracle | MAE forecast | ΔMAE (MW, %) | MAPE oracle | MAPE forecast | ΔMAPE (pp) |
|---|---|---|---|---|---|---|
| HGB mean | 813.6 | 872.2 | +58.6 (+7.2%) | 1.67% | 1.78% | +0.111 |
| HGB q7 | 815.7 | 859.0 | +43.3 (+5.3%) | 1.68% | 1.75% | +0.078 |
| HGB q7 recal | 854.2 | 911.1 | +56.9 (+6.7%) | 1.74% | 1.84% | +0.102 |
| TabICL n8 | 816.2 | 846.6 | +30.4 (+3.7%) | 1.65% | 1.70% | +0.050 |
| Chronos-2 | 786.6 | 827.4 | +40.8 (+5.2%) | 1.64% | 1.71% | +0.071 |
| Kumo-Tabular | 619.2 | 688.4 | +69.2 (+11.2%) | 1.28% | 1.41% | +0.124 |
| KumoForecast | 1804.6 | 1806.9 | +2.3 (+0.1%) | 3.78% | 3.78% | +0.001 |
Each panel is one horizon. Markers show the mean paired difference (oracle − forecast); negative means the oracle-weather error is lower. Large markers: pooled mean difference over all test rows, with a 95% moving-block bootstrap interval (14-day blocks over daily differences; conditional on the fitted models, it ignores training randomness). Small dots: mean difference in each of the 8 CV folds (one dot per fold; hover shows the fold). Folds are not independent, so no t-interval across folds is shown. MAPE uses the median prediction; pinball loss is averaged over the 5%, 50% and 95% quantiles for that horizon. KumoForecast is excluded from these panels (see the note above).
Hourly temperature over the last four weeks of the data. The
oracle is the Open-Meteo Historical Forecast value; the 24h and 48h forecasts
are the GFS previous_day1/previous_day2 values (valid
time minus 24h/48h). Use the dropdown to switch city.
| Model | q0.05 pinball | q0.5 pinball | q0.95 pinball | 90% coverage | 90% width (MW) |
|---|---|---|---|---|---|
| HGB q7 | 0.8295 → 0.8179 | 0.8985 → 0.8931 | 0.8759 → 0.8674 | 0.7792 → 0.7665 | 2926 → 2987 |
| HGB q7 recal | 0.8405 → 0.8302 | 0.8937 → 0.8867 | 0.8852 → 0.8794 | 0.9066 → 0.9031 | 4100 → 4288 |
| TabICL n8 | 0.8416 → 0.8406 | 0.8985 → 0.8947 | 0.9006 → 0.8930 | 0.8927 → 0.8730 | 3841 → 3712 |
| Chronos-2 | 0.8523 → 0.8442 | 0.9021 → 0.8971 | 0.9030 → 0.8996 | 0.8957 → 0.8908 | 3232 → 3338 |
| Kumo-Tabular | 0.8638 → 0.8526 | 0.9230 → 0.9144 | 0.9115 → 0.9075 | 0.9606 → 0.9605 | 4061 → 4313 |
Accuracy vs fit + predict time, oracle vs forecast