Measuring Your Forecast: MAPE, RMSE, and the Holdout

Anyone can produce a forecast. Evaluating it — honestly, reproducibly, before the future arrives — is the part that separates analysis from storytelling. It comes down to three decisions: which error metric, which test period, and which baseline to beat.

The Three Metrics and Their Blind Spots

MAE (mean absolute error) is the average distance between forecast and actual:

MAE = mean( |actual - forecast| )

It is in the units of the data, easy to explain, and robust to outliers. Its blind spot: it treats a 10-unit miss as equally bad whether the true value was 100 or 10,000 — so it cannot compare series of different scales.

RMSE (root mean squared error) squares the errors before averaging, so large misses dominate:

RMSE = sqrt( mean( (actual - forecast)^2 ) )

If your cost structure punishes big misses disproportionately — inventory write-offs, missed service levels — RMSE is the honest metric. But because of the squaring, one outlier can inflate it to the point where it no longer reflects typical performance.

MAPE (mean absolute percentage error) divides each error by the actual value:

MAPE = mean( |actual - forecast| / |actual| ) * 100

Percentages are seductive because they are comparable across series — and they are the most abused metric in forecasting. Two structural problems:

• Division by near-zero. A month where actuals approach zero turns a normal miss into an enormous percentage. Series with seasonal troughs produce MAPE values that are pure noise.
• Asymmetry. MAPE punishes over-forecasting more than under-forecasting by construction, because the denominator is the actual, not the forecast. A symmetric variant (sMAPE) fixes that at the cost of being harder to explain.

The Holdout: Your Only Honest Test

Fit on the past, then forecast a period you deliberately withheld, and measure the error there. The discipline matters:

1. Never tune on the test. If you adjust the model until the holdout looks good, the holdout becomes training data by another name. Keep one untouched period for the final report.
2. Walk forward for stability. A single holdout is one roll of the dice. Walk-forward evaluation — refit on a rolling window, forecast one step, repeat — gives a distribution of errors instead of a single number.
3. Beat a naive baseline. The most important comparison is not "is my error small" but "is it smaller than the trivial alternative": last value carried forward, or the same period last year. If your sophisticated model cannot beat seasonal naive on the holdout, you do not have a better model — you have a more complicated one. This is the standard KPI Master applies when it grades a forecast against the data it was built from.

Choosing

There is no best metric, only the right one for the decision:

• Reconciliation and money: MAE (or integer cents — see floating-point hygiene).
• Capacity and stockouts: RMSE, because the tail hurts.
• Comparing across very different series: MAPE, with the near-zero caveat stated in the report.

Whatever you pick, write down the metric, the holdout, and the baseline before you see the outcome. Pre-registered evaluation is the only evaluation that cannot be argued with afterwards.