Forecast bias and accuracy: compare the same horizon

A forecast can have little average bias and still miss individual months materially. In the illustrative revenue forecasts below, average overprediction is only $1,250, but the mean absolute error is $6,250. A second set, made three months ahead, misses by $20,000 per month. Comparing those sets as if they had the same forecast horizon would obscure a large part of the problem.

Keep the forecast that was actually available at the decision date

Forecast error compares an earlier prediction with the actual result for the same target period. Once actuals arrive, replacing an old forecast with a revised number destroys the evidence needed to assess it. Retain the target month, forecast creation date, information cutoff, horizon and forecast value alongside the actual.

This note uses original illustrative monthly revenue amounts, shown in $ thousands. The one-month-ahead and three-month-ahead columns represent forecasts made at their respective earlier origins. They are assumed examples, not a backtest of a model or a business dataset.

Measure direction and size separately

Define error here as forecast minus actual. Positive error therefore means revenue was overforecast. Some statistical references use the opposite sign; specify the convention before interpreting bias. For the one-month forecasts, the errors are +5, −5, +10 and −5 thousand dollars.

The mean error is (5 − 5 + 10 − 5) ÷ 4 = $1,250. Positive and negative misses largely offset. Mean absolute error, or MAE, averages their magnitudes: (5 + 5 + 10 + 5) ÷ 4 = $6,250. Root mean squared error, or RMSE, takes the square root of the average squared errors: approximately $6,614. RMSE gives greater weight to the larger miss.

Small mean error does not demonstrate accurate monthly forecasting. In this example it says that errors offset over four observations. The business could still face a collection or purchasing decision during the month with the $10,000 revenue miss.

Compare like horizons and like information

Before comparing two forecasting methods, evaluate both on the same target periods, forecast horizon, available information and error measure. A forecast updated two days before month-end has an information advantage over one frozen three months earlier. Its lower error does not, by itself, establish a better forecasting method.

Use rolling forecast origins to create several genuine earlier forecasts and score the results at each required horizon. Keep the later actual result out of model fitting and assumption selection for that forecast origin. Include a simple benchmark, such as a seasonal-naive forecast, where the business has a meaningful seasonal history. A more complex model must improve the relevant decision, not just the fitted historical curve.

Choose the measure that matches the financial exposure

Absolute errors in dollars are useful when evaluating cash and funding decisions. A revenue forecast error is not automatically an equal cash shortfall: contribution costs, collection timing and inventory commitments change the cash effect. Recalculate those downstream movements in the financial model.

Percentage measures need additional care near zero actual values. An actual of zero makes the usual absolute percentage error undefined, while a very small actual can dominate an average percentage score. A scaled or absolute measure may be more interpretable for a young or intermittent-revenue business.

Four observations are enough to illustrate the calculations, not to establish a dependable accuracy distribution. Use a longer history covering relevant business conditions before choosing buffer sizes or judging whether a measured bias is persistent. Inspect the largest individual misses as well as the average.

Method and sources

The amounts and error calculations are original illustrative examples. Hyndman and Athanasopoulos explain genuine out-of-sample forecast assessment, error measures and their limitations in Evaluating point forecast accuracy, and describe repeated origins in time series cross-validation. This note uses forecast minus actual for its bias sign; the error convention must be kept consistent when applying their methods.

Use the company forecast tool to explore operating assumptions. Retain dated forecast versions separately if you intend to assess their later accuracy; a current scenario alone is not a forecast history.