AI Forecasting: Why Forecasts Miss and How AI Helps

AI forecasting is the use of software that learns from historical data to estimate a future figure, with the estimate updated as new data arrives. This article keeps to IBM’s description of it, given below. Business forecasts miss for reasons that start before any model runs. Below are four of those reasons, which of them a model can change and which stay with people. Then comes a way to tell whether a forecast missed, and four questions to ask before you rely on one. The article sits under the Signisys use case on forecasting business trends.

AI Forecasting in Plain Words

Three terms come up throughout:

• Forecast: a written estimate of a figure for a future period.
• Historical data: the figures recorded for past periods.
• Machine learning: a method that learns patterns from data, as the machine learning term page explains.

IBM’s overview of forecasting with machine learning describes AI forecasting as software that finds patterns in past records to produce forecasts, and that keeps adjusting its models when fresh data comes in. IBM then lists how it differs from traditional methods: more drivers as inputs, nonlinear patterns, continuous learning and automation. In the same passage, IBM notes different risks: AI models lean heavily on the quality of the data and on proper checks of their output. IBM also says many forecasting processes combine both approaches.

In other words, the model is one part of the work. The figures fed to it, the people who check them and the decision that uses the result make up the rest. Read that way, AI forecasting changes how the estimate is produced, not who decides what to do with it.

Four Reasons a Business Forecast Misses

In Signisys’ working description, four causes account for a missed business forecast. They are our own grouping rather than a standard or a ranking, and each one comes with a question to ask.

1. The inputs are out of date. A figure last refreshed a quarter ago describes that quarter, not this one. Ask: when was each input last refreshed?
2. One-off events sit unmarked in the history. A large one-time tender, a closure or a promotion in the historical data looks like normal trade to anyone who reads the figures later. Ask: which past periods held an event that will not repeat?
3. Last period’s judgement is carried forward unchecked. Someone adjusted a number once, and the adjustment stayed in each later version. Ask: who changed the number, and why?
4. Nobody compares the last forecast with what happened. Without that comparison, a miss can go unseen, and nobody can say anything about forecast accuracy. Ask: where do the forecast and the actual figure sit side by side?

Where AI Forecasting Helps

The table takes the four causes in turn. The middle column says what a team can change by adding a model, while the last column says what stays with people.

Backtesting means running a model on past periods it has not seen and comparing its output with what was recorded, as IBM describes it. No row here claims that a model makes a forecast more accurate.

Where AI Forecasting Falls Short, and Four Questions to Ask

IBM lists four challenges: data quality, changing conditions, transparency and trust, and operational processes. In plain words, each one has a meaning for the reader:

• A forecast may not improve while the data behind it stays poor.
• A change that nobody recorded stays outside the inputs.
• A figure that nobody can explain is hard to defend in a board meeting.
• A forecast that no one acts on changes nothing in the business.

In Signisys’ working description, four questions come before relying on any forecast:

1. What data went in, and how current is it?
2. Which past events did the team treat as one-offs?
3. Who can explain the figure to the board?
4. Who acts on it, and by when?

How to Tell Whether a Forecast Missed

Forecast accuracy, in this article, means the gap between the forecast and the figure recorded for the same period, and whether the forecast was over or under. IBM names the size of the errors and whether a model leans too high or too low among the checks teams make.

Both measures start from two numbers in writing: the forecast as it was issued and the figure recorded afterwards.

The two figures can sit together in the monthly management pack. For reports produced on a schedule, see the use case on automating management reporting.

One Late Quarter, Traced Back

For example, take an invented mid-size trading firm whose quarterly sales forecast came in above the figure recorded at quarter end. The firm is not a Signisys client; it exists only to show the method, and its answers are not advice for yours.

1. The product list used as an input was a month old, and two lines on it had been withdrawn. Question 1 would have found this, and the person who maintains the list owns the fix.
2. A one-off government tender sat unmarked in last year’s history, which made the same quarter look stronger than normal. Question 2 would have found it, and the sales lead who knew of the tender owns the note.
3. The number had been rolled forward from the earlier quarter without a note. Question 3 would have shown that nobody could explain it. The finance lead then decides whether to keep it or replace it.
4. Nobody had compared the previous forecast with its result. With that comparison in writing, the first two problems could have shown a quarter earlier, and the person who acts on the forecast owns that step.

Nothing in this example predicts what your own firm will find.

What Is Left for Your Own Team to Settle

This explainer stops at causes and questions. Whether your own data, owners and review habits are in place is for your team to settle, and the article is not an audit of them. Whether AI forecasting suits a given figure depends on those answers, not on the model. Methods in steps, scoring, choosing between approaches and rules for cleaning data are separate subjects.

Ask for an outside view when nobody can say where a forecast figure came from. The same applies when two teams answer the four questions differently.

Signisys provides data and AI services through its business intelligence and analytics solution and its data and AI service page. This box describes a Signisys service, not neutral advice, and it makes no promise.

Frequently Asked Questions