Eight studies.
Four decades.
Nobody's marketing department.

The two ways

Restrictive. The planner's number is imposed on top of the model's number. The model gets overruled. This is what your planning system does today. It's what your spreadsheet does. It's what every override button in the industry does.

Integrative. The planner's judgment goes into the model as an input — one more thing the model knows about the future — and the model works out how much weight it has earned.

What they found

And the sentence that decided our product

Every other route to a better forecast asks the planner to give something up: accept a correction, justify yourself, be overruled by a model that doesn't know what you know. This one doesn't. Your judgment isn't overruled. It's taken seriously enough to become part of the model.

So Piwaka never overwrites your number and quietly shrinks it back. Your change goes in as what it actually is — something you know about the future that the model doesn't — and over time the system learns how much weight that kind of call, on that kind of product, has earned.

What we're not going to pretend

One company. Publishing. Weekly forecasts. The authors ask for replication themselves, and note that in the cases where a hard override happened to be right, integrating it clipped some of that upside. It is the best evidence in the field for how to build this. It is not the last word.

The situation

Every demand planner in the world does the same thing: a system produces a forecast, and then a human changes it. Everybody assumes the human is adding something. For sixteen years the evidence was patchy and contradictory — one company here, one industry there. So Fildes, Goodwin and De Baets collected every publicly available dataset they could find, put ~147,000 forecasts through a single common analysis, and asked the question straight.

What they found

Which means the planners were not wasting their time. They were adding real value on one change and losing it on the next — and the two went into the system looking exactly the same. A number, typed over another number. No reason. No author. No confidence.

That's the whole problem in one line: nobody can tell which half is which. Not the planner, not their manager, not the platform, and not the vendor selling them the forecast.

The part the industry doesn't print

The researchers then modelled what actually drives a planner to adjust. It was not, mostly, the new information they'd learned. It was the cues on their screen:

Read that again. The software is teaching the habit. The tools built to help planners adjust are showing them the things that make their adjustments worse.

What they found

Planners picked the right direction 69.1% of the time when cutting a forecast. Going up, they were right 59.9% of the time — barely better than a coin toss, in the direction everyone leans. The authors' explanation is blunt: "optimism bias or motivations to inflate forecasts were widespread."

Why this happens, and it isn't the planner's fault

Think about who pushes a number down. It's the planner, quietly, because they can see the promotion has ended and the spike is over. Nobody is in the room. Nobody is watching.

Now think about who pushes a number up. It's the deal that's going to land. It's the target. It's the exec who wants the quarter to look a certain way. The pressure to raise a forecast almost always comes from someone who doesn't carry the inventory. The optimism isn't a personal failing. It's structural — and it's been measured, in four industries, for forty years.

The situation

A univariate statistical system produced a baseline forecast. Planners then adjusted it with what they knew — most importantly, promotions the time-series model could not see. Promotional weeks are the hardest, highest-stakes case, so the researchers zoomed in on them.

What they found

The pattern was calibration, not carelessness. Managers knew a promotion was coming and the system did not, so they raised the number, and the direction was right. But when the adjustments were overly large, accuracy deteriorated — the value came from modest corrections, not the big swings.

The bit worth keeping

The researchers then built a hybrid — the promotion-aware model and the planners' judgment together — and found that "human experts still added value to the transfer function models." Neither alone was best. The combination was.

That is the integrative result again, from a different dataset: judgment is worth keeping, but it belongs combined with a model that can see what the planner cannot, rather than layered on top of one that is blind.

What they found

Read that slowly

Planners are not responding to the forecast's errors. They are repeating themselves. The adjustment is anchored on the last adjustment, not on evidence — and it happens in 37 countries at once, which rules out any explanation involving one bad team or one bad manager.

Case 02 found exactly the same thing from a completely different direction: when researchers modelled what drives an adjustment across 147,000 forecasts, the planner's own previous adjustment came out as one of the strongest — and least diagnostic — drivers. Two independent methods, two continents, same answer: adjustment is a habit.

The intervention

For a year, the researchers simply watched: every expert's forecast, every statistical forecast, every actual. Then in September 2007 the experts were brought to headquarters, shown feedback on their own adjusting behaviour, and trained on how the statistical program actually worked — its ins and outs, what it could see and what it couldn't.

Then everyone went home and the researchers kept watching.

What happened to their behaviour — clear and solid

What happened to accuracy — and here we owe you the small print

The published abstract says accuracy "improved substantially." We went and read the full working paper. The underlying numbers are weaker than that sentence suggests, and you should know it:

• The headline accuracy gain carries a one-sided p-value of 0.066 — suggestive, not conventionally significant.
• On the relative measure, experts went from 1.8% worse-than-model to 0.2% worse-than-model — and the authors state plainly that this improvement is not significant (p = 0.289).
• The share of adjustments that damaged accuracy fell only from 50.2% to 48.3% (p = 0.069).
• Even after the training, around 25% of forecasts still carried large or extremely large adjustments — and they still skewed upward.

What we're not going to pretend

Before-and-after field study, no control group. The planners got feedback and training and a trip to head office with senior management visibly watching — you cannot cleanly separate those. Three-month follow-up. Twenty-one people.

What is solid is the mechanism, and it's corroborated from a completely different direction: Case 02 found that mathematically damping planners' upward adjustments beat both the planner's number and the untouched model — and that damping the downward ones made them worse, exactly as the direction rule predicts.

What they found

Why this one matters more than it looks

This is the study that should worry every forecasting vendor, including the one that owns us.

Giving a company a better forecasting engine does not automatically give them a better forecast. The number that leaves the building is the one the human touched last. If a smarter model just gives the planner more to over-react to, the improvement evaporates on its way to the system of record — and everybody involved keeps quoting the model's accuracy score as though it were the outcome.

RabbitHawk builds forecasting engines. So we have every commercial reason not to tell you this, and we're telling you anyway: an engine on its own is not enough.

The finding that reframes everything above

Buried in the review's opening is the reason planners have to adjust that has nothing to do with politics, optimism, or ego:

The data itself is wrong, systematically, and a human has to fix it by hand. That is not a bias to be trained away. It's a permanent, structural reason why adjustment can never be eliminated — and why "just stop adjusting" was never a serious answer.

The review also catalogues the rest of the honest reasons: promotions, advertising, price changes, holidays, regulatory change, insufficient inventory, government policy, competitor activity. Real things. None of them in your sales history. All of them arriving as an email.

What they found

This is the sobering backdrop to everything above. The number a planner starts from is, more often than not, already worse than doing nothing. Then a human changes it. The forecast that reaches the business is a shaky baseline and an unrecorded adjustment, stacked on top of each other.