Backtesting AI forecasts: What did the planner know at ten?

A revised promotion date can flatter a forecast in hindsight. Test order suggestions against the information planners actually had at the deadline.

The promotion was scheduled for Tuesday. At least that is what the planning system says today. When the planner placed her order on Monday morning, it still said Wednesday. Test a new forecasting model against today’s export and it may anticipate the Tuesday rush remarkably well. Yet the crucial information would have been missing when the order was due. To assess an AI forecast for use in purchasing, the team needs a way to replay the working day as it actually unfolded.

Consider a fresh produce order for one store. The setting draws on food industry operations, where our venture Deepico develops cloud ERP. The test design below is a proposal for a planning team. It does not describe an installed forecasting feature or report results from a customer. Food industry context

Draw the line at the order deadline

Write one sentence that both planning and engineering can sign off on. “At 10 am local time on Mondays, we propose an order quantity for delivery on Tuesday.” That defines the moment of decision, the forecast horizon and the action. A Monday evening forecast might be useful elsewhere, but it answers a different question if the supplier has already planned the route.

Give each test case a product, a location and an order deadline with a time zone. For each input record both the date it describes and the time the planning system actually received it. Sunday’s sales may arrive from the till at noon on Monday. They belong in Sunday’s sales history but were unavailable for the Monday 10 am decision. Apply the same check to stock corrections and revised promotions.

If you cannot substantiate an earlier value, mark the gap. Start keeping dated copies or a change log for that source today. A shorter test with reliable history tells you more about readiness than years of results contaminated by information from the future.

Three proposals for the same order

Produce three quantities for each deadline: the one planners actually approved, a simple rule and the candidate model’s forecast. For daily planning with a weekly pattern, start with sales on the same weekday of the previous week, provided the item was available then. Apply the same pack rounding and stock deduction to all three. Otherwise the apparent winner may simply have a better rounding rule and the team may give the credit to AI.

Work through the decision dates in sequence. At every deadline the model may train only on observations already available at that point. The same boundary applies when tuning its settings. Reserve a later, uninterrupted period for the final assessment and leave it untouched until then. This design draws on time series validation with a rolling forecasting origin. Method and benchmarks

When the last sale hides demand

A store sells its final crate at 3 pm. The daily report shows 30 crates sold. Whether customers wanted another five is unknown. If the test treats 30 as total demand, it may reward the very proposal that would have left the shelf empty early. Flag days with confirmed stockouts separately. If there are records of refused orders, document which portion of missing demand they capture.

Three views are enough for a first review. Show deviations on days with confirmed availability, the number of excluded or uncertain days, and results during promotions. Put the proportion of unknown cases alongside the headline result. If the missing information concerns the products with the largest losses, improve data capture there first. Extrapolating to the full range would be premature.

Count the cost of checking a suggestion

A model can narrow the gap between proposed quantities and demand while taking up more of a planner’s time. During a parallel trial, record how often someone opens a suggestion, how long the review takes and why the quantity changes. A missing pack weight points to a different problem from a misjudged promotion. One calls for correcting master data, the other for reviewing the forecast.

Here is an illustrative weekly calculation. Of 120 suggestions, 24 need five minutes of review each. That adds up to two hours per week. At an assumed CHF 75 per hour, the time is worth CHF 150. If the current method already takes 90 minutes for the same workload, the additional burden is 30 minutes, or CHF 37.50. That difference belongs in the comparison. Valuing staff time does not by itself establish a cash payment or saving.

From replay to the first live suggestion

Let the new method run in parallel first. Lock its recommendation before the order deadline. Planners continue to order using the current method and, where possible, record why the quantities differ. The next day, add sales, availability and known exceptions. The team can then see whether historical results hold up with today’s interfaces and working hours. Since only one order was placed, the outcome of the other option remains an estimate.

The decision to proceed should fit on one sheet. List the products tested, the forecast horizon, the number of usable cases, outstanding data gaps and the person who switches back to the existing method if an input arrives late. Start the joint review with a single past order. Can your team show what was actually known at 10 am? Design a forecast backtest with Deep Impact

For a related costing example, see What does a resolved AI case cost?.