The baseline is your current process, not a test set

A model that scores 0.91 on a held-out set is not information. It is a number with no unit attached. The question that matters is different and much harder to answer: compared with what your team does today, does this change any decision, and is that change worth money?

What the test set measures

A held-out split measures how well the model fits data drawn from the same distribution as the training data. That is a real and useful thing to know during development. It tells you whether you have leakage, whether you have overfit, whether the features carry signal at all.

It does not tell you:

• Whether the decision maker would have acted differently
• Whether the inputs will be available at the moment the decision is made
• What the cost asymmetry is between a false positive and a false negative
• Whether anyone downstream will trust the output enough to use it

We have seen a demand forecast beat the incumbent process by eleven percent on MAPE and change nothing at all, because the planning team placed orders on a Tuesday against a forecast that refreshed on a Wednesday.

Measure against the incumbent

Before any modelling, write down what happens today. Usually it is one of three things:

1. A rule. “Reorder when stock falls below two weeks of cover.” Rules are excellent baselines and are often much harder to beat than people expect.
2. A person. Someone with fifteen years of context makes a judgement. Sample a few hundred of their past decisions and score them the same way you will score the model.
3. Nothing. The decision is made late, or not made at all. This is the easiest case to beat and the one where value is usually largest.

Then score the incumbent on the same metric, on the same period, with the same data availability constraints the model will face. If you cannot construct that comparison, you do not yet understand the decision well enough to automate it.

The availability constraint is the one people miss

Most disappointing deployments trace back to a feature that existed in the training data but does not exist at decision time. The nightly aggregate has not run yet. The status field gets corrected two days later. The customer record is completed by a human after the fact.

The discipline is simple and tedious: for every feature, write down the moment it becomes available and compare it to the moment the decision is made. Anything that fails that test comes out, and you retrain. The score will drop. The score was wrong before.

What good looks like

A useful evaluation reads something like: against the current reorder rule, over the last four quarters, with only data available at 06:00 on the order date, the system would have reduced stockouts by nine percent at equal holding cost. That is a sentence an operations director can act on. AUC 0.94 is not.