Most AI problems are not model problems

Someone senior in finance told me recently that they had stopped trusting AI for analysis. They had uploaded spreadsheets, asked for the numbers to be worked through, got back something that looked right, and used it. Reviewing it later, some of the formulas were wrong.

The conclusion they had drawn was that the tool was not good enough yet.

I do not think that was the problem.

Look at what was actually asked for

One message. Read this file, understand how the business works, do the arithmetic, interpret the result, and recommend something.

That is five different kinds of work bundled into a single instruction, and only two of them are the sort of thing a language model should be doing on its own. Nothing in that setup was ever going to catch a wrong formula, because nothing in that setup was checking anything. There was no step where the output got compared back against the input, and no point at which a person was required to look.

A better model would have got the arithmetic right more often. It would not have added a checking step, because that is not what models are for.

Better prompting is not the fix either

This is where most AI advice lands, and it is why a lot of AI training produces very little. You can write an excellent prompt for that task and still have no validation, no persistent context, and nowhere for a decision to be recorded. You have improved one step in a process that has no other steps.

The prompt was never the constraint.

The useful instinct

When you are deciding who should do what, this one is close to free:

LLMs reason. Code calculates.

It is not a law. Models can run code now, and often should. But as a first pass at assigning work it removes the most common category of confidently wrong output, which is a model reasoning its way to a number instead of computing it.

Applied to that finance task, the same work looks like this:

• Reading the file is mechanical, so a tool does it

• The calculations go to something deterministic, a spreadsheet formula or a script, so the same inputs give the same answer every time

• The relevant policy gets looked up rather than recalled

• Interpreting what changed goes to the model, which is what it is good at

• The figures get checked against the source

• A person approves it before it goes anywhere

Same task, same tools available. The difference is that each step is assigned to whatever is actually suited to it, and two of those steps exist purely to catch the others being wrong.

What this adds up to

Most businesses I get called into do not have an AI problem. They have an AI systems problem. The model was fine. What was missing was somewhere permanent for the context to live, something enforcing the standard before the work rather than after it, and someone required to look before anything left the building.

That collection of missing pieces has a name worth having, because naming it is what lets you go and build it: context, instructions, knowledge, memory, workflows, tools, models and governance. An operating system for the way you use AI, rather than a chat window you keep starting over in.

The model is one component of that. It is not the system.

Where this goes next

I write up the builds themselves a few times a month, including the parts that go wrong: what I built, how, and what broke. It goes to a small list and it does not get published here in full.

If you want the longer version of this argument and the thing I built off the back of it, that is where it lives.