Spot-checking case files was a workaround, not a strategy

Ask a compliance team why it reviews around 10% of its files and you get one of two answers. The first is "it is risk-based". The second, said more quietly, is "it is what we can get through". The second is the honest one.

The maths explains it. A manual file review runs from under an hour to the best part of two. At that rate, checking every case would need a team few networks or lenders could fund. So the industry checked what it could, called the result a sample and built a process around it. A compliance director once described it to me as "compliance theatre".

That was a fair compromise while every file took an hour or more to read. It is a weak one now.

The pressure is the Consumer Duty. The FCA says its rules require firms to "monitor and regularly review the outcomes their customers are experiencing" and to "identify where customers or groups of customers are not getting good outcomes and understand why". A firm's governing body has to review and approve its assessment of customer outcomes at least annually.

Groups of customers is where a small sample struggles. A 10% sample can show that one file was poorly advised. It cannot easily show that one adviser keeps missing the same vulnerability signal, or that one type of case keeps failing the same check, because the pattern sits in files nobody opened.

Then there is a line I think most firms skip. The FCA suggests firms ask themselves "whether they are using the same MI capabilities they use to inform other elements of their business, such as product development or sales, to also monitor outcomes." Sales MI usually covers every case. File review MI usually covers one case in ten.

It does, and I would not tell anyone otherwise. FG22/5 says that some monitoring may cover all customers, "while other types of monitoring, such as distributional analysis or file reviews will be based on a risk-based sample." It also expects file reviews to happen "at regular intervals or on an ad hoc basis" rather than all the time. A firm that samples is not breaking a rule.

But read the phrase again: a risk-based sample. A sample whose size comes from reviewer hours, and whose cases are mostly picked at random, is a capacity-based sample with a risk-based label. The difference is what a firm has to explain when something goes wrong in the 90% nobody read.

The guidance also says the FCA expects firms "to continually review and develop their frameworks", and it expects firms with more sophisticated data strategies to have more detailed monitoring strategies. A sample that was proportionate when every file took an hour to read does not stay proportionate when it no longer does.

So my argument is not that sampling is banned. It is that the reason for the sample size has changed, and the old reason no longer holds.

• They treat the sample as the control. The sample is one input into outcome monitoring, not the whole of it.
• They pick cases at random and call it risk-based. Risk-based selection needs a risk signal, and most firms have none until a file is read.
• They report the sample's pass rate as if it described the book. A 95% pass rate on one case in ten says little about the other nine.
• They assume checking every case means a machine makes the call. It does not, and it should not.

A consistent check runs on every case, and every finding points to the page and document it came from. A person reviews the findings and approves or overrides them. The deep human file review then goes where the full check points: the advisers, case types and findings that need judgement. Only then is the sample risk-based, because the risk signal comes from every case, not from chance.

The MI changes too. Instead of a pass rate on a sample, the board sees which advisers miss which checks, how often and on what kind of case. That is the "groups of customers" view the Duty asks for.

This is how we built Curvestone: each finding shows its reasoning and source evidence, a person approves or overrides it, and every step goes into the audit trail. The software does the reading. The accountable person does the judging.

1. Write down why your sample is the size it is. If the honest answer is capacity, say so in your next board pack.
2. Look at how cases are selected. If selection is random, the sample is not yet risk-based.
3. Compare the coverage of your sales MI with the coverage of your file review MI.
4. Work out what one manual file review costs you today, so the comparison with checking every case is honest.
5. Ask what your next annual outcomes assessment would say if it covered every case, not one in ten.

An earlier version of this argument appeared in Mortgage Solutions (July 2026).