AI explainability is not optional in regulated compliance

Picture the review. A supervisor asks why a case was signed off as compliant, and the honest answer is that the system flagged it green. That is not an answer the FCA accepts, and it is not one a senior manager can stand behind.

Here is the thesis, stated plainly. Explainability is not a premium feature you add once the AI is working. In regulated lending it is the baseline, the thing that has to be true before the tool is allowed near a live file.

The context makes it urgent. 75% of UK financial-services firms now use AI, up from 58% in 2022, and another 10% plan to within three years (FCA and Bank of England, 2024). Adoption is settled. The open question is no longer whether regulated firms use AI, but whether they can explain what it did.

That question has a clean test. If an AI tool flags a mortgage file as non-compliant, can the compliance officer reviewing that flag see which rule was breached, which part of the file triggered it, and what evidence the model relied on? If yes, the human can genuinely exercise oversight. If no, the firm has automated a guess and dressed it as assurance.

The weakest version of explainability is a number. A model returns 0.86 and calls it confidence. That tells a reviewer nothing about the rule, the document, or the reasoning, and it cannot be repeated back to a customer or a supervisor.

The stronger version is specific. The ICO, with the Alan Turing Institute, sets out six kinds of explanation a firm should be able to give for an AI-assisted decision: the rationale, who is responsible, the data used, fairness, safety and performance, and the impact (ICO, Explaining decisions made with AI). Note what that list is really asking for. Not how the model works in the abstract, but why this decision, on this case, can be justified.

For a compliance flag that means three things on every output: the specific rule or requirement it maps to, the exact text in the file that triggered it, and the evidence trail behind it. Get those three and the reviewer is doing real work. Miss them and you have a black box wearing a lanyard.

There is no separate AI rulebook coming, and that is the point. The FCA's approach to AI is technology-neutral and outcomes-focused. Accountability sits where it already sits, with the senior managers named under SM&CR and the obligations of the Consumer Duty.

Read through that lens, explainability stops being a technical nicety and becomes a named person's problem. Under SM&CR a specific individual answers for the outcome of a case. That person cannot delegate their accountability to a model. To sign, they need to see the reasoning, and to see the reasoning, the tool has to expose it.

This is also why 84% of firms already report a person accountable for their AI use (FCA and Bank of England, 2024). The accountability exists. What too many tools fail to provide is the evidence that lets the accountable person do their job. A compliance oversight lead at a UK mortgage network put the bar simply in one of our conversations: any AI has to give them at least the same level of assurance they can give the FCA today. That is the correct test, and unexplainable AI fails it.

The case for explainability is not only regulatory. Unexplainable systems fail on their own terms, and they fail in the places that matter most.

Regulated file review is full of exceptions: the unusual income structure, the vulnerable customer, the product used outside its typical profile. A model that cannot show its reasoning cannot be corrected when it meets one, because no one can see where it went wrong. One mid-size lender told us their self-built tool worked around 60% of the time and simply could not handle exceptions, model drift, or fraud that was itself AI-generated. Sixty percent is not a compliance control. It is a liability with a dashboard.

The wider research on autonomous systems points the same way. McKinsey found that 80% of organisations have already met risky behaviours from AI agents, including improper data exposure and unauthorised access (McKinsey, 2025). The common thread in those failures is not capability. It is systems acting without a trace anyone can follow. In a regulated firm, a decision you cannot reconstruct is a decision you cannot govern.

None of this is an argument against AI in compliance. It is an argument for building it the way regulated work already runs.

That is the standard we hold ourselves to. Everything in the Curvestone platform is designed from an FCA-first position: every flag references the specific rule behind it, every rule is traceable to its source, and every decision and override is logged with the evidence it relied on. A person makes the final call on each case, with that evidence in front of them, and the audit trail is there whether or not anyone ever asks for it.

Done this way, explainability is not a tax on speed. It is what lets a firm move fast and still answer for every decision. Reviews that took hours happen in minutes, and the record is stronger at the end than it was under manual sampling.

If you are weighing an AI compliance tool, treat explainability as the first filter, not the last. We have written about what governed AI looks like in mortgage compliance and how to weigh building it yourself against partnering. Be sceptical of any vendor who cannot show you the rule, the trigger, and the trail. That is the whole job.