AI in compliance: why explainability is not optional

The FCA has been consistent in its messaging on AI: it is not opposed to the use of AI in regulated activities, but it expects firms to be able to explain AI-assisted decisions — to customers, to their own oversight functions, and to supervisors. For compliance teams adopting AI tools, this creates a clear design requirement: black-box outputs are not acceptable.

Explainability in a compliance context means more than a model confidence score. When an AI system flags a case file as non-compliant, the compliance officer reviewing that flag needs to understand which rule was breached, which part of the documentation triggered the flag, and what evidence the AI used to reach its conclusion. Without that, the human reviewer cannot meaningfully exercise oversight — and the firm cannot demonstrate to the FCA that a human was genuinely in the loop.

This shapes how compliant AI tools should be built. Every flag should reference a specific rule. Every rule should be traceable to the underlying regulatory source. Every AI decision should be logged with the evidence it relied on, in a format that survives the regulatory review process. Firms evaluating AI compliance tools should treat explainability as a baseline requirement — not a premium feature — and should be sceptical of any vendor that cannot demonstrate it.