AI explainability in banking: explain the process, not just the model
Explainability breaks down between process, code, data and decisions. An explanation that stops at the model misses most of it.
AI explainability in banking is the ability to show how an AI system contributed to an outcome, in terms a customer, a model validator, an auditor and a supervisor can each check. Banks usually start with the model: which inputs drove a score, and by how much. That is necessary but not sufficient. Banks have data governance, lineage and regulatory reporting in place, but explainability still breaks down between process, code, data and decisions. An explanation that stops at the model does not say which data reached it, which rules were applied around it, or who reviewed its output.
Two halves of explainable AI in banking
Model explainability answers why the model produced this output from these inputs. Techniques such as feature attribution, reason codes and surrogate models do this well for many model types, and they are what model validators and fair-lending teams rely on. In US consumer credit, a lender that declines an application must give the specific principal reasons, however complex the model behind the decision.
Process explainability answers how the outcome was reached end to end. Which source data fed the model, and what transformed it on the way? Which business rules, thresholds and overrides were applied before and after the model? Which version of each rule? Where did a person review the output, and on what basis? What happened between the model's output and the decision the customer received?
Regulators increasingly ask the second question. The EU AI Act gives people affected by a high-risk decision, such as a credit decision, a right to a clear and meaningful explanation of the role the AI system played, from 2 December 2027. SS1/23 in the UK and SR 26-2 in the US expect firms to understand model inputs and govern adjustments to model outputs. Neither can be satisfied from the model documentation alone.
Where explainability breaks down
Between data and model. The model was validated on one dataset. In production its inputs arrive through extraction jobs, mappings and transformations in application code. A changed mapping changes the inputs without changing the model, and the model explanation will faithfully explain an output built on the wrong data.
Between model and decision. A score becomes a decision through policy rules, cut-offs, overrides and manual review, many of them held in code or spreadsheets owned by other teams. The reason codes describe the score. They do not describe the override that changed the outcome.
Between rule and agent. As banks move from models that score to agents that act, the problem sharpens. An agent's rules can live in its prompt, alongside the copies in policy, code and spreadsheets. Nobody can show the agent applied the approved version. The pilot works, and compliance cannot sign it off.
Explaining the process
A complete explanation joins the model explanation to the process around it: the source data and its path, the transformations in code, the rules and their versions, the human steps and the final decision, each attributable to an owner.
xflow builds that view, which we call process context. It connects the regulatory objective to the process steps and to the code, data, spreadsheets and manual routines that deliver the outcome, and makes the whole executable, so real data can be run through it to show how an outcome was produced. The rules an AI agent must apply are held once as governed definitions, and the agent points at them, so the rule applied at runtime is the rule that was designed, validated and evidenced.
Questions about AI explainability in banking
Why does AI explainability matter in banking?
Because banks must justify decisions to customers, model validators, auditors and supervisors, and because compliance will not approve AI in a regulated process it cannot explain.
Is model explainability enough?
It explains the model's output from its inputs. It does not explain where those inputs came from or what happened to the output afterwards, which is where many errors and most supervisory questions sit.
What does the EU AI Act say about explainability?
Providers of high-risk systems must make them interpretable enough for deployers to use properly, and people affected by a high-risk decision have a right to a clear and meaningful explanation of the AI system's role. For credit scoring these duties apply from 2 December 2027.
How do you explain a decision made by an AI agent?
By showing the process it acted in: the data it used, the governed rules it applied and their versions, the checks around it and the human authority it worked under.