All articles
governance7 min read

Model Risk Management for Agentic AI in Finance

Agentic systems create new model risk vectors because their loops can compound small errors over time. This post walks through the lifecycle controls that keep outputs traceable without freezing operational value.

By Andriy Markiv, Goatechs LLC

Share LinkedIn X Facebook

 

Model risk management was written for a thing that holds still. A model with fixed parameters, training data you can snapshot, outputs you can score against a holdout set. Validate it, document it, revalidate on a schedule.

Now point that framework at an agentic system built on retrieval. The language model changes when the provider updates it. The vector store grows every time somebody uploads a document. The decision loop calls external services whose behaviour nobody in the building controls. Three of the four things MRM assumes are fixed are moving, and the fourth is the prompt.

The temptation at this point is to argue that AI is special and needs its own rules. That argument loses, and it should. The useful move is to show how controls a risk function already enforces map onto a system whose logic is spread across retrieval, reasoning and action.

Where the vocabulary stops matching

It is worth being precise about the mismatch, because most of the friction in these conversations comes from two groups using the same words for different things.

MRM expectsAn agentic RAG system hasWhat to offer instead
A fixed model artifactA hosted model that changes under youPinned model versions, a behavioural test suite run on every version change, and a record of which version served which decision
Snapshotable training dataA corpus that grows continuouslyVersioned corpus state, content hashes per document, and the corpus version recorded with each retrieval
Validation against a holdout setNo ground truth labels, open ended outputsAn eval set built from decisions already made and closed, scored on retrieval and on outcome separately
A bounded functionA loop that can call tools and wanderAn explicit, enumerated tool list and a hard iteration ceiling
Explainable coefficientsReasoning you cannot inspectThe inputs, the retrieved context, the proposed decision and the policy evaluated against it, logged together

Nothing in the third column is novel risk management. It is the same evidence, gathered from a system that does not volunteer it.

Bound the loop before you validate it

An agent is a loop. Read, decide, act, observe, decide again. In a lending context that might mean retrieving underwriting guidance, calling a credit bureau, running a calculation and returning a recommendation.

The risk is that loops wander. If retrieval surfaces the wrong section of policy, or the model misreads a numeric threshold, the agent can produce an answer that breaks a rule everybody assumed was hard coded.

Two constraints do most of the work, and both are set at design time rather than discovered in validation.

First, limit the action space. The agent gets an explicit menu of tools, each with a declared input schema, output schema and the conditions under which it should be called. No general purpose code interpreter, no ability to compose arbitrary API calls. It selects from the menu; it does not invent menu items.

Second, separate retrieval from authority. The agent can read from the vector store and can propose. It holds no write access to any system of record. What comes out is a structured recommendation that something else decides whether to act on.

A third, duller control matters more than either: cap the iterations. An agent allowed to loop until satisfied will, on some input, loop until the budget is gone. A ceiling turns an open ended failure into a bounded one that escalates.

Drift is three different problems wearing one word

"How will you detect drift" is the question that comes up every time, and answering it well means pulling it apart first, because the three kinds have different detectors and different owners.

  • Model drift. The provider updates the model and behaviour shifts. Detection is a fixed battery of cases replayed against each new version, comparing decisions rather than text. This is the only one of the three with a clean trigger, because version changes are announced.
  • Corpus drift. Documents are added, superseded or withdrawn, and retrieval quality moves without anything else changing. Detection is a retrieval eval run on a schedule and on every ingestion, measuring whether known questions still surface known sources.
  • Population drift. The inputs change. The system is being asked about cases unlike the ones it was validated on. Detection looks like any other population stability work, and a risk function usually already does it for other models.

Naming them separately changes the conversation. "We monitor for drift" invites follow up questions. "We replay 200 decision cases on every model version, run retrieval eval on every ingestion, and track input distribution monthly" ends them.

The documentation problem is a logging problem

What a validator ultimately wants is to take one decision the system made and reconstruct it. Not in general terms. That decision, on that day.

Which means the record written at decision time has to carry: the input, the corpus version and the specific documents retrieved with their hashes, the model and version, the proposed decision, the policy version it was evaluated against, and the outcome. Written together, in one record, at the moment it happens.

Assembled later from four systems, it is not evidence. It is an argument.

This is the piece teams most often leave for the end, and it is the piece that determines whether the rest of the work can be used. A system with modest retrieval quality and complete decision records can be validated, operated and improved. A system with excellent retrieval and partial records cannot be defended, and no amount of later accuracy fixes that for decisions already made.

What this looks like as a sequence

If a risk function is waiting on you, the order that tends to work is: bound the loop, log the decisions, build the eval from closed cases, then tune quality. It is roughly the reverse of what an engineering team wants to do, and the reason is simple. The first three produce the evidence that makes the fourth reviewable. Tuning first produces a better system nobody can approve.

It is also worth saying plainly what this work is not. It is not a claim that the system is low risk. A well documented agentic system is still an agentic system, and the honest output of this exercise is often a narrower scope than the one you started with. That is a good result. Most of the architecture work that goes well ends with a smaller autonomous band and far more confidence in it.

Frequently asked questions

Does an agentic system need to be registered in the model inventory?

In most institutions, yes, and usually as more than one entry. The retrieval component and the decision component have different failure modes, different owners and different validation evidence, so treating them as a single inventory item tends to produce documentation that satisfies neither. Deciding this early avoids rework, since the inventory structure shapes what has to be logged.

How do you validate something with no ground truth?

By building ground truth from decisions already made. Closed cases have a documented outcome and the sources that were relied on, which gives you both a retrieval target and an outcome target for each case. It is not a holdout set in the statistical sense and it should not be described as one, but it is defensible, repeatable, and grounded in the institution's own records.

What happens when the provider deprecates the model version we validated?

This is the scenario worth planning for before it happens, because the timelines are not generous. Pinning a version buys time rather than safety. The practical answer is a revalidation battery that can run quickly against a new version, and an agreed threshold for what counts as a material behaviour change. Without that, every provider update becomes an unscheduled project.

Is a self hosted model easier to get through MRM?

It removes one moving part and adds others. The model stops changing without warning, which genuinely simplifies the version story. In exchange you own the serving stack, the patching and the performance characteristics, and those come with their own control expectations. Which trade is better depends on how much of the organisation's existing control framework already covers hosted infrastructure.

If this is the conversation you are in

The hardest part is usually translation: getting an engineering design and a risk framework to describe the same system in words both sides accept. If that is where you are, I am happy to work through it. There is more on how we approach this, or you can get in touch directly.

model riskagentic systemsfinancial services