How do you prevent a RAG system from returning documents the user should not see?

You tag every document at index time with access control metadata like user role, business unit, state, and line of business. At query time, you build a filter expression from the user's session context and apply it before running the vector search. This ensures the search only runs over documents the user is allowed to see. Filtering after retrieval does not work because you lose results and you cannot tell if the filtered documents were the best matches.

What is the right chunk size for insurance policy documents?

It depends on how the documents are structured and how users ask questions. I have seen 256 tokens work well for policies with short, self-contained sections, and 1024 tokens work better for policies with long narrative exclusions that need more context. The way to find out is to build an eval set and test different chunk sizes against it. If recall drops when you make chunks smaller, the chunks are cutting up information that needs to stay together. If precision drops when you make chunks larger, you are including too much irrelevant text.

How do you know when retrieval quality has degraded?

You run an eval set on a schedule, at least weekly and ideally every time you re-index the knowledge base. The eval set is a list of known questions and the document IDs that should be retrieved for each question. You run the eval, count how many test cases passed, and alert if the pass rate drops below your threshold. I use ninety percent as a threshold, but the right number depends on how much risk you can tolerate. This catches regressions from changes to chunking, embedding models, or document quality.

Can you use the same RAG pipeline for claims triage and coverage determination?

You can use the same infrastructure, but you need different retrieval profiles. Triage queries optimize for latency and return fewer results because the adjuster is getting oriented, not making a final decision. Determination queries optimize for completeness and return more results because the adjuster needs to see every relevant exclusion. The application passes a parameter that tells the retrieval logic which profile to use, and the logic adjusts filters and result counts accordingly.

Operationalizing RAG Accuracy in Insurance Claims Workflows

A retrieval system can be confidently wrong for months before anyone says anything. The adjusters notice first. They stop quoting it, start checking the source document every time, and eventually route around it. Nobody files a bug, because nothing is broken in a way that produces an error.

What has usually happened is that the knowledge base moved and the retrieval did not. New documents landed, an endorsement superseded a clause, somebody re-indexed with a different chunk size. Each change was fine. The sum of them was a system returning the wrong policy exclusions to people making coverage decisions.

Operationalizing RAG accuracy in insurance claims workflows is not a prompt engineering problem. It is a versioning problem, an evaluation problem and an access problem, and all three need to hold before the thing goes in front of an adjuster.

The version question is the whole problem

Insurance has a constraint that most RAG demos never encounter. Policy language changes. Endorsements get added mid term. State filings update exclusions.

An adjuster needs the version of the policy that was in force on the date of loss. Not the current version. Certainly not a blended summary of three versions that the vector store returned because their embeddings sat close together.

If your pipeline cannot answer "which document version is this, and why that one", it is not an assistant. It is a liability with a chat interface.

Concretely, this means effective and expiration dates belong in chunk metadata at ingestion, and the date filter belongs in the query path as a required argument rather than an optional one. A missing date should fail loudly. The alternative, where a missing date quietly defaults to the current version, is the exact bug that is hardest to see and most expensive to find.

Build the eval set from closed claims

You need evaluation before you need observability, and it has to run without anyone remembering to run it.

The temptation is to write test questions. Do not. Synthetic questions test the system against your idea of the work rather than the work, and they are reliably easier than the real thing.

Pull a sample of closed claims instead. Each one has a question that was really asked, a decision that was really made, and the policy sections that were really cited. That gives you a test case with a known right answer: given this question and this date of loss, the system must retrieve this section of this version of this policy.

A hundred of those, runnable on demand, changes how the system is maintained. Change the chunking strategy and you know within minutes whether recall held.

The metric is not cosine similarity. It is whether the correct document appeared in the top k actually sent to the model as context, where k is whatever you really use, usually five or ten depending on what the context window costs you. A system that ranks the right document eleventh with a beautiful similarity score has failed the only test that matters.

What to measure	What it tells you	When it moves
Recall at k, against closed-claim cases	Whether the right source reaches the model at all	Re-indexing, chunk size changes, corpus growth
Version accuracy	Whether the retrieved version matches the date of loss	New endorsements, filing updates, any change to date filtering
Unanswerable rate	How often the system should decline and does	Coverage gaps in the corpus, new lines of business
Access denials at retrieval	Whether segmentation is actually being enforced	Role changes, reorganisations, new data sources

Run the first two on every ingestion and every index change. Run all four on a schedule. The scheduled run is what catches slow corpus drift, which is the failure that erodes trust rather than breaking anything.

What goes wrong that evaluation will not catch

Two failures sit outside the eval set, and both are worth designing for explicitly.

The first is the confident answer to a question the corpus cannot support. Retrieval returns the five nearest chunks whether or not any of them is relevant, and a model handed five weakly related passages will often synthesise something plausible from them. The fix is a relevance floor: if the best match is below a threshold, the system declines rather than answers. Picking that threshold is empirical and slightly uncomfortable, because it means accepting a visible refusal rate in exchange for a lower invisible error rate. That trade is almost always correct in claims work.

The second is retrieval that succeeds and should not have. A document the adjuster is not entitled to see, surfaced because it matched. Segmentation in insurance runs by line of business, by state, sometimes by customer, and the entitlement can change over the life of a claim. Filtering after retrieval is not sufficient here, because what the system retrieved is itself a record. The filter belongs in the query.

Drift has a human tell before it has a metric

Worth knowing because it is the cheapest signal available: people stop using the system before any dashboard shows a problem.

If adjusters are opening the source document every time rather than trusting the answer, the system has already failed, whatever the metrics say. That behaviour is observable. Click through rate from an answer to the underlying document, or the share of sessions where someone asks the same question twice in different words, will show you a trust problem weeks before a quality metric does.

Both are easy to log and almost nobody logs them.

A sequence that works

Date filtering first, because retrieving the wrong version is worse than retrieving nothing and it is the error that produces real consequences. Then the eval set from closed claims, because without it every subsequent change is a guess. Then access filtering in the query path. Then the relevance floor. Retrieval quality tuning comes last, which feels wrong to most engineering teams and is nevertheless the right order: tuning a system you cannot measure is just moving it.

None of this is difficult work. It is mostly unglamorous data plumbing done before the interesting part, which is exactly why it tends not to get done. We end up doing a lot of this sort of retrieval and governance work, and the pattern is consistent enough that it is worth saying out loud.

Frequently asked questions

How many eval cases are enough?

A hundred real cases will tell you more than a thousand synthetic ones, and fifty is enough to start catching regressions. What matters more than the count is coverage: the set should include the lines of business, the date ranges and the awkward cases that caused disputes, because those are the ones where a wrong answer is expensive. Add cases as new failure modes appear rather than trying to be exhaustive up front.

Should the system ever refuse to answer?

Yes, and the refusal rate is a feature worth tuning rather than a defect to minimise. In claims work, a visible "I cannot find this in the policy in force on that date" is cheap, while a confident wrong answer that an adjuster acts on is not. Set a relevance floor, measure how often it fires, and review a sample of the refusals to check it is declining for the right reasons.

Does re-chunking mean re-running everything?

It means re-running the retrieval evaluation, which is precisely why the evaluation needs to be a command rather than a project. Chunk size interacts with recall in ways that are hard to predict from first principles, so the practical approach is to change it, measure, and keep the result only if recall held. Without an eval set this decision gets made on impressions, and impressions reliably favour whatever was tried most recently.

How do you handle a policy that was amended mid claim?

The date of loss governs, so the retrieval filter uses it rather than the current date, and the amendment should be retrievable as its own document with its own effective date. The harder part is the interface: the answer needs to show which version it relied on, prominently enough that an adjuster notices when it is not the one they expected. This is one of the few places where a UI decision is also a control.

If you are maintaining one of these

The systems that fail here rarely fail loudly. They drift, people quietly stop relying on them, and the problem surfaces as a usage number rather than an incident. If that sounds familiar, I am happy to take a look at what you have. There is more on what we have built, or just get in touch.