Back to Blog

How to Implement RAG in Fintech: A Step-by-Step Guide

The architecture, the four decisions that actually determine whether it works, and the access control mistake that puts a document in front of a model before anyone checks whether the user was allowed to see it.

Andriy Markiv February 10, 2026 9 min read
Share LinkedIn X Facebook

 

Retrieval augmented generation is simple to describe and easy to get wrong in ways that do not show up in testing. You search your documents for the passages relevant to a question, hand those passages to a language model, and ask it to answer from them. The demo works in an afternoon. The version you can put in front of a compliance officer takes considerably longer, and the gap between the two is almost never the model.

This walks through building the second one. It assumes you are in financial services, which changes four things: your documents have effective dates and supersede each other, your users are not all allowed to see the same material, your answers may be regulated communications, and someone will eventually ask you to reproduce an answer you gave eight months ago.

  1. Question

    With the asking user's identity attached

  2. Hybrid retrieve

    Dense plus BM25, filtered by permission and date

  3. Rerank

    Cross-encoder over the union of both result sets

  4. Generate

    Answer constrained to the retrieved passages

  5. Cite and log

    Sources shown, full exchange recorded

Figure 1. The pipeline. Each stage can fail silently, which is why the evaluation step exists and why retrieval and generation are measured separately rather than as one score.

Step 1: Write the questions before you choose the technology#

Most RAG projects start by picking a use case. Start instead by collecting fifty to a hundred real questions that people actually ask today, each paired with the document and passage that correctly answers it. Get them from support tickets, from the compliance inbox, from analysts, from whatever channel currently absorbs the work.

This set is the most valuable artifact in the project and it has to exist before you build, for a reason that is easy to miss. Once you have a working system, every evaluation set you write will be unconsciously shaped by what you have seen it handle. A set written first is the only honest one you will ever get. It also settles arguments: when someone proposes a different embedding model or a larger chunk size, the question stops being a matter of opinion.

Step 2: Choose the vector store by what you already run#

This decision matters less than the amount of time usually spent on it. All four options below will retrieve adequately at the scale most financial services corpora actually reach. What differs is what you have to operate, and where the data sits.

StoreChoose it whenThe real tradeoff
pgvectorYou already run Postgres and would rather not add a system to monitor, back up and secure.Your vectors live beside your relational data, so permission filters are a join rather than a sync. Slower than a dedicated store at very high dimensionality and volume, which most corpora never reach.
PineconeYou want it managed and your data residency and vendor review will allow a hosted store.Least operational work, and a vendor in your compliance scope. Filtering is good but your access control now lives in two places.
WeaviateYou want hybrid search built in rather than assembled.Saves you writing the fusion logic. More to operate than pgvector if self hosted.
QdrantRetrieval latency is genuinely a constraint you have measured.Strong performance and good filtering. Another service to run for a gain you should confirm you need.

If you already run Postgres, start with pgvector. Not because it wins on benchmarks but because the thing that will actually bite you is keeping permissions and effective dates consistent between your records and your index, and pgvector makes that a query instead of a pipeline.

Step 3: Ingestion is where financial documents punish you#

Generic chunking advice, split every eight hundred characters with an overlap, fails on financial documents in two specific ways that are worth knowing before you discover them.

Tables lose their headers. A fee schedule or a rate table carries its meaning in the header row and the column labels. Chunk it by character count and you get a fragment that reads 0.25% 1.10% 2.00% with nothing to say what those apply to. The retrieved passage is then worse than nothing, because the model will confidently attach them to whatever the question was about.

Defined terms lose their definitions. A chunk from the middle of a contract saying the Borrower shall maintain the Required Ratio is unusable on its own. Those capitalized terms were defined in section 1, which is now a different chunk forty pages away.

Both have the same fix: carry context down into the chunk. Prepend the document title, section heading and table caption to every chunk, and resolve or append defined terms where you can. This duplicates text across chunks and costs storage, and it is worth it every time. A related technique, parent-child retrieval, embeds small precise chunks for matching but returns the larger surrounding section for the model to read, which gets you accuracy on the search and context on the answer.

  • Parse with something that understands layoutUnstructured.io or LlamaParse for PDFs, EDGAR filings and spreadsheets. A plain text extractor will destroy table structure and you will not notice until retrieval quality is unexplainably poor.
  • Chunk on structure, not lengthSection and subsection boundaries, with headings prepended. Fall back to length only inside a section that is genuinely too long.
  • Attach effective dates, not just ingestion datesA regulation or a policy has a date from which it applies and often a date it was superseded. Retrieval that cannot filter on these will cheerfully answer today's question from a withdrawn document.
  • Attach the access permissions at ingestionEvery chunk carries the entitlements of its source document. This is what makes step 4 possible, and adding it later means reindexing everything.
  • Keep a content hash per chunkSo you can prove which exact version of a passage produced an answer when someone asks eight months later.

Step 4: Retrieval. Hybrid search, and permissions inside the query#

Dense vector search is good at meaning and bad at identifiers. Ask it for the KYC obligations that apply to a crypto exchange and it does well. Ask it for 31 CFR 1010.100 and it may return passages about similar sounding provisions, because the embedding of a citation string carries very little of what makes that citation specific.

So run both. Dense search for the conceptual half, BM25 or equivalent keyword search for exact identifiers, section numbers, instrument names and defined terms, then rerank the union with a cross-encoder so the final ordering is decided by a model that has seen the query and the passage together. Reciprocal rank fusion is a reasonable way to merge the two lists before reranking, and it needs no tuning to be better than either list alone.

Step 5: Generation, with citation as a hard requirement#

The prompt layer has one job beyond producing readable prose, which is to make unsupported answers structurally impossible rather than merely discouraged. Ask for structured output where every claim carries the identifier of the chunk it came from, then validate it: if a cited chunk is not in what you retrieved, the response is a failure and gets retried or escalated, not displayed.

Two more constraints specific to this industry. The model must be able to say it does not know, and the retrieval result must be allowed to be empty, because a confident answer assembled from the three least irrelevant passages is the worst possible output. And the boundary around regulated advice has to be enforced in code, not requested in a prompt. If answers reach customers, FINRA Rule 2210 treats them as communications with the public, with the review and recordkeeping obligations that follow, which is a strong argument for keeping the first version internal.

Step 6: Measure the two halves separately#

This is the step that gets skipped and the reason teams end up guessing. A single end-to-end answer quality score tells you the system was wrong but not which half was wrong, and the two have completely different fixes.

MeasureWhat it tells youWhat a bad score means
Recall at kOf your evaluation questions, how often the correct passage appeared in the top k retrieved.An ingestion or retrieval problem. Chunking, metadata, or the dense and keyword balance. Nothing in the prompt will fix it.
Citation faithfulnessWhether each claim in the answer is actually supported by the passage it cites.A generation problem. Prompt, output schema, or model choice.
Abstention rateHow often the system declines when the corpus genuinely does not contain the answer.Near zero is a red flag, not a success. It means the system is answering regardless.
Answer qualityHuman rating against the evaluation set, sampled regularly.Meaningful only once the three above are healthy.

Alongside these, log everything, because the audit requirement and the debugging requirement turn out to be the same artifact. Every query, the user, the retrieved chunk identifiers and hashes, the model and version, the generated answer, and whether a human accepted it. That record is what lets you reproduce an answer from eight months ago, and it is also the only way to find out why retrieval quality drifted after someone reindexed.

What this costs and how long it takes#

A single-source internal system, in production with logging and citation, is a few weeks of work and sits at the lower end of our RAG range, around $5,000. What pushes an engagement to the $15,000 to $20,000 end is rarely the retrieval. It is the number of sources to connect, the state of the documents, and whether permissions and effective dates have to be reconstructed. Document cleanup is the line item that surprises people, so we scope it explicitly rather than finding it halfway through.

Building RAG where the answers have to hold up?

Book a free 30 minute call. If you bring twenty of the real questions your team gets asked, we can usually tell you within the call whether retrieval is the right approach and what would have to be true about your documents first.

Book a Strategy Call →