Bacancy’s Insights from Building RAG Applications with Vertex AI for a UK-Based Client

Quick Summary

Introduction

Most enterprises don’t struggle with a lack of knowledge. They struggle with retrieving the right knowledge at the right time. The information a doctor, analyst, or support representative needs often already exists, scattered across thousands of PDFs, SOPs, and internal knowledge bases. The major hurdle is spotting it quickly. That’s why the retrieval-augmented generation (RAG) market is projected to grow from $1.94 billion in 2025 to $9.86 billion by 2030, representing a CAGR of 38.4%, with healthcare and life sciences emerging as some of the fastest-growing adopters.

In this guide, we walk through building RAG applications with Vertex AI using the same approach our team followed while developing a Clinical Knowledge Assistant for a healthcare organization already operating on Google Cloud.

Insights Into the Client’s Clinical Knowledge Access Challenges

Our client was a hospital group with one of the most common problems. They had built a vast repository of clinical knowledge over the years, thousands of documents covering treatment guidelines, hospital SOPs, nursing protocols, drug formularies, infection-control policies, and emergency procedures. The challenge was not the accuracy of this information; it was accessing it quickly when every second mattered.

A doctor treating a deteriorating patient doesn’t have ten minutes to scroll a 250-page PDF. They need one answer, now: “What is our protocol for managing sepsis in pediatric patients?” Multiply that across every shift, ward, and clinician, and the hospital was quietly losing hours to document hunting, time that should have gone to patients.

Existing solutions had major limitations. Keyword search matches words, not meaning, so a search for “sepsis treatment” can miss the right file if it’s saved under “septic shock management.” A regular AI chatbot sounds confident but answers from whatever it absorbed in training, which may be outdated or simply wrong, and in a hospital, a wrong answer is a safety risk. RAG is the fix that works, and we implemented it.

How a RAG System Actually Works, in 60 Seconds

A RAG system runs on three clocks.

1. Before anyone uses it (setup): This work happens up front, and again whenever documents change. The system reads every hospital document, breaks it into pieces, and files those pieces into a search index organized by meaning. Nobody is asking questions yet; we’re just getting the knowledge ready to search. This is the offline pipeline.

2. While someone is using it (live): The moment a doctor types a question, the system finds the most relevant field pieces and asks the AI to answer using only those. This happens in seconds per question. This is the runtime pipeline.

3. After it’s live (upkeep): Once the assistant is in daily use, the work doesn’t stop. New and updated documents get re-indexed, answer quality gets checked, and costs get watched. This is the operations pipeline.

Setup builds the knowledge, runtime answers questions from it, and upkeep keeps it operating. Everything about building RAG applications with Vertex AI hangs off those three.

We built all three on Vertex AI, Google’s managed AI platform, which supplies most of the parts in one place instead of forcing you to wire together separate vendors. Here’s the stack at a glance:

Building RAG Applications With Vertex AI: Bacancy’s 7-Step Process

With the map in hand, here’s how we approached the build in practice. Each step feeds the next, so the order matters as much as the parts.

Step 1: Define the Use Case and Guardrails

First, we pinned down exactly what we were building and for whom.

• Users: Doctors and nurses.
• Job: Answer clinical questions from approved hospital documents in plain language.

Then the questions that actually shape the architecture:

• Which documents count as trusted sources?
• Who is allowed to see what?
• How accurate must an answer be before it’s safe to show?
• What’s explicitly out of scope?

We documented these decisions upfront to establish clear boundaries for the system. We also established success criteria. Every answer had to include a citation, return within a few seconds, and respond with “I don’t know” whenever the required information wasn’t available instead of making assumptions. These decisions later shaped the permission filters used throughout the retrieval pipeline.

Step 2: Collect and Prepare the Knowledge

The documents were stored across several systems, from Google Cloud Storage and SharePoint to a few shared drives. The first task was to consolidate them, so we copied every file into a single Cloud Storage bucket that served as the staging area.

A script then handled the files one by one. For each file, how it got the text depended on the file type. A PDF or Word file already contains real text, so the script reads it directly. A scanned document is different: it’s an image of a page, with no readable text inside, so the script first ran it through OCR (Google Document AI) to convert that image into text.

Once the text was extracted, the script cleaned it, removing page numbers, headers, and footers, logos, and keeping only the actual content. It wrote the clean text to a new file and left the original unchanged.

Step 3: Chunk and Enrich With Metadata

Now the make-or-break step. You never store a 250-page guideline as one record; you split it into focused chunks, usually a few paragraphs each, so a search can return one precise section. A single sepsis guideline becomes dozens of chunks, roughly one per procedure. Make chunks too big and the answer drowns in noise; too small and it loses the context that makes it make sense.

Then we tagged every chunk with metadata, which included document name, department, version, date, access level, and page number. That metadata is what later powers permission filtering, version control, and exact-page citations. More than any model choice, this step is where building RAG applications with Vertex AI was won or lost.

Step 4: Generate Embeddings and Build the Vector Index

Now the text has to become something a computer can search by meaning. For every chunk from Step 3, we sent the text to Vertex AI’s embedding model, which hands back an embedding: a list of a few hundred numbers, like coordinates, that mark where that chunk’s meaning sits. Two chunks that mean the same thing get coordinates close together, even if they use completely different words, so “how do we treat sepsis” lands near “managing a blood infection in patients.”

We then put all of it into Vertex AI Vector Search, one record per chunk holding the numbers, the original text, and the tags from Step 3. At this stage, we’re only placing each chunk on the map, not sorting or ranking anything. That comes later. Vector Search simply lays the coordinates out so that when a question arrives in Step 5, it can find the nearest ones in milliseconds instead of reading every document top to bottom.

That collection is the searchable knowledge base, and every step after this depends on it.

Step 5: Build the Retrieval Layer

This is the “R” in RAG, and it decides the answer quality more than the model does. When a doctor asks a question, the retrieval layer:

• Converts the query into an embedding using the same gemini-embedding-001 model that was used for the documents, ensuring both exist in the same vector space.
• Searches Vertex AI Vector Search for the closest matching chunks, beginning with the top ten candidates.
• Applies metadata filters so that only the current document version and content the signed-in user is authorized to access remain in the results.
• Uses hybrid search, combining semantic and keyword search, for queries containing drug names, codes, or other terms where exact matches are just as important as meaning.
• Reranks the remaining results using the Vertex AI Ranking API before passing only the top three chunks to the language model.

Get this layer right, and the model has exactly what it needs. Get it wrong, and no prompt tuning will save the answer, because the model can only work with what retrieval hands it. This is the layer where building RAG applications with Vertex AI lives or dies on accuracy.

Step 6: Generate Grounded Answers With Gemini

Only now does the language model enter. We build a prompt that combines a clear instruction, the retrieved passages, and the user’s question, something like: “Answer only from the context below. If it isn’t there, say you don’t know. Cite the source.” That prompt goes to Gemini through Vertex AI, which writes an answer from the supplied context rather than its own training data. The app then attaches citations.

What the doctor sees: the sepsis protocol in a sentence or two, followed by “Source: Pediatric Sepsis Management Guideline, p. 12.” If the documents don’t cover the question, the assistant says so instead of guessing.

Gemini never searches the documents itself; all the findings happened in Step 5. That separation is what lets the assistant show its sources, and in healthcare, traceability isn’t a nice-to-have. It’s the point.

Step 7: Build the Application, Then Operate It

We wrapped the pipeline in an application clinicians could actually use. We built a backend that receives the user’s question, runs retrieval, sends the retrieved context to Gemini, and returns the generated answer with citations. On top of that, we added a chat interface with conversation history, a source viewer, Google IAM authentication, and role-based access so a nurse and a specialist only see the documents they’re permitted to access. Then we connected it to the systems clinicians already use.

After that, the work shifts to keeping it operating. Before launch, we ran the system against a set of crucial clinical questions and measured retrieval accuracy, groundedness, hallucination rate, latency, and citation quality with Vertex AI Evaluation, then tuned chunk sizes, retrieval settings, and prompts based on the results. After launch, new and revised documents are re-indexed automatically, and Vertex AI Monitoring tracks speed, failures, token use, and cost. A RAG system is a living service; left unmaintained, it slowly starts quoting last year’s guidance.

The Most Common Mistakes That Hold Back RAG Applications

Most RAG projects don’t fail immediately. They launch successfully, perform well in demos, and then gradually lose accuracy as they move into production. The usual culprits, and the ones we designed around from day one:

• Embedding whole documents. Skip chunking, and every search returns a 200-page haystack. This is the number-one cause of weak answers.
• Ignoring metadata. Without tags, you can’t filter by permission or version, so users see documents they shouldn’t or outdated ones.
• Over-retrieving. Stuffing fifty chunks into a prompt buries the answer and inflates cost. Precise retrieval beats bulk every time.
• Fixing bad retrieval with better prompts. If the right passage was never retrieved, no prompt can conjure it. Fix retrieval first.

Never re-indexing. Knowledge changes; an index frozen at launch guarantees slow decay.
Treating every user the same. In regulated fields, document-level access control isn’t optional.
Designing around these from the start is the gap between a weekend prototype and a system clinicians trust. It’s also why building RAG applications with Vertex AI is an engineering discipline, not a plug-in you switch on.

Getting RAG Right on Vertex AI