RAG on Databricks: 5 Problems We Faced in Production and How We Solved Them

Quick Summary

Introduction

Building a basic RAG system on Databricks is not a difficult task. You just need to load your source documents into a Delta table in Unity Catalog, create a Mosaic AI Vector Search endpoint, and build an index on top of that table. Generation runs through a foundation model over the retrieved context, and Databricks keeps the index in sync as your data changes.

This basic setup works just fine until the traffic and usage start growing. The sample data you earlier worked on starts getting larger, and the queries start coming from real users, whether internal staff or your own customers, each with different roles and permissions, rather than the small test group you planned for.

We’ve worked on multiple projects for RAG on Databricks, and here are the five problems we faced while building for clients in insurance, healthcare, pharmacy, banking, and fintech.

The Top 5 Problems We Faced Building RAG on Databricks

These five RAG on Databricks problems span multiple clients and industries, including insurance, healthcare, pharmacy, banking, and fintech. They reflect what we have learned delivering RAG development services on Databricks across regulated sectors.

Problem 1: Slow Response Rate During High Traffic

This is one of the most common problems teams face while building RAG on Databricks, and the impact of this problem is also high, as no customer would like to wait, and they may easily switch to a faster alternative.

The usual reason for this is not the language model, but the retrieval pipeline around it. When every query retrieves too many chunks, reranks all of them, and sends a large context to the model, each of those steps adds latency. Without caching, even the most common questions are processed from scratch on every request.

We saw this with a fintech client on a customer support assistant built on Databricks to answer common account and payment questions.

During pilot testing, this RAG based assistant worked completely fine. About 80 internal testers used it, and median latency sat near 1.5 seconds, so it easily passed the pilot stage. At launch, thousands of users hit it at once, and the latency climbed to 9-12 seconds, and customers began leaving before they received an answer.

The Solution: Reduce the Work Done on Each Query

We reduced how many chunks each query retrieved and reranked, which also cut the size of the context sent to the model. We cached the answers to frequently asked questions so they were not recomputed on every request, and moved generation to provisioned-throughput Model Serving, which gave it reserved capacity instead of queuing under load. Together, these brought the response time back to around two seconds under full traffic.

Problem 2: Sensitive Data Being Leaked into Answers

This is one of the most serious problems teams face while building RAG on Databricks, and the impact of this problem is very high, as a single leak of sensitive data can become a compliance violation rather than just a poor experience.

The reason this happens is that sensitive information gets embedded into the index along with everything else, and the retriever only checks for relevance. If a chunk holds sensitive data and matches the query, it ends up in the answer, and without classification and masking during ingestion, there is no way to keep it out.

We ran into this twice while delivering Databricks consulting for healthcare clients. In one case, a hospital’s clinical assistant gave a user in one department an answer containing another patient’s PHI. In another case, a health insurer’s support assistant returned claim notes for the wrong member. In both cases, the data was relevant to the question, but the receiving user had no right to see it, which under HIPAA is a reportable exposure.

The Solution: Mask Sensitive Data for RAG with Unity Catalog on Databricks

The fix was the same for both. We used Unity Catalog to classify and mask PHI before embedding, so it never entered the index, and we kept each department’s and member’s data in separate indexes. After this implementation, both these clients’ RAG assistants passed their HIPAA reviews.

Problem 3: RAG Assistant Generating Inconsistent, Hallucinated Answers

This is one of the hardest problems to catch while building RAG on Databricks, because the system looks like it is working, but in reality, it is just generating a different answer every time.

Such inconsistency can cause a big impact for teams working in regulated industries, as these industries require every answer to be accurate, consistent, and traceable to its source. An answer that changes on each attempt cannot be verified or defended in an audit, which turns a technical flaw into a compliance risk.

We were building a RAG based assistant for a banking client who has to answer staff questions about internal policies and regulatory procedures, such as account-handling steps, KYC requirements, and reporting timelines. This project ran completely fine in the pilot testing phase, but when it was introduced to all of the employees of the banking client, inconsistencies started appearing. The same policy question, asked by different staff or at different times, often returned different answers, and some responses described procedures that were not part of the current policy documents.

The Solution: Restrict the RAG Model on Databricks to Its Source

We set up Mosaic AI Agent Evaluation to check answers against the source material and flag anything that was wrong or unsupported. We then restricted the assistant to answer only from the retrieved documents, cite the policy behind each answer, and run at a lower temperature (meaning randomness level for the language model) so the same question gave the same result. These changes made the answers consistent and traceable back to their source.

Problem 4: RAG Assistant Returning Stale, Outdated Answers

This is one of the easiest problems to miss while building RAG on Databricks, because the system keeps working normally; it just keeps answering from outdated information. When a source document changes but the index is not updated, the assistant continues to answer from the older version.

In regulated industries, policies, rates, and rules change often, and staff or customers may act on an answer that was correct last month but is wrong today. These answers are easy to trust and hard to catch, because they are wrong only in the sense that the data behind them is now irrelevant.

We experienced this in three recent engagements. For an insurance client, the assistant kept quoting coverage limits and rates that had already been revised. For a fintech client, it kept returning transaction fees that had already changed. For a healthcare client, it cited a treatment guideline that had since been replaced. In each case, the source documents had been updated, but the index still held the older version.

The Solution: Use Delta Sync to Keep Index Information Updated

The fix was to keep the index up to date with the source. The documents were already stored in Delta tables, so we switched to a Delta Sync index and enabled Change Data Feed, which updates the index automatically whenever a document changes. We used continuous sync for content that changes frequently and triggered sync for the rest. After this, the assistant always answered from the current version of each document.

Problem 5: RAG Assistant Returning Wrong Answers Due to Poor Data Quality

This is one of the most underestimated problems teams face while building RAG on Databricks, because the retrieval and the model can both be working correctly while the answers are still wrong. When the source data is inaccurate, the assistant simply retrieves and repeats those inaccuracies.

The source data sets are mostly found to be messed up. They often contain scanned files with recognition errors, duplicate versions of the same document, and tables that lose their structure when parsed into plain text. In areas where an exact detail matters, such as a dosage or a code, a small error in the source can become a confident and incorrect answer.

We were building a RAG based assistant on Databricks for a US based pharmacy client to help staff answer questions about medicines, including dosages, usage, and drug interactions. In testing, a question about a product’s dosage sometimes returned the wrong strength, and two medicines with similar names could be merged into a single answer. The retrieval was fine; the problem was the source data, which was full of scan errors, duplicate files, and badly parsed tables.

The Solution: Structure and Validate Data During Ingestion

We addressed this at the ingestion stage, before any data reached the index. We used this Databricks native function, ai_parse_document, to pull tables and key fields, such as dosage and product codes, into a structured format instead of leaving them as flattened text. We added validation checks in the Delta pipeline to catch bad records early, and removed the duplicate files. After this, the assistant answered drug-related questions accurately, based on clean and structured data.

Also read: The five problems we faced while building RAG on Bedrock for our clients, and how we solved these problems.

Conclusion

Frequently Asked Questions (FAQs)

How much does it cost to run RAG on Databricks?

It depends on index size and traffic, but the endpoint type is your biggest lever. Databricks’ storage-optimized Vector Search endpoints reportedly run up to 7x cheaper than the standard tier on large workloads, which is what makes billion-scale indexes affordable. After that, retrieving fewer candidates, caching frequent queries, and right-sizing the generation model handle most of the remaining spend.

How do you stop a RAG based system built on Databricks from hallucinating?

Measure it first, then constrain it. Mosaic AI Agent Evaluation scores groundedness and correctness so you catch regressions before release. Pair that with a prompt that answers only from retrieved context and refuses otherwise, forced citations to the source chunk, a low temperature, and reranking to improve the context you’re grounding on.

How do you enforce data access controls in RAG on Databricks?

Bind retrieval to Unity Catalog. Row filters and ABAC policies let the vector index respect the same permissions as the underlying tables, so a user only retrieves what they’re entitled to see. Tag chunks with entitlement metadata at ingestion and add a per-user filter at query time.

How should I chunk documents for RAG on Databricks?

Start with a test, not a fixed number. Databricks suggests trying small (256 tokens), medium (512), and large (1024) chunks rather than guessing, and notes that parsing quality and semantic structure matter more than the exact size. For long, clause-heavy documents like policies and monographs, parent-child retrieval (embed small chunks for precision, return the larger parent for context) usually beats fixed-size splitting, and tables should be extracted as structured fields instead of flattened into text.