Key Takeaways
Generative AI for data analytics lets a non-technical team query a data warehouse in plain English by turning their questions into SQL, but in production the accuracy comes from the semantic layer around the model, not the model itself.
- On Snowflake, the practical path is Cortex Analyst reading a semantic model, so every question runs as governed SQL inside your own warehouse.
- Raw-schema text-to-SQL is unreliable on real enterprise data, dropping to around 21% accuracy, while a well-built semantic layer lifts accuracy by roughly 20%.
- Done right, it cuts a data team’s backlog of one-off requests by about half and turns answers that once took days into ones that take under a minute.
- For regulated data like insurance, governance is the point: read-only scoped roles, masked PII, and role-based access control applied to every AI-generated query.
- It stays probabilistic, so complex multi-join questions land near 75% accuracy and the answers that matter need a human in the loop.
Introduction
A claims operations lead at a US property and casualty insurer we work with used to start every Monday the same way, by filing a ticket. She needed to know how many auto claims opened in the last 30 days were still unresolved, split by state and adjuster. A reasonable question with an answer that lived in Snowflake. The catch was that only a two-person data team could pull it, and that team was already buried under a reporting backlog. By the time the numbers came back, the week had moved on.
That backlog is what the client brought to Bacancy. They asked us to build Generative AI for Data Analytics on top of the Snowflake warehouse they already ran, so operations staff could ask questions in plain English and trust the answers, without routing every request through data engineers. Here is how we did it, what broke along the way, and why the hard part had almost nothing to do with the language model.
Why Generative AI for Data Analytics Looked Simple and Wasn't
On the surface this is a solved problem. Wire a modern model to a warehouse, type a question, get SQL back. The demo always works. Production is where it falls apart.
The numbers make the gap obvious. Text-to-SQL systems score around 90% on the academic Spider benchmark but collapse to roughly 21% on Spider 2.0, which uses enterprise-scale schemas and real SQL dialects like Snowflake and BigQuery. The model did not get worse. Real warehouses just do not look like benchmarks:
- Benchmarks are clean, warehouses are not: academic tests use small schemas with obvious column names. An insurance warehouse has hundreds of tables and cryptic fields inherited from a policy admin system.
- The rules live in people’s heads: business logic like what counts as an “open claim” is rarely written into the schema, so the model has to guess.
- The dangerous failures are silent: a model can write perfect SQL against a schema it does not understand and return a number that is confidently wrong. Nobody notices, because nothing errors out.
Text-to-SQL is a textbook example of this. The model works fine, and everything around it is where the project actually lives or dies.
The Insurance Data Problem of Schema Complexity and Compliance Guardrails
Insurance data is its own special problem. Our client’s warehouse pulled from three systems, and each carried its own idea of what the business meant:
- Policy administration: its own definition of a “policy,” premium, and effective dates.
- Claims: reserves, status, and adjuster assignments that did not line up cleanly with policy records.
- Billing: written versus earned premium logic that differed again.
Terms that sound simple to a business user, earned premium, loss ratio, open reserve, each mapped to different columns and different aggregation logic depending on which table you started from. Ask two analysts for “loss ratio this quarter” and you could get two defensible but different numbers.
On top of that, the data was full of personally identifiable information and claimant records, which put it inside regulatory scope. The NAIC Model Bulletin on the use of AI by insurance companies, adopted in December 2023, sets clear governance expectations for how insurers deploy AI, and regulators are now piloting an evaluation tool across a dozen states. Handing a language model unrestricted read access to raw claimant tables was never an option.
Ready to make your Snowflake data answer plain-English questions?
Our Snowflake Consulting Services team builds the semantic layer and governance that make it accurate and audit-ready in production.
Building the Semantic Layer Behind Natural Language to SQL
The fix that carried the whole project was the semantic layer, not the model. On Snowflake we used Cortex Analyst, which drives natural language to SQL off a semantic model rather than the raw schema. Snowflake is direct about why: a bare schema lacks the business definitions and metric rules the model needs, which is exactly what a semantic model supplies.
In practice, that meant doing the slow, unglamorous work:
- Curated views, not raw tables: we exposed under a dozen entities the ops team actually asked about, instead of the full warehouse.
- One definition per metric: “earned premium” and “open claim” each got a single agreed definition with the correct aggregation, encoded once.
- Synonyms mapped to logic: “unresolved,” “open,” and “pending” all resolved to the same filter.
- Join paths written out: the relationships between policy, claims, and billing were predefined so the model never guessed.
- Verified example questions: a seed set of real questions paired with their correct SQL to anchor accuracy.
The client’s warehouse had recently been consolidated onto Snowflake, so the physical tables were clean. The business layer on top of them was what was missing, and that layer is where the accuracy came from. Snowflake’s own engineering team found a well-built semantic model lifts text-to-SQL accuracy by roughly 20% over a model working alone. A modest model with a rich semantic layer beat a stronger model flying blind, every time.
Read-Only Access, Query Cost Control, and Verified Queries on Snowflake
Governance was the next build, not an afterthought. Because Cortex Analyst runs the generated SQL inside your own Snowflake virtual warehouse and honors role-based access control, the AI-generated queries obeyed the same permissions a human would hit. The guardrails we set:
- Read-only, scoped role: the ops team could reach the curated views and nothing else, with PII columns masked or excluded.
- Data stayed in the boundary: by default the service runs on Snowflake-hosted models, so no prompt or data left Snowflake’s governance boundary. That single fact let the compliance team sign off.
- Cost controls: a small warehouse with a statement timeout, resource monitors, and row caps, so an accidental full-table scan could not run away with the bill.
- A verified-query loop: frequently asked questions and their SQL got reviewed by an analyst, then reused as reference examples. Consequential questions stayed human-reviewed.
That last point matters most. The failure mode here is semantic, not syntactic, so you cannot unit-test your way out of a query that runs perfectly and returns the wrong answer. The model handled the volume. People kept judgment over the answers that mattered.
Keeping the Ops Team in Flow With Follow-Up Questions
The first question is almost never the real one. Someone asks “how many open auto claims this month,” sees the number, and immediately wants “now just California,” then “compare that to last quarter.” If each of those means starting over, people give up and go back to filing tickets.
So we built a multi-turn conversation. The system carried context between questions, so “just the ones over 60 days” applied to the previous result instead of resetting. That behavior did more for adoption than any accuracy gain. It turned the tool from a search box into something closer to how the ops team already thought about their data, one question leading to the next.
What Changed for the Ops Team After Plain-English Querying
The directional outcomes lined up with what other teams report. GitLab took a similar Snowflake conversational setup from a 60% proof of concept to 85 to 95% accuracy on simple questions, cutting analytics requests by about half and taking time-to-insight from weeks to seconds. Our client’s experience rhymed with that:
- The data team’s pile of small requests dropped by about half in the first two months.
- Answers in under a minute for questions that used to take two or three days.
- Two engineers redeployed onto the modeling and pipeline work stuck behind the backlog for months.
That mirrors the broader picture, with EY finding carriers expect average cost savings above 20% from AI-driven productivity over two years. We were honest about the ceiling, though. Complex, multi-join questions still landed near 75% accuracy, not 100%, which is why those stayed on the human-reviewed path. Generative AI in data analytics is a probabilistic tool, and pretending otherwise is how you end up trusting a wrong number.
Demo Build vs Production Build: What Actually Differs
The gap between a version that impresses in a demo and one an insurer can actually run comes down to a few things:
| Area | In a quick demo
| What production actually needs
|
|---|
| What the model sees
| The raw warehouse, all its tables
| A dozen curated views, nothing more
|
| Business terms
| The model guesses what "open claim" means
| One agreed definition per metric, set once
|
| Data access
| Broad read access to everything
| Read-only role, personal data masked, access rules on every query
|
| Cost control
| Any question can trigger a huge scan
| Small warehouse with timeouts and row limits
|
| Follow-up questions
| Each new question starts from scratch
| Context carries over, so questions build on each other
|
| Wrong answers
| Slip through silently
| Caught by a verified-query check and human review
|
What We'd Tell Any Insurer Evaluating Generative AI for Data Analytics
If you are planning something similar, four things matter most:
- Spend your time on the semantic layer, not on picking a model. Getting your metric definitions and views right is what drives accuracy. The model you choose barely moves the needle by comparison.
- Give the tool a few clean views, not the whole warehouse. A small, well-defined set of data is more accurate to query and far easier for compliance to approve.
- Have a person check the answers that matter. Let the tool handle everyday questions, but keep human review on anything high-stakes before someone acts on it.
- Treat this as a data governance project, not just an AI one. The model was never the real risk. Open access and wrong answers that slip through unnoticed are.
Closing Thought
Generative AI for Data Analytics lives or dies on the unglamorous work behind it. Picking a language model is the easy part. The real work is everything around it, the metric definitions, the read-only access rules, the cost limits, and the checks that catch wrong answers, and that is what decides whether an insurer’s ops team trusts what comes back. It is also where Bacancy Technology spends its time. If your team is sitting on a warehouse full of answers only a few people can reach, our Data Analytics Services team can help you open that up the same way, with the right controls built in from day one.
Frequently Asked Questions (FAQs)
Generative AI for data analytics turns plain-English questions into SQL so non-technical teams can query a data warehouse directly, without waiting on analysts. In production, the accuracy depends on a semantic layer that encodes business definitions, not on the language model alone.
Yes. On Snowflake, Cortex Analyst reads a semantic model and converts natural-language questions into SQL that runs inside your own virtual warehouse. Because it honors role-based access control, the generated queries respect the same permissions a human user would.
Text-to-SQL models score around 90% on clean academic benchmarks but drop to roughly 21% on enterprise-scale schemas. Real warehouses have hundreds of tables, cryptic column names, and business rules that live outside the schema, so a model working from the raw schema alone guesses wrong.
It can be, if governance is built in rather than added later. That means read-only scoped roles, masked PII, and access controls applied to every AI-generated query, with high-stakes questions kept on a human-reviewed path. For insurers, the NAIC Model Bulletin sets clear expectations for how AI is governed.
Accuracy varies by question complexity. Well-built systems reach 85 to 95% on simple questions but land near 75% on complex multi-join queries, which is why a verified-query loop and human review on consequential answers matter. It is a probabilistic tool, not a deterministic one.