How We Established Databricks Data Governance in Existing Environments For Our Clients

Quick Summary

Introduction

Setting up Databricks data governance is easier when you are just starting with the platform. You just need to define the catalog structure, write the access model, and turn on enforcement. So, that’s usually not a problem.

The actual problem we see is when clients come to us with a Databricks environment that’s been running for years, with real users and data, but with no data governance in place. Because no one planned for it at the start, and now, as the teams are scaling, the governance gap is turning into a critical bottleneck.

In such cases, trying to establish data governance can actually cause more problems than it solves, and so it needs to be done right. pipeline that has run untouched for years can break. A team can end up blocked in the middle of a project. Someone from the team can lose access they have used for a long time. And fixing these all can make things even more messy.

So, how do you set up data governance in Databricks without breaking what already works, and without losing the team’s trust along the way?

Read this article, where we cover the seven data governance problems we faced with our Databricks clients, how we solved them, and the best practices we learned from the experience, all of which can help get your Databricks data governance position in your existing Databricks environment set right.

6 Common Databricks Data Governance Challenges We Faced and How We Solved Them

These are the six common data governance challenges that we have seen our clients face in their existing Databricks environments, across different industries, including BFSI, healthcare, and retail. For each of these challenges, we have also mentioned the solutions implemented by our data governance team that have worked well and that our audience can also implement.

Problem 1: Legacy Tables & Views Going Undocumented and Unowned

This is usually the first problem, and the most common problem we faced with Databricks data governance for existing environments. You cannot govern what you cannot account for. Most environments that have been running for a few years cannot account for everything in them.

On an insurance client’s claims platform on Databricks, the legacy Hive metastore held roughly 9,000 tables. About half of them had no owner tag, no description, and names like tmp_claims_v3_final or backup_do_not_use. Nobody on the team knew which ones were still needed and which weren’t. That made it way more difficult to write an access policy for the stakeholders.

Restrict the wrong table, and you break something in production. Leave a live one out of the policy entirely, and it stays ungoverned, with risk of exposure to sensitive data.

The Solution: Inventory Before Writing a Single Policy

In the Databricks environment, an access policy is the ground rule for who can use or see what, so it needs to be right. But we were facing difficulty writing one, because nobody from the insurance client’s team knew what tables existed or who they belonged to. So, we queried system.information_schema and usage logs, and scored every table by when it was last used and what else relied on it.

That gave us two groups: tables still in active use, and everything else nobody had touched in the last 12 months. Until this was done, we didn’t start on access or classification work.

Out of the original 9,000, only 27% turned out to still be active. That’s the group we built the access policy around, instead of writing rules for thousands of tables nobody was using.

Problem 2: Sensitive Data Being Exposed in Unmasked Production Tables

We ran into this twice within the last 12 months. Once on a hospital’s clinical data analytics setup, and once with a health insurer’s claims data workspace on Databricks. Once on a health insurer’s claims workspace. In both cases, several teams queried the same tables every day, and the patient identifiers inside them were never masked.

On the hospital engagement, a user in one department pulled up another patient’s PHI by mistake. This happened during UAT, not production, which is the only reason it didn’t become a reportable HIPAA event.

The Solution: Classify and Mask Data at the Column Level

Masking just one table cannot help solve this. So, for both clients, we applied Unity Catalog classification tags to the columns with sensitive data, then bound masking policies to the tag instead of the table. Because table level masking has to be reapplied every time someone creates a new table. Whereas tag-level masking follows the classification automatically.

Every table carrying the tag got masked automatically going forward, including tables created after the initial pass. Both clients passed their next HIPAA review with no PHI exposure across the test set.

Problem 3: Workspace-Admin Access Becoming the Default Grant

Granting broad access is often the fastest way to unblock someone. Add them as a Databricks workspace admin, and the request disappears right away. Nobody comes back to review or update the access later.

We’ve seen this happen with a number of our banking clients. These clients handle sensitive financial data for large numbers of customers, with far too many staff for anyone to review access one by one. Here, any extra access control granted to any individual by mistake can cause troubles too big to solve in this lifetime.

The Solution: Grant Roles Based on Functions, and Not Individually

We could have gone through every admin grant one by one and removed the ones that looked wrong, but that would not have changed how access got granted going forward.

So, we moved users onto Databricks account-level groups, synced from SSO and SCIM, instead of granting workspace-admin rights to individuals. We then cut workspace-admin rights down to a small list of people who actually need them, reviewed on a set schedule.

Problem 4: Production Jobs Breaking During the Catalog Migration

Setting up Unity Catalog on an existing Databricks environment means migrating out of the legacy Hive metastore. That migration carries a risk. Existing jobs and dashboards are already built around the old table paths.

So, if you cut over without considering what still depends on those old paths, a job can fail without warning. We have seen multiple clients of ours make this mistake when migrating to Unity Catalog for the first time.

The Solution: Migrate in Stages, Not in a Single Cutover

A single cutover leaves no room to catch a broken job before it causes damage. So, we run the Unity Catalog upgrade one table at a time. We keep both paths resolvable during validation, and only cut a job over once it has completed a full production cycle clean against the new catalog.

Across every engagement that followed this sequence, zero jobs failed silently after cutover. No dashboard needed an emergency rewrite mid-migration.

Problem 5: Lineage and Audit History Going Untracked

This is one of the most common gaps we see across almost every Databricks client. A regulator or an internal auditor asks who accessed a given table and what fed into it. In an ungoverned environment, nobody can actually answer that. Getting the right answer here may require many hours and working with multiple stakeholders just to collect the logs from memory.

This is common because many Databricks users have still not migrated from Databricks Hive Metastore to Unity Catalog. And lineage tracking only works for tables that have actually been migrated into Unity Catalog.

The Solution: Turn On Unity Catalog Lineage and Retain the Audit Logs

For clients still on the Hive Metastore, we migrate their tables into Unity Catalog first, one table at a time, keeping old and new paths live until each job has run clean on the new catalog. Lineage only works once tables are actually inside Unity Catalog. Then we turn on the automatic lineage graph and keep the audit logs in a queryable history.

Problem 6: Access Enforcement Locking Out Users Ahead of Schedule

We recently saw this on a retail client’s Databricks environment, while rolling out a new group-based access model. We revoked the old access and applied the new groups in the same change. But not every analyst had been added to their new group yet. So, a group of analysts lost query access in the middle of the week, with no warning.

This resulted in a failure to access reporting dashboards these analysts accessed at the start of their day. The access itself was right. The problem was timing. The new groups matched exactly who should have had access. We just turned them on before every analyst was added to one. Something that had worked fine for years had suddenly broken, with no warning.

That’s the kind of incident that makes a team stop trusting the next governance change too, no matter how carefully it’s actually planned.

The Solution: Reconcile Old Access Before You Revoke It

The fix here is not rolling these changes slowly but actually confirming if the newly assigned access groups cover everyone the old access groups did, with the required access rights, before these old access rights get revoked. Every user with access under the old model has to show up in an equivalent new group and be checked by name, not assumed by role. Only once that list matches exactly does the old access come out.

In the retail client’s environment, we added that reconciliation check before the next cutover. It helped solve the lockout before anyone lost access again.

Top 6 Best Practices for Databricks Data Governance Recommended by Bacancy Technology

Here are six best practices we recommend to Databricks clients for keeping data governance secure and manageable as their environment grows, whether they’re dealing with the problems above or not.

1. Structure Catalogs Around Business Domains

Structure the top level of Unity Catalog by business domain, such as claims, finance, or underwriting, instead of by environment like dev, staging, and prod. A domain-based catalog keeps a team’s tables together in one place. Access becomes easier to manage too, because the boundary follows how the business works, not how the infrastructure is organized.

2. Run Production Jobs on Service Principals

Scheduled pipelines and production jobs should run under a service principal, not a person’s own credentials. If that person changes teams or leaves the company, the job keeps running without interruption, instead of failing or carrying access nobody intended to leave active. We set this up for every client during onboarding, because a job running on someone’s personal credentials is one of the first things to fail when that person leaves or changes roles.

3. Deploy One Metastore Per Cloud Region

Deploy a single Unity Catalog metastore per region instead of spreading catalogs across regions under one shared metastore. This keeps data closer to where it gets processed, and it matters most for clients under strict data residency rules, especially with healthcare and BFSI Databricks clients, where we see this come up most of the time.

4. Tag and Describe Data for Discovery

Tagging is not only for protecting sensitive columns in Databricks, it is also how people find data in the first place. Add real descriptions and tags to tables and columns, so Catalog Explorer actually helps someone locate the right table instead of guessing from a name. Well-governed tables still go unused when nobody can find them, no matter the industry.

5. Share Data Through Delta Sharing

Share data live through Delta Sharing instead of exporting a copy to another team or an outside partner. A copy goes stale the moment the source changes, and it needs its own permissions managed separately from the original. Using Delta Sharing avoids both problems. The other side queries the same data directly, without a copy ever being made, and without needing to be on Databricks themselves.

6. Centralize Audit Logging in System Tables

Log every platform event, data access, and permission change in the Databricks system tables, things like who ran a query, who was granted or lost access, and who changed a table’s permissions. Don’t split that across separate logs that nobody checks. Once it’s all in one place, a compliance audit becomes a single query against those system tables, instead of a search through every log source the platform produces.

Conclusion

Frequently Asked Questions (FAQs)

How long does it take to establish data governance in an existing Databricks environment?

The timeline depends on how much undocumented data exists in the environment. You can inventory a few unaccounted-for tables in weeks. Years of untracked, unowned tables take longer to sort through, and that work needs to finish before anyone can estimate the rest of the project.

Does adding data governance in Databricks slow teams down?

Only if it’s done poorly. If old access gets revoked before the new Unity Catalog groups are confirmed to cover everyone, people lose access they still need, and that slows the whole team down. Checking that first, before revoking anything, avoids the problem. After that, most people get access through a group they already belong to, instead of filing a request and waiting on someone to approve it.

How do you classify sensitive data at scale without reviewing every Databricks table manually?

Classify at the column level, not the table level. A PHI or PII tag on a column carries its masking or row-filter policy with it automatically to every table that column appears in, including ones created later. The protection doesn’t depend on anyone remembering to apply it again.

How is setting up Databricks data governance in an existing environment different from setting it up from scratch?

Setting up governance before anyone starts using a Databricks environment just means writing the rules and turning them on. Nothing’s at risk yet, because nothing depends on it. An environment that’s already live is different. Pipelines, dashboards, and daily work already depend on how things function today. A change there has to be checked against what’s already running, and access has to be confirmed before it’s revoked. Skip that step, and the change ends up breaking something that was already working.