How We Handled Databricks Cost Optimization: Insights From Real Client Projects
Last Updated on September 1, 2026
Quick Summary
This article walks through Bacancy Technology’s approach to Databricks cost optimization, based on our work with clients across healthcare, finance, banking, insurance, and other key industries. Here is what it covers:
How Databricks pricing works, including DBUs, cloud infrastructure costs, storage, and data transfer.
The six cost problems we see most often, from idle and oversized clusters to unnecessary.
The six best practices we recommend to keep Databricks costs under control as the account grows.
Table of Contents
Introduction
Databricks bills you twice. The first invoice covers DBUs, which basically tell the platform’s unit of consumption. The second bill arrives from AWS, Azure, or Google Cloud for where those DBUs run.
We have seen teams plan and approve the budget for the first number, only to find later that the actual total spend is close to double what they were approved for.
We have helped several clients with Databricks cost optimization across banking and finance, healthcare, insurance, and other key industries, and this is just one problem, or a starting point among several.
So, read this article, as we cover how the platform’s pricing works, the cost challenges we have usually seen, how we solve them, and the best practices we recommend following.
Understanding the Databricks Pricing Model
Just like Snowflake, Databricks also runs on a consumption model, so your bill is the total of what each part of the platform uses, and not an upfront licensing fee. The difference is just the pricing per component and the number of components involved. Here is an overview of the Databricks pricing structure:
What you pay for
How Databricks bills it
Typical rate
Compute (DBUs)
Per second while a cluster runs; the compute type you pick sets the rate
All-purpose ~0.55perDBU, jobscompute~0.15, roughly 70% less
Cloud infrastructure
Billed by your cloud provider for the VMs, disks, and network traffic under every classic cluster
Often matches or exceeds the DBU half; spot workers cut it sharply
Storage
Per GB in your own object store; Delta file history, checkpoints, and small files add to the footprint
Standard S3, ADLS, or GCS rates, no Databricks markup
Serverless and always-on services
One all-in rate, no separate VM bill; endpoints and continuous streams bill while provisioned
Billed in DBUs, with no cluster in the workspace to watch
Pricing tier and commitment
Sets your per-DBU rate and available governance features
Standard, Premium, or Enterprise; on-demand or committed use
Compute is where most of your Databricks spending goes, and you pay for it twice, once to Databricks and once to your cloud provider. Storage and serverless cost less but are easier to miss, and which of the Databricks pricing plan/tier you select sets the rate you pay for everything else.
Spending More on Databricks Than You Expected?
Talk to our Databricks consultant to review your setup and find where the DBUs are going before the next invoice comes.
Top 6 Databricks Cost Challenges And How We Solve Them
In this section, we cover the six cost challenges we see most often in Databricks environments and how we address them for our clients.
Challenge 1: Idle Clusters and Over Provisioning
A Databricks cluster charges the same whether queries are running on it or not. Nothing shuts it down unless auto-termination is configured, and most workspaces unintentionally leave it as the default. The same is the case with cluster over-provisioning. Teams oversize or provision a cluster assuming heavy traffic in the future, and forget to adjust it for normal days.
We have seen this mostly with teams running development, testing, and ad hoc analytics workloads, where clusters remain active between jobs or are provisioned with more CPU and memory than the workload requires. Over time, these small inefficiencies add up, especially when multiple teams maintain their own clusters without regular usage and sizing reviews.
The Solution: Set Up Auto Termination and Right Size Clusters
We set auto-termination on interactive clusters at 10 to 15 minutes and move ad hoc SQL to a serverless warehouse to eliminate anything running between queries. We set the minimum worker count to what the workload needs on a normal day, and let autoscaling add capacity when demand rises.
Where a team cannot tolerate startup delay, we use an instance pool, which keeps a few machines ready so a cluster can start quickly without the whole cluster staying up, and you do not get billed for the unused compute; you just need to pay for the cloud you host on.
Challenge 2: Double Billing (Cloud + DBUs)
Databricks charges for DBUs, and your cloud provider charges separately for the machines those DBUs run on. So, a single mistake of oversizing without need, and you get a spike on both the bills. Most teams only track the Databricks invoice. The cloud bill goes to a different owner and arrives on a different cycle, so usually teams avoid validating it against the workloads that caused it.
We had a banking client who had recently migrated to AWS, with their Databricks workloads running on AWS compute. During our cost review, we found that the team was tracking Databricks DBU costs but not the corresponding AWS compute costs, which meant the actual cost of several workloads was much higher than expected.
The Solution: Workload-Level Cost Attribution
The solution is to track Databricks DBU costs and underlying cloud infrastructure costs together at the workload level. By mapping clusters to cloud resources and monitoring both costs against actual usage, teams can identify expensive workloads, right-size compute, and avoid paying twice for unnecessary capacity.
For this client, we brought both sides of the cost into the same view by mapping Databricks workloads to their underlying AWS compute resources. This helped us identify where compute costs were disproportionate to workload usage, right-size the affected clusters, and give the team a clearer view of the actual cost of running each workload.
Challenge 3: Photon Running on Workloads That Do Not Need It
Photon is a high-performance vectorised query engine that speeds up data processing but costs more DBUs per hour. On heavy scans, joins, and aggregations, the speedup it provides is worth more than the cost it charges. But on small batches and short jobs, the time saved is too small to cover the cost involved.
Teams usually run it across the workspace because of the myth that Photon automatically makes every workload faster and therefore justifies the additional DBU cost.
The Solution: Use Photon Based on Workload Requirements
Enable Photon selectively based on workload characteristics, rather than treating it as a default for every cluster.
In a recent Databricks cost optimization engagement for a healthcare client, we reviewed workloads based on runtime, query patterns, and DBU consumption, then disabled Photon where the performance gains did not justify the additional cost. This helped the client reduce unnecessary DBU consumption while retaining Photon for workloads where its performance benefits outweighed the cost.
Challenge 4: Data Transfer and Storage Costs You Never See
This is another Databricks cost challenge that is related to the cloud platform it runs on. So, the cloud platform you deploy your Databricks workloads on doesn’t just charge you for the machines you use, it also charges you for the data you store and transfer across regions.
We have seen clients make this mistake by focusing only on Databricks compute costs while overlooking the storage and data transfer charges on their cloud bill. In one recent engagement with an insurance client, their team was moving data between different AWS environments and retaining multiple copies of older datasets, which added to their overall AWS costs without providing much additional value.
The Solution: Data Transfer and Storage Optimization
Teams should review where data is stored, how often it moves, and how long it needs to be retained. Reducing unnecessary cross-region transfers, removing redundant data copies, and applying appropriate retention policies can help control these costs.
For the insurance client, we reviewed the data flows between their AWS environments, identified unnecessary transfers and duplicate datasets, and adjusted their storage and retention practices to reduce the AWS costs associated with moving and storing that data.
Challenge 5: Spending on Databricks Without Tagging and Cost Attribution
Databricks costs can become difficult to track when teams, projects, and workloads all start sharing the same compute resources without any proper cost attribution. Without consistent tagging, organizations may know how much they are spending overall but not which team, application, or workload is driving that spend.
We have seen this with clients where multiple teams were running workloads in the same Databricks environment, but the costs were appearing under shared resources. This made it difficult to identify which teams were responsible for rising costs or determine whether the spend was aligned with the goals and value the business aims to achieve.
The Solution: Implement Cost Attribution and Tagging
A consistent tagging strategy can link Databricks workloads and cloud resources to specific teams, projects, and environments, making costs easier to track and allocate.
For our clients, we recommend standardizing tags across their workloads and mapping the associated compute costs to individual teams and projects, for a clearer view of where the Databricks spend is coming from & who is responsible for it.
Challenge 6: Running Everything On Demand
We have seen many clients running a Databricks workload as an on-demand job. They use the compute for recurring ETL jobs, scheduled data processing, and reporting workloads that run at regular intervals, even though some of these workloads could use more cost-efficient compute options like job clusters or spot instances. This leads to paying a premium price for the flexibility that was not even needed.
The Solution: Match Your Compute Usage with Workload Demand
The right compute option depends on how frequently a workload runs, how predictable its demand is, and how much flexibility it needs. For recurring workloads, teams can evaluate job clusters, serverless compute, or spot instances where interruptions are acceptable.
In one recent insurance engagement, we found several recurring jobs using on-demand compute simply because it was the default. We changed the compute option for workloads that did not require it, bringing down the cost of those jobs.
Looking for Help With Your Snowflake Costs?
We also have a dedicated guide on Snowflake cost optimization, covering the common cost issues we see and the ways we help clients bring their Snowflake spend under control, along with the best practices we recommend to follow.
The 6 Best Practices for Databricks Cost Optimization We Recommend
The six fixes above address specific cost problems. The six Databricks cost optimization best practices we listed below are derived from our experience across multiple Databricks engagements and help keep cost problems from coming back by putting the right controls, ownership, and regular reviews in place.
1. Set Cluster Policies and Budget Limits
Cluster policies should be the default control for Databricks compute. They can limit instance types and cluster sizes, enforce auto-termination, require cost attribution tags, and prevent users from creating configurations that fall outside your cost limits.
On top of this, add budget alerts for teams and workspaces so you know when spending goes beyond the expected range.
2. Schedule Regular Table Maintenance
Regular table maintenance helps keep Delta tables from accumulating unnecessary files and becoming slower to query. As more files build up, queries can take longer and use more compute, which adds to the Databricks bill.
We have seen this in healthcare and insurance workloads where claims, policy, and transaction data are updated throughout the day. For these workloads, we review how tables are being written and maintained, then address file growth and old data that is driving unnecessary compute usage.
3. Assign an Owner to the Databricks Bill
Someone from your team needs to own the Databricks bill. Without defined ownership, Databricks cost discrepancies can easily get overlooked, with no one checking why a workload suddenly consumed more DBUs or why the monthly spend went up.
We have seen this a lot with our BFSI clients, where multiple teams run Databricks workloads across claims, policy, and reporting use cases, making it difficult to trace a sudden increase in spend to a specific workload or team. So, recommend a weekly review of Databricks spend, with the owner checking which workloads drove any increase and why. This makes it easier to catch cost issues early instead of finding them after the bill has already gone up.
4. Review Your Pricing Tier and Commitment Once a Year
Your Databricks pricing tier and commitment should be reviewed as your usage changes. We see teams select a tier that they deemed fit when they first set up their workspace, and never revisit it, even after their workloads and DBU consumption have grown.
Review your actual usage, the features you need, and your consumption pattern once a year to see whether a different tier or committed use option would make more sense.
5. Estimate Cost During Design, Not After Deployment
Estimate what a workload is likely to cost before you start building it. Look at how much data it will process, how often it will run, the type of compute it will use, and how much storage it will require.
We have seen teams discover the actual cost of a new workload only after it went into production, when changing the design was already more difficult. So, we recommend a Databricks cost estimation in advance to spot an expensive workload before it hits production and make changes while the design is still easy to adjust.
6. Check Always-On Services on a Schedule
Some Databricks services keep running until someone turns them off. Model Serving endpoints, Vector Search endpoints, SQL Warehouses, and continuous streaming jobs are a few examples.
We have seen teams leave these running after a pilot or project has ended, adding to the bill without anyone noticing. Check these services regularly and turn off the ones that are no longer being used.
Conclusion
Databricks cost optimization is not something you do once the bill gets out of control. The same flexibility that makes Databricks easy to scale can also make it easy to overspend, especially when teams add new workloads, increase compute, or leave resources running without reviewing their actual need.
In the engagements above, the focus was not simply on cutting compute. We looked at how workloads were running, where the spend was coming from, and which changes could bring the cost down without affecting the work those workloads needed to do.
The goal is to make Databricks match actual usage. Set the right cluster policies, track costs by workload, review compute regularly, and define ownership for the bill.
This way, teams can keep their Databricks costs under control as they grow in size.
Doing this well requires expert help from those who know how Databricks compute, DBUs, storage, and cloud costs work together. You can hire Databricks engineers from Bacancy Technology who have worked on Databricks cost optimization across healthcare, insurance, and banking environments, helping teams reduce unnecessary spend without affecting workload performance.
We use Databricks System Tables and billing data to analyze DBU and workload costs, along with AWS Cost Explorer for the underlying cloud infrastructure costs in AWS environments. We also use cluster policies and tagging to put cost controls and attribution in place.
A typical Databricks cost optimization exercise can uncover around 25% to 45% in savings, although the actual figure depends on how much idle, oversized, or unnecessary compute is in the environment. More extensive optimization can deliver higher savings when there are issues across compute, cloud infrastructure, storage, and data transfer.
Idle and oversized clusters are among the most common causes we see. Teams may leave clusters running between jobs or provision more compute than their normal workloads need. Other cost factors, such as cloud infrastructure, Photon, data transfer, and always-on services, can also add significantly to the bill.
It should be ongoing. A Databricks cost optimization initiative can address the main issues in an existing environment, but new workloads, clusters, services, and changes in usage can increase costs again. Best practices like Cluster policies, cost attribution, regular bill reviews, and checks on always-on services help keep those costs under control over time.
Supan Shah
Lead Data Engineer at Bacancy
Data engineering expert building scalable, reliable, and insight-driven data pipelines and cloud solutions.
