How We Help Our Clients With Databricks Performance Optimization
Last Updated on October 22, 2026
Quick Summary
This article walks through Bacancy Technology’s approach to Databricks performance optimization, based on our work with clients across banking, fintech, insurance, healthcare, and other industries. Here is what it covers:
How slow Databricks performance affects reporting, analytics, and business operations.
The seven steps our engineers follow for effective Databricks performance tuning.
The five Databricks performance challenges we see most often in client workloads, and how we solved them.
Table of Contents
Introduction
For organizations that use Databricks for analytics, reporting, and machine learning, workload performance directly affects how quickly business teams receive their data. Finance, risk, and operations teams rely on Databricks pipelines to deliver data on schedule, so a delay in one pipeline also delays every report and model that depends on it.
When these pipelines slow down, data teams often apply quick fixes to meet the next deadline. They move jobs to larger clusters, add workers, rerun failed stages, or schedule extra maintenance jobs on the slowest tables. These fixes bring runtimes down for a while, but they add to the cloud bill and leave the underlying cause in place. In our client engagements, our engineers traced most recurring slowdowns to table layout, pipeline code, and jobs competing for shared compute.
In this article, we explain how slow workloads affect business teams and share the approach we follow for Databricks performance optimization with our clients. We also walk through five performance challenges we solved for clients in fintech, insurance, healthcare, and logistics, and how we fixed them.
Understanding the Impact of Slow Databricks Performance on Business Operations
A Databricks pipeline that runs past its scheduled window delays every report, dashboard, and model built on its output. The table below lists the business areas these delays affect most and the teams that get affected.
Business Area
Effect of Slow Performance
Teams Affected
Pipeline SLAs
Batch jobs exceed their scheduled windows and delay dependent data loads
Data Engineering, IT Operations
Analytics and Reporting
Dashboards load slowly and show outdated figures at the start of the business day
Business Users, Senior Management
Machine Learning and Risk Models
Feature and scoring pipelines finish late, so fraud and risk decisions rely on outdated data
Data Science, Risk Management
Regulatory Reporting
Compliance and audit reports miss submission and review deadlines
Compliance, Finance
Compute Cost
Longer job runtimes keep clusters active and increase cloud spend
CTOs, FinOps
Engineering Capacity
Senior engineers spend planned delivery time investigating slow jobs
Data Engineering Leads
Most of these delays trace back to a few causes inside the workload itself, such as table layout, pipeline code, and compute configuration. Our engineers begin every Databricks performance tuning engagement by identifying the cause of each slow workload and then apply the fix that matches it. The next section explains the seven steps our engineers follow to tune Databricks workloads for our clients.
How We Approach Databricks Performance Optimization for Our Clients
Before our Databricks engineers change any setting, we review how each workload in the client’s workspace performs today. The review records the runtime and compute usage of every pipeline, streaming job, dashboard, and machine learning workload, and we use it as the baseline for every change that follows. The work varies by environment, but our engineers generally follow these seven steps.
1. Set Performance Targets for Each Workload
We start the Databricks performance tuning work by listing the workloads that run in the client’s Databricks workspace and agreeing on a performance target for each one with the client’s team. A batch pipeline gets a completion time that fits its scheduled window. A streaming job gets a limit on how far its output can lag behind incoming data, and a dashboard gets a maximum load time. We also agree on how much compute each workload can use to meet its target.
These targets tell our engineers which workloads to work on first and when a workload no longer needs tuning. At this stage, we also record the client’s Databricks Runtime versions and whether their tables are Unity Catalog managed tables, because several of the optimizations in later steps apply only to managed tables. For clients who have recently completed a Databricks migration, we do check whether tables still sit in the legacy Hive metastore as external tables, since Predictive Optimization covers only Unity Catalog managed tables.
2. Identify the Cause of Slow Jobs and Queries
Next, our engineers find the cause of each slow workload before changing any configuration. For Spark jobs, we use the Spark UI to find stages that take the longest, and we check their task durations, shuffle volumes, and memory spills. For SQL warehouse and serverless queries, we use Query Profile to see how much time each query spends scanning, joining, and aggregating data, and how many files it skips.
We then use system tables, such as system.query.history and the job run history tables, to see how each workload’s runtime has changed over the past weeks. A gradual slowdown usually points to data growth or table layout, while a sudden drop in performance usually follows a code change or a runtime upgrade. Here is how we read the most common findings:
A few tasks run far longer than the rest: the data is spread unevenly across tasks in a join or aggregation.
Stages spill large volumes to disk: the stage processes more data than executor memory can hold, often because filters run too late in the pipeline.
Queries read most of a table’s files to return a small result: the table layout does not match how teams filter the data.
Jobs wait several minutes before processing starts: cluster startup time adds to the runtime.
3. Optimize Table Layout and Maintenance
While helping our clients with Databricks performance optimization, we have achieved the most performance gains with changes to table layout. We confirm that every table in the workload uses Delta Lake, then review how each large table is partitioned or clustered. Databricks recommends against partitioning tables smaller than 1 TB, and against partitions holding less than 1 GB of data. Many of the slow tables we review break both rules.
For new tables and unpartitioned existing tables, we apply liquid clustering on the columns that queries filter on most often. Databricks now recommends liquid clustering in place of partitioning and ZORDER, and it cannot be combined with either. Partitioned tables therefore need a planned rewrite, which we schedule outside business hours. On Unity Catalog managed tables, we evaluate automatic liquid clustering, which selects clustering keys based on how teams query each table.
For ongoing table maintenance, we enable Predictive Optimization on managed tables, so Databricks runs OPTIMIZE, VACUUM, and ANALYZE based on how each table is used. External tables fall outside Predictive Optimization, so we schedule compaction and statistics collection for them within the client’s job workflows. We also enable deletion vectors on tables that receive frequent updates and merges, so Databricks marks changed rows without rewriting entire files.
4. Choose the Right Compute for Each Workload
With the table layout in order, we choose the compute type for each workload. Many of the slow Databricks pipelines we review run on compute that do not suit their workload.
Compute type
Best suited for
Performance considerations
Classic job compute
Scheduled pipelines that need full control over Spark settings
Startup takes several minutes, use instance pools reduce this
All-purpose compute
Interactive development in notebooks
Shared by several users; not recommended for scheduled production jobs
Serverless jobs and notebooks
Pipelines that need fast startup without cluster management
Starts in seconds and Databricks manages scaling; fewer Spark settings can be changed
Serverless SQL warehouse
BI dashboards and SQL analytics
Photon runs by default; capacity scales with the number of queries running at the same time
On classic compute, we enable Photon for workloads built on SQL and DataFrame operations, such as large joins and aggregations. Photon carries a higher DBU rate on classic compute, so we compare runtimes with and without it before keeping it on. For workloads that scale evenly across nodes, we also test a larger cluster, since it finishes the job sooner at a similar total cost.
The Databricks Runtime version also affects how fast each workload runs, because every LTS release includes Spark and engine performance improvements. So, we move clients to the latest LTS Databricks Runtime, and we test each upgrade against the baseline before rollout.
These compute choices also affect the monthly Databricks bill. To learn how we reduce compute costs without slowing workloads down, read our Databricks cost optimization Insight.
5. Fix Inefficient Pipeline and Query Code
We then review the pipeline and query code itself. We keep Adaptive Query Execution (AQE) enabled on every workload, since it adjusts query plans while they run, switches to broadcast joins when a table turns out to be small, and splits skewed partitions. When AQE does not resolve a skewed join, our engineers apply skew hints or add a random value to the most frequent join keys, which spreads their records across more tasks.
Beyond AQE, we make these changes in the code:
Apply filters early: We move filters ahead of joins and aggregations, so Databricks reads and shuffles less data.
Select only the columns needed: We replace SELECT * in production pipelines with explicit column lists.
Replace Python UDFs with native functions: We rewrite Python UDFs using built-in Spark functions, which run inside Photon and process data faster. Where Python logic is unavoidable, we use Pandas UDFs to reduce data transfer between Python and Spark.
Broadcast small tables in joins: we broadcast small dimension tables, so the large table in the join does not need to be shuffled across nodes.
Use disk caching for repeated reads: We choose worker types with local SSDs for workloads that read the same data repeatedly, and remove manual Spark caching that consumes executor memory.
For dashboards, we review the full path from the BI tool through its connector to the SQL warehouse, because a slow connector or a non-deterministic query that bypasses the result cache can slow a dashboard as much as the query itself.
6. Test Changes Against Full Production Data Volumes
Before rollout, we test every change on production data in read-only mode, or on a copy that matches production in volume, file layout, and data distribution. Smaller test datasets hide the skew and file problems that slow production jobs, so they give misleading results.
We run each workload twice, once on a freshly started cluster and once after its cache is populated, because the first run on any cluster is always slower. We then compare both runtimes against the performance targets we agreed with the client’s team and the runtimes we recorded before we started with performance tuning in Databricks.
7. Track Workload Performance After Rollout
After releasing the tuned Databricks workloads, we set up monitoring so the client’s team can see when a workload starts to slow down again. We build dashboards on the job and query system tables to track runtimes and compute usage for each workload, and we configure duration alerts on Lakeflow Jobs so the team receives a notification when a job runs past its target.
At handover, we give the client’s team a record of each workload’s runtime before and after tuning, the changes we made, and the reason for each change. We also advise the team to review workload performance after any large increase in data volume or any Databricks Runtime upgrade, as either one can slow a workload down again.
Common Databricks Performance Challenges in Client Workloads and Our Solutions
Here are the five Databricks performance challenges we see most often in client workloads, and how we solve them.
Challenge 1: Batch Pipelines Running Past Their Scheduled Window
Batch pipelines slow down when every run processes more data than it needs. Some jobs reload the full history on each run, and others join full tables before filtering them. Pipelines also collect old jobs that no team uses anymore, and these still run on every cycle. As data grows each month, the pipeline takes longer to finish. Adding compute brings the runtime down for a while, but the pipeline still processes the same excess data.
A US capital markets fintech client ran its transaction data pipelines on an aging Hadoop cluster. Its Hive and MapReduce batch jobs slowed down as transaction volumes grew, and the workloads that depended on fresh data, including fraud detection, ran late. The team also had little documentation for these jobs and could not tell which ones were still in use.
The Solution: Reduce the Data Each Batch Job Processes
We remove jobs that no team uses and replace full reloads with incremental ingestion. Our engineers also filter tables and select only the required columns before joins and aggregations.
For this fintech client, our engineers first audited the Hive and MapReduce jobs. They traced the dependencies of each job and removed the jobs the business no longer used. They then migrated the active pipelines to Databricks on AWS, rebuilt them in Apache Spark, and stored the data in Delta Lake. They also set up Auto Loader to ingest only new transaction data on each run. This helped bring the batch processing time down by more than half, and its fraud detection moved from batch runs to near-real-time processing.
Challenge 2: Queries on Large Tables Slowing Down as Data Grows
Many data teams find that queries on their largest Delta tables get slower every month, even though the queries themselves have not changed. The cause usually lies in how the table stores its data. When a team partitions a table on a column that its queries do not use as a filter, or frequent writes split the table into thousands of small files, Databricks cannot skip the files a query does not need. Each query then reads most of the table, so every day of new data adds to its runtime.
An insurance client partitioned its claims table by claim date, while claims adjusters and their dashboards filtered the table by policy number and claim status. Daily writes had split the table into thousands of small files for each date partition. Each lookup read files across the full date range, and dashboard load times grew steadily as the table expanded.
The Solution: Reorganize Table Layout Around How Teams Filter the Data
In such situations, we rebuild the table layout around the columns that queries filter on most often. Our engineers replace partitioning with liquid clustering, compact small files, and enable Predictive Optimization so Databricks keeps the layout in order as new data arrives.
For the insurance client, we rewrote the claims table with liquid clustering on policy number and claim status, and enabled Predictive Optimization on it. Claims queries now read a small share of the table’s files, and dashboard load times dropped by almost 80%.
Challenge 3: A Few Long-Running Tasks Delaying Spark Jobs
Spark divides each job into tasks and sends records with the same key to the same task. In many datasets, a few keys hold most of the records, so the tasks that process those keys run far longer than the others. The job cannot finish until its slowest task does, so most of the cluster sits idle while a handful of tasks complete.
A fintech client ran a payments pipeline that joined transactions with merchant data. A small number of large merchants accounted for most of the transactions. Most tasks in the join finished within minutes, while the tasks holding those merchants ran for over an hour and held up the entire job.
The Solution: Spread the Largest Keys Across More Tasks
Data engineers call this problem data skew. Databricks fixes many cases of it on its own through Adaptive Query Execution, which splits overloaded tasks while a job runs, so we first check that this feature works on the affected join. When the overloaded tasks remain, our engineers break up the records for the largest keys, so Spark divides them across more tasks.
For the fintech client, we identified the merchants causing the skew from the task metrics in the Spark UI, then split their records across additional tasks. The longest tasks now finish in line with the rest, and the job’s runtime fell significantly.
Challenge 4: Streaming Pipelines Falling Behind Incoming Data
We see this challenge in many of our Databricks performance optimization engagements. A streaming pipeline in Databricks processes incoming data in small batches at a set interval. When the pipeline takes longer to process a batch than the interval allows, the next batch starts late, and the delay grows with every batch that follows. On top of this, steps that track past records, such as removing duplicate events, add to this delay when they keep every record they have ever processed.
We saw this with a hospital network operator that streamed patient admission, discharge, and transfer events into Databricks for its bed management dashboards. During the morning peak hours, the pipeline took longer to process each batch than its interval allowed, and the dashboards showed hospital staff outdated bed availability. The pipeline’s deduplication step also kept a record of every event since the stream first started, which slowed each batch further.
The Solution: Match Processing Capacity to the Incoming Data Rate
We compare how fast data arrives with how fast the pipeline processes it, and track batch duration and state size over a full day. Our engineers then cap the size of each batch, add watermarks so steps such as deduplication discard old records, move these stored records to the RocksDB state store, and scale compute for the busiest hours.
For the hospital network, we added a watermark to the deduplication step, moved its state to RocksDB, capped each batch, and sized the cluster to handle the morning peak. Processing now keeps pace with incoming events, and the bed management dashboards show availability based on the latest admissions, discharges, and transfers.
Challenge 5: Workload Dependencies Delaying Data Reporting and Analytics
Teams usually build a Databricks workflow one task at a time and add each new task to the end of the chain. Over time, tasks that do not depend on each other end up running one after another, and the workflow takes as long as all its tasks combined. Many of these tasks also reload full tables on every run, even when only a small share of the data has changed since the last run.
A logistics client ran its daily shipment tracking workflow as a single chain of more than 20 tasks in Lakeflow Jobs. Several tasks reloaded entire tables on every run. As a result, the delivery performance reports for the operations team arrived late in the morning, after dispatch planning had already begun.
The Solution: Run Independent Tasks in Parallel and Process Only New Data
We map task dependencies in each workflow and restructure it so independent tasks can run in parallel. Our engineers also convert full reloads to incremental processing, using Auto Loader for new files and merges that apply only new or changed records.
For the logistics client, we split the workflow into parallel branches and converted the full reloads to incremental loads. The workflow’s runtime fell by half, and the delivery reports now reach the operations team before dispatch planning begins.
Conclusion
Our engineers trace most Databricks performance problems to table layouts, pipeline code, and compute settings that a team chose when it first built each workload. Those choices fit the data at the time. As data volumes grow and more teams use the workspace, the same choices slow pipelines and dashboards down, and teams add compute to keep them on schedule.
If you are a CTO or data leader reviewing the performance of your Databricks workloads, here are a few points to keep in mind:
Find the cause of each slow job before you add compute. Uneven data, an outdated table layout, and a shared cluster each need a different fix.
Set a performance target for every workload, so your team knows which jobs to fix first.
Organize large tables around the columns your teams filter on, and use liquid clustering and Predictive Optimization where your tables support them.
Run scheduled production jobs on dedicated job or serverless compute, separate from the clusters your analysts use.
Test each change on full production data, and review workload performance after large data growth or a Databricks Runtime upgrade.
This is how we approach Databricks performance tuning at Bacancy Technology. You can reach out to our Databricks consultants for a detailed review of your current setup and recommendations for the right table layout, compute configuration, and code improvements for your workloads.
The timeline depends on how many workloads need tuning and what is slowing them down. Setting changes, such as moving a job to different compute or enabling Photon, take less time than rewriting a large table or restructuring pipeline code. We start every engagement with a review of your workloads and share a timeline for each change once we know the cause of each slowdown.
Databricks applies several optimizations by default. Adaptive Query Execution adjusts query plans while queries run, disk caching speeds up repeated reads on supported instance types, and dynamic file pruning skips files that a join does not need. On Unity Catalog managed tables, Predictive Optimization also runs OPTIMIZE, VACUUM, and ANALYZE on its own. These features do not choose your table layout, fix inefficient pipeline code, or select the right compute for each workload, so those areas still need an engineer’s review.
It depends on the cause of each slowdown. Enabling Photon or moving a job to a different cluster type does not change pipeline code. Moving a job to serverless compute can, because serverless does not support RDD APIs, manual DataFrame caching, or most custom Spark settings. A Databricks Runtime upgrade can also change how some functions behave, so we review the release notes before each upgrade. Changes to joins, filters, or Python UDFs require code updates, and moving a partitioned table to liquid clustering requires a planned rewrite of that table. Before rollout, we test every change on production data and confirm that each pipeline still returns the same results as before.
In most cases, yes. Databricks and your cloud provider both bill for the time a job runs, so a faster job costs less. Databricks performance tuning also reduces the data each job processes, and a well-tuned workload often runs on a smaller cluster. Some changes, such as enabling Photon on classic compute, raise the DBU rate, so we compare runtime and cost before keeping them.
Supan Shah
Lead Data Engineer at Bacancy
Data engineering expert building scalable, reliable, and insight-driven data pipelines and cloud solutions.