Confidential client, name withheld under NDA

Real-Time Fraud Detection Pipeline on Databricks for a US Payments Company

Quick answer: A real-time fraud detection pipeline on Databricks scores each payment as it streams in, using Delta Lake, Spark Structured Streaming and MLflow models. Bacancy built this for a US fintech that processes digital payments.

Outcomes deliveredOngoing since August 2025 with a team of 3 engineers
35%

Lower false positive rate

In fraud detection decisions

45%

Higher fraud detection rate

Using adaptive behavioral signals

50%

Less manual fraud review work

Through automation

5M+

Transactions handled per day

Without performance degradation

Client name withheld under NDA. To honour our non-disclosure agreement and compliance obligations, we describe this client by industry and region only. The project scope, technology and outcomes are shared without details that could identify them.

About the Client

The client is a US fintech company that processes digital payments for global financial institutions and enterprises. It focuses on fraud prevention, risk reduction and safe transaction handling with real-time streaming data.

The client needed to catch fraud in real time across high transaction volumes. False positives affected legitimate customer payments. Suspicious activity across linked accounts, payment instruments and merchants was hard to detect, and feature engineering had to match across streaming and batch.

Bacancy built a Delta Lake lakehouse with Spark Structured Streaming that scores each transaction instantly. The client then extended the engagement to add cross-account relationship analysis. The pipeline now handles 5M+ transactions a day, false positives fell 35% and coordinated fraud detection rose 25%.

Client type
Digital payment processing company
Industry
Fintech
Location
USA
Team
3 engineers
Timeline
August 2025 to ongoing
Project size
Non-disclosable
Engagement
Databricks development, extended

Four gaps in payment fraud detection

The client needed to score high payment volumes in real time while keeping legitimate customer transactions moving.

High volume, real-time need

What was happening

Fraud had to be detected as transactions streamed in, at high volume.

Business Impact: Detection had to keep pace with the live transaction stream.

Too many false positives

What was happening

Fraud decisions flagged legitimate payments as suspicious.

Business Impact: Customer transactions were held up in processing.

Linked-account fraud missed

What was happening

Suspicious activity spread across linked accounts, payment instruments and merchants.

Business Impact: Transaction-level checks alone could not see coordinated fraud.

Features split across modes

What was happening

Fraud features had to stay consistent between streaming and batch processing.

Business Impact: Detection precision depended on matching features in both modes.

Challenges & Solutions

Our team matched each detection gap with a part of one lakehouse pipeline.

Challenge 1: Ingesting high transaction volumes in real time

The client had to process high transaction volumes and detect fraud as each payment arrived.

Solution: Our Databricks developers designed a lakehouse architecture on Delta Lake to ingest and process real-time transaction streams. The pipeline now handles 5M+ transactions a day without performance degradation.

Challenge 2: Cutting false positives on customer payments

False positives held up legitimate customer transactions.

Solution: We built Spark Structured Streaming pipelines that score each transaction instantly on behavioral signals, patterns and historical anomalies. False positives dropped 35%.

Challenge 3: Detecting coordinated fraud across linked accounts

Suspicious activity spread across linked accounts, payment instruments and merchants.

Solution: Our Databricks consultants assessed cross-account fraud patterns and built relationship analysis across accounts, instruments and merchants. Detection of coordinated payment fraud rose 25% over transaction-level detection alone.

Challenge 4: Keeping fraud features consistent in streaming and batch

Feature engineering had to match across streaming and batch environments.

Solution: Our team built a dynamic feature engineering framework that updates indicators such as transaction velocity, IP mismatch and geo-location deviation. Feature optimization improved model accuracy by 40%.

How the Platform Works

The client fraud detection dashboard screen

Core Features

  • Real-time fraud scoring engineEvery transaction monitored and scored as it streams in.
  • Delta Lake streaming ingestionOne lakehouse layer for real-time transaction streams.
  • Cross-account relationship analysisLinks accounts, payment instruments and merchants to expose coordinated fraud.
  • Consistent fraud featuresSame velocity, IP and geo-location features in streaming and batch.
  • MLflow model lifecycleFraud models tracked in MLflow and served through Databricks Model Serving.

The stack behind the pipeline

Data engineering
Databricks | Delta Lake | Apache Spark Structured Streaming
AI and ML
MLflow | Python | PySpark
Streaming
Kafka integration
Query layer
SQL on Databricks
Deployment
Databricks Model Serving
Cloud
Databricks Lakehouse Platform

Ready to start?

2500+ projects delivered by Bacancy Technology

Get access to an experienced team of developers and engineers from Bacancy Technology, handpicked to ace your goals. Kickstart within 48 hours, no-risk trial.

Book A 30 Min Call
14+Years of business experience
1458+Happy customers
30+Countries with happy customers
1050+Agile-enabled employees

More Case Studies

View All Case Studies