Overview

Halcrest Insurance Group is a US insurer writing auto, home, and commercial lines. It had built a data lake on Amazon S3 over several years, but writes were never governed, so claims extracts, policy snapshots, and telematics feeds accumulated as overlapping copies with no owner, no lineage, and no way to tell which version was current. Actuaries and the special investigations unit stopped querying it and went back to pulling spreadsheets from source systems, which meant fraud patterns spanning claims, policy, and telematics data were never examined together. Bacancy Technology remediated the lake in place rather than rebuilding it, adding governed Iceberg zones, a working catalog with lineage, and automated quality checks over the data already sitting in S3. Analysts now query one trusted dataset per domain and run fraud-signal analysis directly against the lake instead of reconstructing it by hand.

Technologies Used

Project Highlights

checkmark

Ungoverned S3 lake remediated in place, with no rebuild and no migration

checkmark

Claims, policy, and telematics data re-ingested through governed zones

checkmark

Every dataset cataloged with an owner, column-level lineage, and quality gates

checkmark

Fraud-signal analysis moved from manual exports to self-serve queries

The Challenges

1

Years of unmanaged writes left the lake full of duplicate claims and policy extracts. The same dataset existed in several prefixes under different naming conventions, with nothing recording which one was current, so two analysts querying the same subject returned different numbers.

2

The lake had no catalog or lineage. Teams could not confirm where a file originated, who owned it, or whether it had already been transformed upstream, which left every result requiring manual verification before anyone would act on it.

3

No quality controls ran at ingestion. Schema drift in claims extracts and interrupted telematics feeds went undetected for weeks, surfacing only when a downstream report looked wrong, and by then the affected period had already been reported on.

4

Fraud analysis depended on manual spreadsheet exports. Claims, policy, and telematics data could not be joined reliably in the lake, so patterns that only appear across all three, such as repeat claimants across separate policies, went unexamined.

Solutions by Bacancy

1

We audited what was already in S3 under our data lake consulting services, identified the authoritative copy of each dataset, and designed raw, curated, and serving zones backed by Apache Iceberg tables. Iceberg gave the lake schema evolution, snapshot isolation, and time travel, so a table can be corrected or rolled back without leaving another orphaned copy behind. Nothing was migrated to a new platform, and existing storage stayed in place.

2

As part of our data governance services, we registered every table in AWS Glue Data Catalog with a named owner, business definition, and column-level lineage back to its source system. AWS Lake Formation applies access permissions at table and column level, so policyholder identifiers and payment fields are restricted by role, and duplicate prefixes were retired once their lineage confirmed they were superseded.

3

Through our AWS automation services, we built Glue jobs that deduplicate records at ingestion and validate each feed against expected schema, row counts, and value ranges before it reaches the curated zone. Failed batches are quarantined and flagged rather than published, so a broken telematics feed or a changed claims field is caught the day it happens instead of weeks later in a report.

4

We modeled the serving zone for analysis and connected it to Amazon Athena for self-serve querying and to Power BI dashboards maintained by a dedicated Power BI developer from our team. The special investigations unit now runs cross-domain fraud signals directly against the lake, joining claim history, policy records, and telematics on a single trusted record per policyholder and vehicle.

Core Features

checkmark

Governed Iceberg zones layered over existing S3 storage

checkmark

Catalog with named ownership, definitions, and column-level lineage

checkmark

Automated deduplication and validation gates at ingestion

checkmark

Self-serve Athena queries and fraud-signal dashboards

No. of Resource

04

No. of Resource

Time Frame

April 2025 to November 2025

Time Frame

Project Snapshot

halcrest-fraud-analytics-dashboard

Outcomes

One authoritative dataset per domain, replacing several overlapping copies of claims and policy data

Fraud-signal analysis moved from manual spreadsheet exports to self-serve queries against the lake

Claims, policy, and telematics data joinable in a single governed query for the first time, with no manual exports

Broken feeds and schema drift caught at ingestion instead of surfacing in published reports

Storage footprint reduced by retiring duplicate and superseded copies once lineage confirmed them

Every table carrying a named owner, definition, and lineage record for audit and regulatory review

Technical Stack

Object storage Amazon S3
Table format Apache Iceberg
Catalog and lineage AWS Glue Data Catalog
Data transformation AWS GluePySpark
Orchestration AWS Step FunctionsGlue Workflows
Query engine Amazon Athena
Data quality Great Expectations
BI/dashboards Power BI
Security and governance AWS Lake FormationIAM
Version control and CI/CD GitGitHub Actions

Experience With Bacancy

Optimize Your E-Commerce Workflow for Maximum Efficiency

iconWatch the video

Fintech Platform Scalability & Growth Optimization

iconWatch the video

Legacy Code Upgrades for Modern User Experience and Enhanced Data Analytics

iconWatch the video

Empowering Retail Marketing With Seamless User Experiences

iconWatch the video

Migrating To Laravel From Core PHP With Twilio Integration

iconWatch the video

Upgrading PHP Laravel version while scaling the Admin Dashboard

iconWatch the video

Optimizing CRM Performance With Data Management and Salesforce Integration

iconWatch the video

AI Solution Engine for Improved Performance and Error Handling

iconWatch the video

Enhancing Managed IT Services With Streamline Automation Process

iconWatch the video

SEO and Bulk Data Upload Solutions For An E-Commerce Healthcare

iconWatch the video

Elevate Your E-Commerce with Scalable UI and Enhanced Calculators

iconWatch the video

Simplifying Annuity Rate Management for Complex Calculations

iconWatch the video

Addressing Null Safety, Code Structure Issues, and Manual Deployment Issues

iconWatch the video

Optimize Task Management With Advanced Tracking

iconWatch the video

Revolutionize Applicant Tracking: Advanced Image Optimization Techniques

iconWatch the video

Cost-Effective Media Management Solution and Resolving Data Upload Issues

iconWatch the video

Enhance Your Management Service with User-Centric Innovations

iconWatch the video

2500+ Projects Experienced Innovation with Bacancy!

Get access to an experienced team of developers and engineers from Bacancy, handpicked to ace your goals. Kickstart within 48 hours, no-risk trial.

Book a 30 min call

14+

Years of Business Experience

1458+

Happy Customers

12+

Countries with Happy Customers

1050+

Agile Enabled Employees