Overview

FluxChain is a supply chain intelligence platform that helps organizations manage sustainability risks and conduct end-to-end supply chain due diligence. Its existing data processing workflows struggled with slow synchronization, complex data model migrations, limited automated test coverage, and manual recovery from failed or partial synchronizations across its UK and Hong Kong operations. These challenges made it harder to maintain data consistency, validate processing changes, and scale the platform reliably. Bacancy Technology replaced the legacy Ruby on Rails synchronization process with a Node.js based ETL pipeline and introduced Kafka-based distributed processing, automated testing, and data validation to improve processing efficiency and data accuracy. The solution reduced synchronization time, strengthened data validation, improved regression coverage, and made the platform easier to scale and maintain. As the engagement expanded, Bacancy Technology also implemented automated recovery workflows for failed and partial synchronizations, reducing manual reprocessing and improving recovery efficiency.

Technologies Used

Node.js
TypeScript
SQL
Prisma ORM
Ruby on Rails
Apache Kafka

Project Highlights

checkmark

Reduced data synchronization time from 24 hours to 30 minutes with a Node.js-based ETL pipeline.

checkmark

Built an end-to-end testing framework with 250+ Gherkin tests integrated into CI pipelines.

checkmark

Implemented a Data Correctness Checker for daily data quality monitoring and validation.

checkmark

Automated recovery for failed and partial synchronizations, reducing manual reprocessing efforts.

Challenges & Solutions

1

Processing millions of supply chain records through legacy workflows resulted in long synchronization cycles and delayed data availability. Large processing windows made it difficult to keep newly updated records synchronized on time, while increasing data volumes placed additional strain on existing workflows.

2

Client’s legacy data model made it difficult to consistently verify whether migrated data was complete and accurate. With changes happening across data mappings and processing workflows, identifying missing or inconsistent records during migration required additional validation and made it harder to maintain data accuracy across ongoing syncs.

3

Limited automated test coverage made it difficult to validate complex data workflows consistently, particularly when changes were made to data models or processing logic. Overreliance on manual testing increased regression testing efforts and made it harder to identify workflow issues before deployment.

4

Failed or partial synchronizations left data transfers incomplete and required manual recovery work. So, the teams had to manually identify each of the affected records, reprocess them manually and validate the recovered data, which increased recovery time and added to the operational effort.

Solutions by Bacancy

1

Bacancy Technology replaced the legacy Ruby on Rails synchronization process with a Node.js-based ETL pipeline and added checksum-based validation logic. The new pipeline reduced synchronization time from nearly 24 hours to approximately 30 minutes while maintaining data accuracy across large data transfers.

2

Our data engineers developed a Data Correctness Checker that performs daily validation of migrated data. The checker we created identifies missing or inconsistent records during the model migration process and provides greater visibility into data quality across ongoing synchronization workflows.

3

Our data integration team developed an automated end-to-end testing framework using Gherkin with more than 250 tests integrated into the pipeline. This improved regression coverage and helped maintain consistent system behavior as data models and processing workflows changed.

4

Kafka-based distributed processing was introduced with dedicated Monitor Fetcher and Processor roles to improve fault tolerance and horizontal scalability. Automated recovery workflows were also implemented for failed and partial synchronizations, reducing manual reprocessing and helping restore incomplete data transfers efficiently.

Core Features

checkmark

Real-Time Data Synchronization & ETL Pipelines

checkmark

Automated Data Validation & Quality Monitoring

checkmark

End-to-End Test Automation for Model Changes

checkmark

Automated Recovery & Reprocessing of Failed Synchronizations

checkmark

Low-Latency Data Processing & Pipeline Optimization

checkmark

Legacy Model Migration & Data Consistency Validation

No. of Resource

03

No. of Resource

Time Frame

August 2025 – Ongoing

Time Frame

Project Snapshot

Fluxchain Dashboard

Outcomes

Data synchronization time reduced from 24 hours to approximately 30 minutes.

Legacy model migration was completed with improved data accuracy and validation.

Daily data validation made it easier to detect missing or inconsistent records early.

250+ automated tests improved confidence in model changes and reduced regression risks.

The platform became more reliable and easier to scale for growing data processing demands.

Reduced operational effort by minimizing manual intervention in data processing workflows.

Technical Stack

Application & ETL Node.jsTypescript
Legacy Application Ruby on Rails
Data & ORM SQLPrisma ORM
Data Processing Apache Kafka

Experience With Bacancy

Optimize Your E-Commerce Workflow for Maximum Efficiency

iconWatch the video

Fintech Platform Scalability & Growth Optimization

iconWatch the video

Legacy Code Upgrades for Modern User Experience and Enhanced Data Analytics

iconWatch the video

Empowering Retail Marketing With Seamless User Experiences

iconWatch the video

Migrating To Laravel From Core PHP With Twilio Integration

iconWatch the video

Upgrading PHP Laravel version while scaling the Admin Dashboard

iconWatch the video

Optimizing CRM Performance With Data Management and Salesforce Integration

iconWatch the video

AI Solution Engine for Improved Performance and Error Handling

iconWatch the video

Enhancing Managed IT Services With Streamline Automation Process

iconWatch the video

SEO and Bulk Data Upload Solutions For An E-Commerce Healthcare

iconWatch the video

Elevate Your E-Commerce with Scalable UI and Enhanced Calculators

iconWatch the video

Simplifying Annuity Rate Management for Complex Calculations

iconWatch the video

Addressing Null Safety, Code Structure Issues, and Manual Deployment Issues

iconWatch the video

Optimize Task Management With Advanced Tracking

iconWatch the video

Revolutionize Applicant Tracking: Advanced Image Optimization Techniques

iconWatch the video

Cost-Effective Media Management Solution and Resolving Data Upload Issues

iconWatch the video

Enhance Your Management Service with User-Centric Innovations

iconWatch the video

2500+ Projects Experienced Innovation with Bacancy!

Get access to an experienced team of developers and engineers from Bacancy, handpicked to ace your goals. Kickstart within 48 hours, no-risk trial.

Book a 30 min call

14+

Years of Business Experience

1458+

Happy Customers

12+

Countries with Happy Customers

1050+

Agile Enabled Employees