The legacy challenge
The original system was a monolithic .NET application responsible for the entire data journey. Over time it had become expensive to run, difficult to change, and hard to scale to keep pace with growing data volumes. Fixed infrastructure meant paying for capacity around the clock, whether or not data was flowing.
Distribute needed a pipeline that could ingest high volumes of data from multiple external operator sources continuously and reliably, transform it into a consistent format for downstream consumption, scale automatically without paying for idle capacity, and give the team clear operational visibility and simple controls over the flow of data.
The full data journey, end to end
What made this engagement distinctive is that we were trusted with the full breadth of Distribute's data journey. We joined the "bookends" of the platform - the collection of source data at one end, and the APIs that serve it at the other - with a brand-new serverless ETL system in the middle. The result is a single, coherent pipeline from raw source data through to partner-ready output.
An event-driven serverless architecture on AWS
We rebuilt the pipeline as a set of small, independent AWS Lambda functions, coordinated by events rather than a single long-running application. It has two complementary halves.
Ingestion. The team decides what to collect in an ordinary spreadsheet, uploaded to Amazon S3. A Lambda expands that spreadsheet - a handful of rows describing destinations, dates, durations and party sizes - into the full combination of individual queries it implies, on the order of a million a day. Those queries are then metered onto an Amazon SQS queue at a carefully controlled rate so ingestion stays smooth and comfortably within each operator's API limits. The rate limiter is itself serverless and stateless: once a minute it reads only the next slice of the query file straight from S3 (using S3 Select byte ranges), keeps its position in DynamoDB, and spreads that minute's work evenly across the window - so it resumes exactly where it left off and never runs a server of its own. Worker Lambdas then retrieve the source data, transform it into a standard format, and publish it onto the platform's central real-time streaming bus, built on Amazon Kinesis. To avoid disrupting anything downstream, we reverse-engineered the exact record format the legacy system produced - down to its hashing scheme - so the new pipeline's output is byte-for-byte compatible with what consumers already expected.
Consumption and delivery. A fleet of consumer Lambdas reads from that streaming bus. A single, config-driven transform engine turns each canonical record into both a fast-access item in Amazon DynamoDB and a row batched into an Amazon Redshift data warehouse via Kinesis Data Firehose - one engine feeding many destinations. Generating the partner feed files is fully event-driven: an hourly Lambda fires an asynchronous Redshift query and stops, and a separate Lambda wakes only when that query finishes to stream the results into feed files on S3. No function ever sits idle waiting on the warehouse.
Key AWS services
- AWS Lambda: the entire pipeline runs as small, single-purpose serverless functions - no servers to manage or patch.
- Amazon Kinesis: a real-time streaming bus that decouples data collection from downstream processing.
- Amazon Kinesis Data Firehose: reliably batches streamed data into the analytics warehouse.
- Amazon Redshift: a scalable data warehouse queried asynchronously to generate output feeds.
- Amazon DynamoDB: a fast, serverless key-value store for the hot cache and the rate limiter's cursor.
- Amazon SQS: rate-controls high-volume ingestion so it stays smooth and reliable.
- Amazon S3: durable storage for configuration inputs and generated output files.
- Amazon EventBridge & CloudWatch: event-driven scheduling, dashboards and alerting for full operational visibility - including automated phone-call paging to the on-call engineer when something needs attention.
Built for scale, cost and control
Because every component is serverless, the pipeline scales from zero to high throughput automatically, and Distribute pays only for the compute it actually uses - a dramatic change from the fixed cost of the legacy system. And because collection is split into independent per-operator pipelines, the team has fine-grained command over the flow: any single pipeline can be paused instantly with one switch, without touching the rest.
