The challenge of dynamic pages
Many modern websites are single-page applications. When you request the page, the server returns very little; the real content is assembled moments later by JavaScript running in the browser, often loading in stages as the user scrolls or interacts. A traditional server-side fetch sees an almost-empty page and misses the data entirely.
Capturing the page's own live state
Rather than fight the browser, Databuilder works with it. A lightweight, purpose-built script runs on the fully-rendered page and reads the data from the application's own in-memory state - not just the visible HTML. Depending on how a site is built, it reads directly from the page's live data store (subscribing to it so that results which stream in lazily, as the visitor scrolls, are captured as they arrive), from framework component instances, from web-component properties, or from state embedded in the page. It batches what it finds and submits it to a serverless ingest endpoint. Because the page has already done the rendering work, capture is fast, accurate, and adds virtually no compute cost of its own.
A serverless ETL pipeline
Captured data flows into an event-driven pipeline on AWS. There is no server, and no function, in the hot path: Amazon API Gateway hands each incoming batch straight to an Amazon SQS queue, which absorbs bursts and decouples capture from processing (with a dead-letter queue and bounded retries so a bad batch can never jam the flow). A commit AWS Lambda then drains the queue in batches, loads the appropriate parsing template, validates every extracted record against a strict JSON schema, enriches it with reference data, transforms it into the platform's standard format, and publishes it onto the same central streaming bus that powers the rest of Distribute's data platform. In other words, Databuilder is a drop-in alternative source of data for a pipeline we had already built - and if one record in a batch is bad, only that record is retried, never the whole batch.
Template-driven, with quality guaranteed
Databuilder is deliberately data-driven rather than hard-coded. Each page is described by a JSON parsing template, so once a site's underlying technology is supported, bringing on a new page means authoring a template - not writing and deploying new code. Two schemas keep it honest: one validates the template itself, and one validates every record the template produces, so nothing malformed can enter the pipeline. To make authoring easy, we built a Validator: a browser-based editor for writing and testing templates against real pages, backed by the very same parsing and validation code running server-side against a headless browser. An author sees exactly what production will see - the extracted, enriched, transformed result - before a template ever goes live.
Key AWS services
- Amazon API Gateway: receives captured data batches and passes them straight to the queue, with no function in between.
- Amazon SQS: absorbs bursts of incoming data and decouples capture from processing, with a dead-letter queue for safety.
- AWS Lambda: validates, enriches, transforms and commits captured data, and renders pages headlessly for the Validator - entirely serverless.
- Amazon Kinesis: the shared real-time streaming bus that carries data into the wider platform.
- Amazon S3: hosts the parsing templates, reference data and the Validator tool.
- Amazon CloudWatch: monitoring and metrics across the pipeline.
