Building a high-throughput sports-data ingestion platform
Automated collection and processing across five sports, designed for throughput, data integrity and reliable downstream reporting.
- Role
- Consulting CTO (principal-led experience)
- Coverage
- Five sports, collected continuously
- Core
- Rails, PostgreSQL (JSONB), Sidekiq
Situation
The downstream product depended on complete, timely data. The sources were outside the client’s control: formats varied, schedules overlapped, and sources changed or failed without warning.
The platform had to collect continuously, keep processing predictable, and store data flexibly enough to absorb source variation.
Responsibility
Work delivered
Pipeline
- Scheduled collection, isolated by source
- Parsing that keeps raw payloads beside the parsed form
- A normalised model for the data downstream systems rely on
- Reporting outputs for recommendation and decision systems
Reliability
- Retry with back-off, then quarantine of bad payloads
- Asynchronous jobs instead of work tied to the happy path
- Monitoring of throughput and completeness, not only uptime
Outcomes
Technical details
The pipeline runs schedule → collect → parse → normalise → persist → validate → report. Failures in collection or parsing take a separate branch: retry with back-off, then quarantine.
PostgreSQL holds normalised tables for stable data and JSONB where source structures vary. Sidekiq runs scheduled, asynchronous jobs, so throughput stays predictable when sources slow down.
Do missing or late inputs become a production problem for you?
Tell us what arrives, how often, and what depends on it.