Skip to content

Building a high-throughput sports-data ingestion platform

Automated collection and processing across five sports, designed for throughput, data integrity and reliable downstream reporting.

Role
Consulting CTO (principal-led experience)
Coverage
Five sports, collected continuously
Core
Rails, PostgreSQL (JSONB), Sidekiq

Situation

The downstream product depended on complete, timely data. The sources were outside the client’s control: formats varied, schedules overlapped, and sources changed or failed without warning.

The platform had to collect continuously, keep processing predictable, and store data flexibly enough to absorb source variation.

Responsibility

Tarun Mookhey, Pacifica’s principal, is consulting CTO on the platform, responsible for its ingestion architecture, engineering decisions and operation.

Work delivered

Pipeline

  • Scheduled collection, isolated by source
  • Parsing that keeps raw payloads beside the parsed form
  • A normalised model for the data downstream systems rely on
  • Reporting outputs for recommendation and decision systems

Reliability

  • Retry with back-off, then quarantine of bad payloads
  • Asynchronous jobs instead of work tied to the happy path
  • Monitoring of throughput and completeness, not only uptime

Outcomes

The platform runs continuously across five sports. A broken source is retried and then quarantined, so it does not hold up the rest of the pipeline, and gaps in completeness are visible before downstream users notice them.

Technical details

The pipeline runs schedule → collect → parse → normalise → persist → validate → report. Failures in collection or parsing take a separate branch: retry with back-off, then quarantine.

PostgreSQL holds normalised tables for stable data and JSONB where source structures vary. Sidekiq runs scheduled, asynchronous jobs, so throughput stays predictable when sources slow down.

Do missing or late inputs become a production problem for you?

Tell us what arrives, how often, and what depends on it.

Discuss a project