SERVICES / DATA
EVIDENCED BY THE ARCHIVE
Point-in-time correct market-data pipelines with deterministic replay.
WHO THIS IS FOR — Teams whose research and systems depend on market data that must be point-in-time correct, reproducible, and honest about its own gaps.
THE PROBLEM IT ADDRESSES
- Vendor and internal data that disagree with each other with no canonical model.
- Silent lookahead: reads that can reach future information and quietly invalidate research.
- Replays that cannot be reproduced, so bugs and anomalies cannot be investigated after the fact.
EXPECTED DELIVERABLES
- An ingestion → validation → quarantine pipeline with explicit failure states.
- A canonical model with exact-decimal storage and nanosecond timestamps where the source supports them.
- Deterministic replay plus adversarial tests written to extract the future rather than confirm correctness.
OUT OF SCOPE
- Exchange data licensing or redistribution rights negotiation.
- Vendor selection on your behalf — recommendations are advisory, contracts are yours.
- Real-time trading signal delivery at exchange co-location scale.
TECHNICAL APPROACH
HOW THE WORK IS DONE
Point-in-time safe market-data infrastructure: ingestion, normalization, validation and deterministic replay.
SCHEMATIC · DATA ENGINEERING
METHODS
SQLite WAL · Nanosecond timestamps · Exact decimals
EVIDENCE
market-data-replay ingests into SQLite WAL with nanosecond timestamps and exact decimals, and replays it deterministically.
LIMITATIONS
SCOPE & LIMITS
- Point-in-time correctness is provable for the sources ingested; historical gaps in the source remain gaps.
- Determinism is verified for the pipeline, not for upstream vendor revisions to historical data.
- Performance figures are measured on the stated hardware with the stated methodology.
WORKING PROCESS
HOW THIS ENGAGEMENT RUNS
- 01INGEST
- 02VALIDATE
- 03QUARANTINE FAILURES
- 04CANONICAL MODEL
- 05DETERMINISTIC REPLAY
Each stage produces something checkable. The full process, with what every stage produces across all work, lives on the process page. Every engagement starts the same way: describe the problem, the data, and the evidence that would count as success.