CASE STUDY / 03
POINT-IN-TIME SAFE MARKET-DATA INFRASTRUCTURE
QUANT MARKET DATA REPLAY
Point-in-time safe market-data infrastructure for quantitative research.
A market-data infrastructure that ingests raw exchange data, normalizes, validates, quarantines and reorders it, then serves point-in-time safe access with deterministic replay — eliminating lookahead bias by construction. Exact decimal arithmetic, nanosecond timestamps, SQLite WAL storage, sliding watermarks and deterministic tie-breaking.
EVIDENCE INSTRUMENT / PROJECT-SPECIFIC
INSTRUMENT / POINT-IN-TIME REPLAY
STRUCTURAL DEMONSTRATION — SYNTHETIC DETERMINISTIC SERIES
POINT-IN-TIME REPLAY LAB
LOOKAHEAD VS POINT-IN-TIME SAFE
The signature research instrument. Scrub the timestamp to see the information boundary — what was knowable at t, and what only existed after t.
POINT-IN-TIME SAFE VIEW (STRUCTURAL GUARANTEE)
A query at t can only read the solid candles. The ghosted region is information that did not yet exist — in a naive backtest it silently leaks in (lookahead); in point-in-time infrastructure it is structurally unreachable. Series is deterministic and synthetic.
9-STAGE PIPELINE — RAW DATA → ANALYTICS
PROBLEM
Lookahead bias is the quiet killer of quantitative research. It enters through ordinary engineering decisions: a resampled candle that includes the future tail of its window, a join that matches on post-hoc knowledge, a timestamp normalized to a coarser clock.
Once contaminated, a backtest cannot be repaired — the result must be regenerated from clean data. Prevention is the only cure, and prevention has to happen in the data layer, not the strategy layer.
MOTIVATION
I kept encountering the same failure pattern: research code that was careful about strategy logic and careless about information flow. The strategy was never the problem — the data access was.
This project makes point-in-time safety a property of the infrastructure itself, so no downstream researcher has to remember to be careful.
RESEARCH QUESTION
Can market-data infrastructure guarantee that a consumer at time t can only access information that existed at time t — deterministically, and at full research throughput?
OBJECTIVE
- Build an ingestion → normalization → validation → quarantine → reorder → storage pipeline that never silently drops or fabricates data
- Serve point-in-time safe reads with a sliding watermark
- Make replay deterministic: same query, same clock, same result — including tie-breaking
- Use exact decimal arithmetic and nanosecond timestamps so precision itself cannot introduce bias
ARCHITECTURE
VISUAL METAPHOR — TEMPORAL DATA STREAM / REPLAY ENVIRONMENT
ARCHITECTURE — DATA PATH
ARCHITECTURE DIAGRAM — THE DOCUMENTED LAYERS, NOT A MEASUREMENT
- 01
INGESTION
Raw data acquisition
- Raw exchange feeds captured unmodified
- No transformations at the boundary
- 02
NORMALIZATION
Canonical representation
- Exact decimal arithmetic — no binary floating point drift
- Nanosecond timestamps normalized to one clock
- 03
VALIDATION
Integrity gates
- Schema, range and continuity checks
- Corrupt or suspect records never reach storage silently
- 04
QUARANTINE
Failure containment
- Invalid records isolated for inspection
- Quarantine is explicit — nothing is silently dropped
- 05
REORDER
Causal ordering
- Events ordered by arrival, not by post-hoc knowledge
- Deterministic tie-breaking for equal timestamps
- 06
STORAGE
Point-in-time ledger
- SQLite in WAL mode for concurrent read access
- Sliding watermark defines visible data horizon
- 07
PIT ACCESS
Point-in-time safe reads
- Reads at time t can only see data that existed at t
- Resampled windows never include future ticks
- 08
DETERMINISTIC REPLAY
Reproducible research
- Same inputs, same clock — same output, every run
- Tie-breaking is deterministic by construction
- 09
ANALYTICS
Downstream consumption
- Backtests and studies build on guaranteed-clean access
- Adversarial tests attack the boundary
DATA FLOW
PIPELINE — 9 STAGES, IN ORDER
- 01RAW DATA captured unmodified at the boundary
- 02INGESTION into the pipeline with zero transformation
- 03NORMALIZATION to canonical form: exact decimals, nanosecond timestamps
- 04VALIDATION runs integrity gates; failures route to QUARANTINE
- 05REORDER establishes causal order with deterministic tie-breaking
- 06STORAGE in SQLite WAL behind a sliding watermark
- 07POINT-IN-TIME ACCESS serves reads visible only as of t
- 08DETERMINISTIC REPLAY reproduces any historical state exactly
- 09ANALYTICS consume guaranteed point-in-time safe data
METHODS
- Point-in-time access semantics enforced in the data layer, not by researcher discipline
- Exact decimal arithmetic — monetary and price values never pass through binary floats
- Nanosecond-precision timestamps normalized to a single clock
- Sliding watermark defines the visibility horizon for concurrent consumers
- Deterministic tie-breaking so equal-timestamp events order identically across runs
- Adversarial testing that deliberately attempts to extract future information
RISK / VALIDATION
- The primary risk this system manages is informational: no consumer may see data from after its query time
- Quarantine prevents silent data loss — every rejected record is accounted for
- Determinism eliminates a class of irreproducibility that plagues backtest research
EXPERIMENTS
Lookahead adversarial suite
COMPLETEDCan any access pattern extract information from after its query time?
Adversarial tests attack the boundary; point-in-time semantics hold.
Determinism verification
COMPLETEDDo identical replays produce byte-identical output across runs?
Deterministic replay verified — including tie-breaking on equal timestamps.
Decimal precision audit
COMPLETEDDoes exact decimal arithmetic eliminate floating-point drift in stored prices?
Exact arithmetic verified across ingestion, storage and replay.
RESULTS
9
PIPELINE STAGES
RAW → ANALYTICS
ns
TIMESTAMP PRECISION
NANOSECOND CLOCK
0
SILENT DATA DROPS
QUARANTINE MODEL
WAL
STORAGE MODE
SQLITE
This is infrastructure, not a strategy — its result is a guarantee. Downstream backtests inherit point-in-time safety by construction rather than by discipline.
The interactive replay lab demonstrates the core property: select any timestamp and see exactly which information was available then, and which only existed after.
LIMITATIONS
- Point-in-time safety covers the data layer; strategy-level misuse of resampled windows is still possible in consumer code
- Coverage is limited to the exchanges and instruments ingested so far
- Quarantine policy is conservative — some late-arriving but valid data may be held for inspection
CURRENT STATUS
RELEASED
Core infrastructure implemented; 70-test suite passes CI on Python 3.10-3.12.
EVIDENCE
LAST VERIFIED 2026-09-17 — GitHub Actions CI: full unittest suite passed on Python 3.10-3.12 · BASELINE a72d803 · VIEW THE RUN →
INTERACTIVE LAB
POINT-IN-TIME PREVIEW — SCRUB THE TIMESTAMP, WATCH THE INFORMATION BOUNDARY
A query at t can only read the solid candles. The ghosted region is information that did not yet exist — in a naive backtest it silently leaks in (lookahead); in point-in-time infrastructure it is structurally unreachable. Series is deterministic and synthetic.