Quant Research Backtester (Walk-Forward + Robustness)

The backtester underneath most of what Daniel Gatto publishes on daru.finance/research/quant-framework.

The readable reference half of a dual-engine walk-forward backtester. Strict no-look-ahead, walk-forward optimization (WFO), robustness stress tests, realism controls (fees, slippage, funding, SL/TP), Monte Carlo diagnostics, and overfitting statistics (DSR/PSR/MinTRL/MinBTL, PBO/CSCV). A Rust port reproduces this engine's metrics to within 1e-3, checked in CI on every push.

It answers one question: does an apparent edge survive out-of-sample evaluation under realistic frictions, or is it just fitting the past?

Two engines, one spec

This repository is the Python reference, the readable specification. A separate Rust port re-implements it for speed, and a parity oracle runs both on identical input, asserting the metrics agree within 1e-3. If the port drifts from this reference, CI goes red: the correctness claim is enforced, not asserted. Current measurements and their exact boundaries are reported below.

                ┌────────────────────────────────────────────────┐
                │                same OHLC input                 │
                │     data/SOLUSDT_1h.csv · EURUSD_1h.csv · …    │
                └───────────────┬────────────────┬───────────────┘
                                │                │
                ┌───────────────▼──────┐  ┌──────▼───────────────┐
                │  Python reference    │  │      Rust port       │
                │  backtester/         │  │  (sibling repo, …-rs)│
                │  (this repo, the spec)│ │  measured below     │
                └───────────────┬──────┘  └──────┬───────────────┘
                                │ metrics        │ metrics
                                ▼                ▼
                ┌────────────────────────────────────────────────┐
                │              parity oracle  (CI)               │
                │      tools/parity_*.py  ·  assert |Δ| ≤ 1e-3   │
                │   default 56/56 · regime+WFO 98/98 · fx 56/56  │
                └────────────────────────┬───────────────────────┘
                                         │
                                 red if the port drifts

Reproduce it in under 5 minutes

pip install -r requirements.txt
make repro

make repro runs the no-look-ahead property suite (Hypothesis, over a generated input space) and the look-ahead leak demo (listings/lah_demo.py, the paper's), which replaces every bar after #400 with noise and checks the signals before #400 are unchanged. The causal strategy is untouched; a leaky one peeking 5 bars ahead moves exactly the 4 bars that reach across the boundary:

[PASS] good (paper Listing 1, .take(idx-1) shift): 0 of 400 bars affected by post-bar-400 pollution.
[FAIL] buggy (deliberate close.shift(-5) peek): 4 of 400 bars affected by post-bar-400 pollution; first leak at bar 395, last at bar 398.

The cross-engine parity surfaces (Python vs Rust: 56/56 · 98/98 · 56/56 metric points at 1e-3) are driven from the Rust port repo; run make parity there.

What this is / what it isn't

It is a correctness-first WFO backtesting engine with the Python↔Rust equivalence machine-checked: realism controls on by default (fees, slippage, funding, SL/TP with intrabar high/low checks), strict no-look-ahead enforced by ledger-level invariant tests, and overfitting diagnostics (DSR/PSR/MinTRL/MinBTL, PBO/CSCV, deflated multiple-testing haircuts).

It isn't:

• Not alpha. The bundled strategies (EMA-cross, ATR-cross, …) are plumbing to exercise the engine, not trade signals. There is no edge here to deploy.
• Not live trading. No broker connectivity, order management, or execution: it evaluates strategies on historical bars.
• Tested on crypto, FX, and synthetic GBM only. SOL/BTC/DOGE-USDT, EUR/USD, USD/JPY, and a GBM generator. Equities, futures, and options are untried.
• Parity-gated on the core surfaces only. The stationary-bootstrap module (backtester/bootstrap.py) and the examples/ml_* strategies are Python-only: no Rust counterpart, no cross-engine check.

Quick Start

Install from a source checkout; this project is not currently published on PyPI.

git clone https://github.com/DaruFinance/quant-research-framework.git
cd quant-research-framework

pip install -r requirements.txt

# Run on the bundled real SOLUSDT hourly candles.
BT_CSV=data/SOLUSDT_1h.csv python -m backtester

The same thing from PowerShell:

pip install -r requirements.txt
$env:BT_CSV = "data/SOLUSDT_1h.csv"; python -m backtester

No data download is required. The bundled Binance spot fixture contains 53,182 hourly OHLCV bars, from SOLUSDT's first available bar on 2020-08-11 06:00 UTC through 2026-09-05 23:00 UTC, and the run prints the IS / OOS / walk-forward metrics.

To download another market:

python binance_ohlc_downloader.py --symbol DOGEUSDT --interval 30m --market spot \
    --source api --since 2017-11-01 --until now --out data/DOGEUSDT_30m.csv
BT_CSV=data/DOGEUSDT_30m.csv python -m backtester

BT_CSV overrides the CSV_FILE constant in backtester/__init__.py without touching the source. If you prefer, edit the constant at the top of backtester/__init__.py instead.

Add --volume to the downloader command for volume-based strategies. Without it, the downloader retains its five-column OHLC format. Both API and archive downloads support the flag; a resumed download must use the existing file's schema.

examples/volume_strategies.py and examples/end_to_end/end_to_end.py use the bundled OHLCV fixture by default, including when launched from another directory.

Output files and concurrent runs

The default ledger is trade_list.csv. Set BT_EXPORT_PATH or Config(export_path="runs/my-run/trades.csv") to choose another destination, with surface files written beside the ledger.

Parent directories are created automatically. Each batch run keeps its ledgers in separate directories, and the batch summary includes their paths.

A run holds an adjacent .lock file until all its ledger stages finish. Another run targeting that path fails immediately instead of overwriting it. A forced termination can leave a lock behind; remove it only after verifying the owning process has stopped.

Note: the framework was repackaged from a single backtester.py script into a backtester/ package in v0.3.0; the python -m backtester form replaces the legacy python backtester.py invocation.

---

What's Included

• backtester/: the engine package, restructured in v0.3.0 from the legacy single-file backtester.py script. Sub-modules:

  • backtester/__init__.py: the engine: IS / OOS / WFO + robustness overlays + Monte Carlo + trade ledger export.
  • backtester/__main__.py: python -m backtester CLI entry point.
  • backtester/indicators.py: TradingView-style indicator helpers (EMA, SMA, RSI, ATR, MACD, Stochastic).
  • backtester/dsr.py: Bailey & López de Prado (2014) Deflated Sharpe Ratio utility.

• binance_ohlc_downloader.py: download and format Binance OHLC candles into the CSV format the engine reads.

• gen_synthetic.py: GBM-based synthetic OHLC generator. No network required; writes data/SYNTHETIC.csv in the same format the Binance downloader emits.

• indicators_tradingview.py: backwards-compatibility shim that re-exports backtester.indicators. Pre-v0.3.0 user scripts that do from indicators_tradingview import compute_atr keep working unchanged.

• examples/: strategy and ML examples + parallel runner + the walkthrough notebook:

  • examples/atr_cross/: ATR-cross with an RSI ≥ 50 confluence (worked-example strategy).
  • examples/regime_custom/: three pluggable regime detectors (vol2, vol4, ml5).
  • examples/ml_precomputed/ and examples/ml_callback/, two ML-strategy integration patterns against the (df, lb) -> int8[] contract.
  • examples/ml_sklearn/: scikit-learn classifier wired into the strategy contract.
  • examples/ml_regime_kmeans/, KMeans-based regime detector matching the detect_regimes() API.
  • examples/batch_runner/: multiprocess runner for sweeping a parameter grid in parallel.
  • examples/notebook/walkthrough.ipynb The 10-cell tour referenced by the Binder badge above.
  • examples/README.md: the (df, lb) raw-signal contract spelled out.

• docs/: Sphinx + autodoc API reference (Furo theme), built and published to GitHub Pages by .github/workflows/docs.yml.

• tests/: pytest suite (including Hypothesis property tests on parse_signals and walk_forward_regime invariants).

• binder/: Binder configuration (requirements.txt, runtime.txt) for the launchable notebook.

---

Adding your own strategy

A strategy is a single function returning a numpy.int8 array of {-1, 0, +1} per bar:

def my_strategy(df: pd.DataFrame, lb: int) -> np.ndarray:
    ...  # return +1 (long), -1 (short), or 0 (no signal), no look-ahead.

import backtester as bt
bt.create_raw_signals = my_strategy
bt.main()

See examples/README.md for the full contract and examples/atr_cross/atr_cross.py for a worked example.

---

Key Features

Walk-Forward Evaluation (WFO)

• Baseline IS and OOS backtests
• Rolling re-optimization and forward testing per window (USE_WFO, WFO_TRIGGER_MODE ∈ {candles, trades}, WFO_TRIGGER_VAL)
• Training windows default to rolling. Set Config(wfo_window_mode="expanding") to keep the first IS start fixed while each later optimisation includes the history observed so far; on the bundled SOL data, use Config(oos_candles=10_000, wfo_window_mode="expanding"), where backtest_candles remains the initial IS length. The engine rejects expanding runs whose data cannot hold the initial IS plus the OOS span.
• Aggregated WFO performance curve + replication checks
• WFO + regime segmentation: WFO walks its standard cadence; the per-regime LB just rotates inside each window, fixed in v0.2.0 (was previously re-anchoring the IS window on every regime change).

Optimization metric

OPT_METRIC defaults to Sharpe. Daniel Gatto designed the optional Consistency score as an example objective for this project. The name does not refer to an established financial metric: the score splits trade returns into 5 chronological segments, weights later segments more heavily and blends that result with total ROI.

The single-asset optimiser selects a named field from its metric dictionary. A user can implement another optimization metric by adding its calculation to the engine's metric output and choosing that name through Config.opt_metric or OPT_METRIC; this interface has no arbitrary objective callback. backtester.objectives.MultiTermObjective is a separate objective implementation for panel research.

Realism Controls

• Configurable fees and slippage on entry/exit
• Optional funding fees for crypto at scheduled UTC times
• Optional stop-loss / take-profit with intrabar checks (high/low)
• Forex mode (FOREX_MODE): pip-based risk units (auto-detects JPY pip size), no funding fees, R-denominated PnL. Off by default.
• Session mode (TRADE_SESSIONS): restricts trading to a NY-time window (SESSION_START, SESSION_END); positions are force-closed on the last in-session bar of each day.
• Second OOS split (USE_OOS2): doubles the OOS window so the framework can score the strategy on two contiguous OOS blocks (OOS1 + OOS2) for an extra layer of out-of-sample evidence.

Regime segmentation

• 3-regime EMA-based detector by default (Uptrend / Downtrend / Ranging)
• Pluggable: override REGIME_LABELS (length 2..5) and detect_regimes(df) -> pd.Series to plug in any analytic or ML-based detector. See examples/regime_custom/ for three demos (vol2, vol4, ml5).
• Per-regime LB optimisation, OOS LB rotation, optional regime/direction filters (FILTER_REGIMES, FILTER_DIRECTIONS).

ML-driven strategies

• The signal contract (df, lb) -> np.ndarray[int8] is unchanged, so any model that can produce per-bar long/short scores plugs in.
• Two patterns shipped:
  • examples/ml_precomputed/: train offline, attach a pred column, threshold inside the strategy fn.
  • examples/ml_callback/: keep a model in memory and call predict(features) per bar (online / stateful).

Robustness / Stress Tests

Optional scenarios such as:

• fee shocks
• slippage shocks
• indicator variance
• entry drift
• synthetic “news candle” volatility injections

Statistical Diagnostics (Optional)

• Completed-trade Monte Carlo has two explicit modes: resampling draws with replacement, while permutation shuffles the observed trades without replacement. resampling is the default. Both default to 1,000 simulations, use a configurable seed and report midrank percentiles with tie counts.
• mc_bar_permutation/ provides the separate expensive mode. It reconstructs OHLCV, regenerates a frozen strategy and runs the full backtest 500 times by default. Run counts remain configurable in every mode.

Versioning

The framework follows Semantic Versioning. See CHANGELOG.md for what changed in each release; the version field in pyproject.toml is the source of truth (currently 0.7.6).

---

Requirements

• Python 3.10+ recommended
• Common packages: pandas, numpy, matplotlib, numba, pytz
  (Exact dependencies depend on your local environment.)

Install:

pip install pandas numpy matplotlib numba pytz

---

Rust port

A 1-to-1 Rust port of this backtester is available at DaruFinance/quant-research-framework-rs. Same strategy logic, same metrics, same WFO/robustness pipeline. The Rust port reproduces every IS/OOS/baseline/optimised/WFO metric line within 1e-3 relative tolerance against this Python reference on three independent surfaces:

• Default config (56/56 metric points): verified by tools/parity_check.py.
• Regime + WFO (98/98 metric points): verified by tools/parity_regime.py on the coordinated Python and Rust revisions in CI.
• Forex mode (56/56 metric points on EURUSD 1h): verified by tools/parity_forex.py.

These three are the original parity surfaces; v0.6.0 adds more (volume, shared indicators, IS-surface, overfitting statistics). Maximum observed relative deviation on the default surfaces is below 5e-5 (the metric ledger's %.4f print precision floor), 20× tighter than the declared 1e-3 tolerance. We avoid the term byte-identical for the cross-language axis: parity there is tolerance-bounded by construction, not bit-equality, and is enforced continuously by the parity_*.py suite in CI.

Cross-architecture parity, by contrast, is byte-identical. The Rust port compiled for aarch64-unknown-linux-gnu reproduces every printed metric digit exactly against committed x86_64 goldens, across all six bundled datasets (194 metric lines each, 1,164 in total), gated in CI by tools/parity_arch.py on an x86_64 drift guard, a QEMU aarch64 run, and a native ARM runner.

Performance

The current benchmark uses 150,000 real BTCUSDT 30-minute bars. One fresh process per engine runs ten fixed configurations, five signal families with two lookbacks each, through 28 full WFO windows. Python's first Numba compile and persisted ledgers are included; Rust compilation is excluded.

Engine	Wall	User + sys	Peak RSS
Python reference	112.37 s	109.60 s	287.37 MiB
Rust port	7.20 s	5.54 s	10.75 MiB

This single batch observed 15.61 times less wall time and 26.73 times less peak memory for Rust. All ten ledger row counts matched, and the largest deterministic metric difference was 6.67e-14. This is one observation under the stated workload, not a general speed guarantee.

A separate matched execution test compares QRF, vectorbt and Backtesting.py on identical frozen events. It does not include WFO or signal calculation. The full method, competitor results, data recipe and machine-readable evidence are in benchmarks/.

Comparison vs other open-source backtesters

What this framework emphasises that mainstream open-source alternatives do not (verified against primary docs as of 2026-04):

Framework	License	Built-in WFO	Per-regime LB optimisation	Strict-LAH property tests	Cross-language parity tests
this (Python + Rust)	Apache-2.0	✓	✓	✓	✓
vectorbt	Apache-2.0 + Commons	✓ (Splitter)	✗	✗	n/a
backtrader	GPL-3.0	✗ (community)	✗	✗	n/a
NautilusTrader	LGPL-3.0	✗ (engine only)	✗	✗	✗ (bilingual; no parity asserts)
zipline-reloaded	Apache-2.0	✗ (3rd-party)	✗	✗	n/a
QuantConnect Lean	Apache-2.0	✓	✗	✗	n/a
bt	MIT	✗	✗	✗	n/a

The combination is the contribution: WFO + per-regime LB + strict no-look-ahead enforced by ledger-level invariant tests + a Python reference and Rust port whose metric outputs agree within $10^{-3}$ relative tolerance on every validated surface, enforced continuously by the parity_*.py harness suite in CI. Each cell individually exists somewhere; no other framework ships the whole bundle.

Citation

If you use this framework in academic or research work, please cite via CITATION.cff. The Rust port has its own CITATION.cff and citing either implies the other (sibling cross-reference).

License

See LICENSE for details.