Reproduce our benchmarks in 60 seconds

Every claim on this website links to a reproducible script. When someone tweets "prove it," this is the answer.

The headline claim

3–8× faster than PySR on the Nguyen suite (4× median, 8× on the trigonometric subset), 88 / 100 recovered on Feynman. Wall-clock, single RTX 4090, Python 3.11 + Julia 1.10 (PySR), batch size 100 problems.

Reproduce it yourself

git clone https://github.com/formulize-dev/formulize
cd formulize
git checkout v0.4.0                    # exact tag the site claims are pinned to
pip install -e ".[bench]"              # installs formulize + pysr + gplearn + operon
bash benchmarks/reproduce.sh --seed 42 # deterministic, ~60s on RTX 4090

Output (~60 seconds on a single RTX 4090, seed 42):

engine       | nguyen_median | nguyen_trig | feynman_recovered | structural_ci
-------------|---------------|-------------|-------------------|--------------
formulize    |      8.4 s    |    3.1 s    |     88 / 100      |     yes
pysr         |     34.0 s    |   24.1 s    |     63 / 100      |     no
operon       |     67.0 s    |   58.9 s    |     61 / 100      |     no
gplearn      |    142.0 s    |  118.3 s    |     41 / 100      |     no

The "3–8×" range is honest and comes from the two columns above: 34.0 / 8.4 = 4.05× on the Nguyen median, 24.1 / 3.1 = 7.8× on the trigonometric subset. Anywhere on the site that says just "8× faster" is quoting the best-case row; the honest median is 4×. Both are in the table.

Full seed & hardware manifest

Item	Value
Git tag pinned to site claims	`v0.4.0` (SHA in `benchmarks/PINNED_SHA`)
Deterministic seed	`42` (search RNG, bootstrap resampler, train/test split)
Bootstrap iterations	`1000`
OOS eval	5-fold cross-validation, folds seeded from `42`
Hardware	Single NVIDIA RTX 4090 (24 GB), AMD 7950X, 64 GB DDR5, Ubuntu 22.04, driver 535.129
Python / Julia	Python 3.11.7 / Julia 1.10.2 (PySR only)
PySR settings	`niterations=40, populations=15, procs=32` (their published default for Nguyen)
Operon settings	`generations=1000, population=1000, evaluator=r2`
gplearn settings	`population_size=5000, generations=40`
Wall-time measurement	Median of 5 runs, taken after 1 warm-up run (JIT + GPU init discarded)

Dataset provenance

Suite	Origin	Files in repo
Nguyen (12 problems)	Nguyen et al. 2011, An Investigation of the Efficacy of Genetic Programming for Symbolic Regression	`benchmarks/data/nguyen/*.csv`
Nguyen-trig subset (5)	Nguyen subset containing sin/cos/tan — the workload where our typed-program search dominates	`benchmarks/data/nguyen/trig/*.csv`
Feynman (100)	Udrescu & Tegmark 2020, AI Feynman. Downloaded from MIT mirror	`benchmarks/data/feynman/` (auto-fetched by `reproduce.sh`)
SRBench-lite	La Cava et al. 2021, subset of 20 real-world tabular problems	`benchmarks/data/srbench/`

What benchmarks we run

Nguyen suite — 12 canonical SR problems (Nguyen et al. 2011). Simple polynomial + trigonometric expressions.
Feynman equations — 100 physics equations from Udrescu & Tegmark (2020, "AI Feynman"). Real physics with noise + realistic units.
SRBench — La Cava et al. (2021) real-world regression benchmarks with noisy tabular data.
Our own industrial datasets — Fischer-Tropsch catalyst kinetics, Pt-C electrocatalyst oxidation, and 3 anonymized fintech alpha primitives.

How we compute "confidence"

Every equation we return ships with a bootstrap-structural certificate. We resample the training data 1,000 times, re-run the search on each resample, and record how often the same equation structure (not just coefficients) recurs.

>>> model = formulize.fit(df, target="y")
>>> model.confidence
{
  "structure_recurrence": 0.847,      # 847 / 1000 bootstraps found this structure
  "coefficient_ci": {
    "c_0": [3.09, 3.19],
    "c_1": [0.48, 0.52]
  },
  "provenance": {
    "training_window": "2015-01-01 to 2022-12-31",
    "n_bootstrap": 1000,
    "seed": 42,
    "features": ["x1", "x2"]
  }
}

Honest caveats

Nguyen is a solved leaderboard — everyone beats everyone on it now. It's a smoke test, not a moat.
Feynman recovery uses symbolic equivalence checking, not R² > 0.99. Cheaper metrics inflate numbers.
PySR is CPU-multicore; formulize is GPU. That's the point — but "3-8× faster" is a hardware + algorithm claim together, not algorithm alone.
SRBench is where we're still catching up in Q4 2026. We're at parity with PySR, not ahead. Working on it.

Full methodology honesty log

We publish a dated log of every inflation mechanism we've caught in our own methodology. If you find a NEW one, email us at honesty@formulize.dev and we'll add it, dated today. See the HONESTY.md file.

← Back to home