Gene-Bench Report: Token Savings from Gene Reuse

This page publishes the measured results of the Gene-Bench v3 benchmark: on a 778-task common pool, Gemini + Gene saves 62.6% of tokens overall versus the bare Opus model. Every formula on this page shares its definition with the site-wide "Tokens Saved" statistics, governed by the savings-core spec (v0.3.0) and locked across Hub, Desktop and evox by golden conformance vectors.

Experiment setup

• Benchmark: Gene Bench v3 -- 808 tasks across 4 domains (math_reasoning / rule_following / agent_env_synth / code_generation); strict with_gene evaluation uses the 778-task common pool where both sides have complete assets
• Arms: bare Opus (no context assets) vs Gemini + evolved Gene (evolved-v3, distilled from trajectories the model itself solved and a verifier accepted)
• Accounting: full token counts -- input + output + thoughts; run v3_final_common778 (2026-04)
• Measurement scripts: eval/compare_gene_rollout_tokens.py / eval/compare_runs.py (Gene-Bench repo)

Formula 1: overall savings rate

savings = 1 − (Gemini+Gene tokens / bare-Opus tokens)
        = 1 − 182,943 / 489,273  ≈  62.6%

Formula 2: where the savings come from

total saved = ΔInput (Gene compresses the prompt) + ΔOutput (removes generation redundancy)
            =  88,125 (−42.4%)                    +  218,205 (−77.5%)

Output-side savings are ~2.5× the input-side savings -- the Gene's main value is cutting generation redundancy, not compressing the prompt.

Formula 3: rollout-folding savings

rollout savings = 1 − 1 / N(avg rollouts) = 1 − 1/1.48 ≈ 32.4%

Bare Opus needed 1.48 rollouts per task on average (retry on failure); the Gene folds this to one shot. This is the structural source of the savings: what disappears is not a shorter answer but entire retry rounds.

Formula 4: effective savings rate (failures removed)

effective savings = 1 − (Gemini tokens on solved tasks / Opus tokens on those tasks) = 52.8%

62.6% is the headline number -- it includes Gemini's "cheap failures" (wrong answers tend to generate less). 52.8% is the savings when the task is actually completed -- the more conservative, more honest reading.

Formula 5: maximum single-task savings

max single-task savings = (14,340 − 2,179) / 14,340 ≈ 84.8%   (typical code_generation case)

Intuition

savings = (N_rollout − 1) × avg cost per round  +  ΔT_structure (Gene's structured compression)

In plain words: drop the N−1 retry rounds, then save additional generation inside each round because the Gene makes the prompt precise.

How this relates to the site-wide statistics

Both bases belong to the savings-core spec (private repo, v0.3.0): constants and formulas are frozen by golden vectors, and the public Hub (Node), private Hub (Go), Desktop (Go), evox (Rust), deck (TS) and evolver (Node) implementations must reproduce the same vectors bit-for-bit, with a daily drift-check. The measured results on this page are the calibration target for a future revision of the estimate coefficients.

Caveats

1. The measured numbers come from one specific run (v3_final_common778); different model versions / task pools will vary.
2. When quoting across tasks, prefer 52.8% (effective); 62.6% includes cheap failures, and 84.8% is a single-task ceiling that must not be extrapolated.
3. Genes are distilled from the model's own verified successful trajectories (generation_source = evolved); evaluation uses sanitized Skills/Genes with no oracle leakage.