Skip to content

Do not classify a benchmark too fast to time (#240) - #254

Merged
kurok merged 1 commit into
masterfrom
fix/layout-spread-tiny-benchmarks
Sep 17, 2026
Merged

kurok merged 1 commit into
masterfrom
fix/layout-spread-tiny-benchmarks

Conversation

@kurok

@kurok kurok commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #240 / #249, from running the instrument for real.

The defect

The first dispatches of layout-ab.yml named attachment_reread and headers_repeat_read layout-sensitive, at 14% and 15% spread. Both run in tens of nanoseconds — they exist to show that a cached read costs nothing, which is the point of them — and at that scale the timer's own granularity is a large fraction of the measurement. Those percentages were quantisation, not placement.

That mattered rather than being untidy: the table those two rows sat in is the one an x86 build-flag decision was about to be read off, and they were the two largest numbers on it. An instrument that names its loudest result out of noise is worse than no instrument.

The fix

A benchmark whose slowest salt is under a microsecond is excluded from the classification and labelled too fast to classify — labelled rather than silently dropped, so a reader can see the instrument declined to judge it instead of wondering where it went.

Two tests pin it: the real case that prompted this (two sides 10 ns apart reporting +14%), and a microsecond-scale benchmark with a genuine 20% spread, which must still be classified. The second is the one that matters — a floor that swallowed real findings would be a worse bug than the one it fixes.

What the four dispatches measured

Recorded in CONTRIBUTING.md so it does not have to be rediscovered:

CPU worst spread
plain Intel Xeon 6973P-C 7.5%
plain AMD EPYC 9V74 7.1%

parse_qp_message and parse_message are layout-sensitive on every CPU tried. The gate's threshold is 7%, so a verdict at or below that figure on those benchmarks is not evidence about the code — which is what three of this session's PRs ran into.

The alignment trial is recorded as a negative result:

Benchmark plain spread aligned spread cost
parse_message 3.3% / 4.8% 1.2% / 0.8% −1.8% / −1.2%
parse_qp_message 3.6% / 5.0% 0.7% / 1.8% +2.3% / +2.9%
parse_metadata 2.4% / 5.5% 2.1% / 6.4% −2.1% / −3.3%
parse_lazy_untouched 6.0% / 3.0% 5.1% / 6.7% +1.0% / −3.2%
full_read 1.2% / 3.3% 2.6% / 3.6% −1.7% / −1.0%

-C llvm-args=-align-all-nofallthru-blocks=6 compiles on rustc 1.98.0 and cuts the two hot decode benchmarks to under 2% scatter — but has no consistent effect on the lazy or metadata paths, makes full_read's spread slightly worse on both runs, and costs 2–3% on quoted-printable decoding. Not adopted, per #240's option (ii); .cargo/config.toml is untouched and the shipped wheels are unchanged.

I would still adopt it if you disagree with that reading — the gate pair becoming 4–6× more stable and faster is a real argument — but it changes every x86_64 wheel on the strength of a flag name LLVM can rename, and the evidence is consistent on two benchmarks out of five. That felt like yours to call rather than mine.

Checks run locally

pytest tests/test_layout_spread.py (11 passed), pytest tests --ignore=tests/benchmark, ruff check .. No Rust change; the extension is byte-identical.

The first dispatches of layout-ab.yml named attachment_reread and
headers_repeat_read layout-sensitive, at 14% and 15% "spread". Both run in
tens of nanoseconds -- they exist to show that a cached read costs nothing,
which is the whole point of them -- and at that scale the timer's own
granularity is a large fraction of the measurement. Those percentages were
quantisation, not placement.

That mattered rather than being untidy: the table those two rows sat in is
the one an x86 build-flag decision was about to be read off, and they were
the two largest numbers on it.

Benchmarks whose slowest salt is under a microsecond are now excluded from
the classification and labelled "too fast to classify" rather than silently
dropped -- a reader should see that the instrument declined to judge them,
not wonder where they went. Two tests pin it: the real case that prompted
this, and a microsecond-scale benchmark with a genuine spread, which must
still be classified.

CONTRIBUTING now records what the four dispatches measured -- up to 7.5% on
a Xeon 6973P-C and 7.1% on an EPYC 9V74, with parse_message and
parse_qp_message sensitive on every CPU tried -- and the alignment trial's
outcome: -C llvm-args=-align-all-nofallthru-blocks=6 cuts those two to
0.7-1.8% at a 2-3% cost to quoted-printable decoding, with no consistent
effect elsewhere, and is not adopted. Recording the negative result so the
next person does not spend four dispatches rediscovering it.

Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
@kurok
kurok merged commit 370893f into master Sep 17, 2026
15 checks passed
@kurok
kurok deleted the fix/layout-spread-tiny-benchmarks branch September 17, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant