Do not classify a benchmark too fast to time (#240) - #254
Merged
Merged
Conversation
The first dispatches of layout-ab.yml named attachment_reread and headers_repeat_read layout-sensitive, at 14% and 15% "spread". Both run in tens of nanoseconds -- they exist to show that a cached read costs nothing, which is the whole point of them -- and at that scale the timer's own granularity is a large fraction of the measurement. Those percentages were quantisation, not placement. That mattered rather than being untidy: the table those two rows sat in is the one an x86 build-flag decision was about to be read off, and they were the two largest numbers on it. Benchmarks whose slowest salt is under a microsecond are now excluded from the classification and labelled "too fast to classify" rather than silently dropped -- a reader should see that the instrument declined to judge them, not wonder where they went. Two tests pin it: the real case that prompted this, and a microsecond-scale benchmark with a genuine spread, which must still be classified. CONTRIBUTING now records what the four dispatches measured -- up to 7.5% on a Xeon 6973P-C and 7.1% on an EPYC 9V74, with parse_message and parse_qp_message sensitive on every CPU tried -- and the alignment trial's outcome: -C llvm-args=-align-all-nofallthru-blocks=6 cuts those two to 0.7-1.8% at a 2-3% cost to quoted-printable decoding, with no consistent effect elsewhere, and is not adopted. Recording the negative result so the next person does not spend four dispatches rediscovering it. Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #240 / #249, from running the instrument for real.
The defect
The first dispatches of
layout-ab.ymlnamedattachment_rereadandheaders_repeat_readlayout-sensitive, at 14% and 15% spread. Both run in tens of nanoseconds — they exist to show that a cached read costs nothing, which is the point of them — and at that scale the timer's own granularity is a large fraction of the measurement. Those percentages were quantisation, not placement.That mattered rather than being untidy: the table those two rows sat in is the one an x86 build-flag decision was about to be read off, and they were the two largest numbers on it. An instrument that names its loudest result out of noise is worse than no instrument.
The fix
A benchmark whose slowest salt is under a microsecond is excluded from the classification and labelled
too fast to classify— labelled rather than silently dropped, so a reader can see the instrument declined to judge it instead of wondering where it went.Two tests pin it: the real case that prompted this (two sides 10 ns apart reporting +14%), and a microsecond-scale benchmark with a genuine 20% spread, which must still be classified. The second is the one that matters — a floor that swallowed real findings would be a worse bug than the one it fixes.
What the four dispatches measured
Recorded in
CONTRIBUTING.mdso it does not have to be rediscovered:parse_qp_messageandparse_messageare layout-sensitive on every CPU tried. The gate's threshold is 7%, so a verdict at or below that figure on those benchmarks is not evidence about the code — which is what three of this session's PRs ran into.The alignment trial is recorded as a negative result:
parse_messageparse_qp_messageparse_metadataparse_lazy_untouchedfull_read-C llvm-args=-align-all-nofallthru-blocks=6compiles on rustc 1.98.0 and cuts the two hot decode benchmarks to under 2% scatter — but has no consistent effect on the lazy or metadata paths, makesfull_read's spread slightly worse on both runs, and costs 2–3% on quoted-printable decoding. Not adopted, per #240's option (ii);.cargo/config.tomlis untouched and the shipped wheels are unchanged.I would still adopt it if you disagree with that reading — the gate pair becoming 4–6× more stable and faster is a real argument — but it changes every x86_64 wheel on the strength of a flag name LLVM can rename, and the evidence is consistent on two benchmarks out of five. That felt like yours to call rather than mine.
Checks run locally
pytest tests/test_layout_spread.py(11 passed),pytest tests --ignore=tests/benchmark,ruff check .. No Rust change; the extension is byte-identical.