Skip to content

test(sweep): Sourcify corpus sweep harness for the Slang frontend - #685

Open
nebasuke wants to merge 9 commits into
mainfrom
slang-sourcify-sweep
Open

test(sweep): Sourcify corpus sweep harness for the Slang frontend#685
nebasuke wants to merge 9 commits into
mainfrom
slang-sourcify-sweep

Conversation

@nebasuke

@nebasuke nebasuke commented Sep 2, 2026 •
edited by hedgar2017
Loading

Copy link
Copy Markdown
Member

Sourcify test for 0.8.34 and >= cancun used to previously test solx. Using this PR to get an idea of how much we can successfully compile at this point, and to find bugs.

Adds a CI sweep that compiles every verified Sourcify contract for solc 0.8.34 and 0.8.36 with evmVersion >= cancun (119,117 contracts) through solx and reports what fails and how. Each failure is re-run with solx 0.1.8, so a frontend gap (solx-fail) is told apart from input that no solx compiles (both-fail). The report ranks failures by signature with example contracts, which makes it a progress meter for the frontend now and a regression net later, once solx-fail is low enough to gate on.

  • Runs on every push to main, and on PR pushes while the PR carries ci:sourcify-sweep. On a PR the report is a sticky comment; on main it is the run summary.
  • Report-only: only harness breakage fails the run.
  • 24 shards, 12–18 minutes each on warm caches.
  • The latest run on this branch compiled 99.24% of the corpus (703 frontend-only failures, 206 shared with 0.1.8).
  • tests/sourcify-sweep/README.md covers local runs and the corpus pin.

@github-actions

github-actions Bot commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Coverage Summary

Crate Line Coverage Function Coverage
solx 🟢 83.6% 🔴 20.0%
solx-benchmark-converter 🔴 0.0% 🔴 0.0%
solx-codegen-evm 🟡 57.5% 🟡 51.5%
solx-compiler-downloader 🔴 0.0% 🔴 0.0%
solx-core 🔴 46.3% 🟡 54.5%
solx-dev 🔴 3.2% 🔴 4.2%
solx-mlir 🟡 55.5% 🟡 50.9%
solx-slang 🔴 24.7% 🔴 30.5%
solx-solc-test-adapter 🔴 1.7% 🔴 2.1%
solx-standard-json 🔴 40.7% 🔴 44.6%
solx-tester 🔴 36.6% 🔴 34.6%
solx-utils 🔴 35.9% 🔴 39.8%
Total 🔴 13.2% 🔴 16.4%

Codecov Report | HTML Report | Workflow Run

@nebasuke nebasuke added the ci:sourcify-sweep Run Sourcify corpus on this PR label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Sourcify sweep (Slang frontend vs solx 0.1.8)

  • solx: solx v0.1.8, LLVM-based Solidity compiler for the EVM, Front end: Slang, LLVM build: e7c67b950b32c984e9ae6d8e46839a0227a5e3ae
  • baseline: solx v0.1.8, LLVM-based Solidity compiler for the EVM, Front end: solc, LLVM build: 7d0702e169889fe4f1a2241c57bef7d2c1c68737
  • contracts: 119117
outcome count share meaning
ok 118184 99.22% compiles: no errors and bytecode for the target source
solx-fail 744 0.62% candidate fails, baseline compiles (or no baseline given)
both-fail 189 0.16% candidate and baseline both fail: corpus artifact or documented solx limitation

candidate wall time over ok contracts: median 0.19s, p90 0.60s, p99 2.24s, max 24.48s, total 37640s

Outcome by corpus trait

trait contracts ok solx-fail both-fail timeout
single source 66463 66292 (100%) 115 (0%) 56 (0%) 0 (0%)
multiple sources 52654 51892 (99%) 629 (1%) 133 (0%) 0 (0%)
remappings 11888 11489 (97%) 308 (3%) 91 (1%) 0 (0%)
viaIR 5957 5650 (95%) 217 (4%) 90 (2%) 0 (0%)
mentions assembly 33921 33038 (97%) 696 (2%) 187 (1%) 0 (0%)
explicit evmVersion 17028 16513 (97%) 408 (2%) 107 (1%) 0 (0%)

Candidate-only failures (744)

kinds: error 690, panic 51, abort 3

count kind signature examples
527 error This contract cannot be compiled due to a combination of a memory-unsafe assembly block and a stack-too-deep error. solx can automatically fix the stack-too-dee 100_0xdbdbbbb35a21e6a52b63ba942026e92ddb1e0ff2, 100_0xf5e903c0cbf8d82349fc8c0f160380a44b0f42d8, 10_0x0000043fc6f4c10a667298279f7029e1c3900000
62 error MLIR translation 100_0x13d9c35f4901491ac4ffebd3100238bbffbd7181, 100_0x5c4f2eacbdc1eb38f839bddd7620e250a36819d4, 100_0xcdef0a216fcef809258aa4f341db1a5ab296ea72
34 error Member 'setAddr' not found or not visible after argument-dependent lookup. 97_0x116d755bd61f7d406ea2df965720175c36bb6485, 97_0x2107e7f2f9957598aab9ecb676dbbba1abff3e1a, 97_0x21b6293bd1bf1d4f7e2c998ac8fe5392d8868f84
34 panic the binder types every expression @ src/type.rs 10_0xd0690557600eb8be8391d1d97346e2aab5300d5f, 11155111_0x04962f8b2c0ab0864996a6abfb34f12b827f052f, 11155111_0x057aeeeaa7135898aa2eb5ca3bfb8ccb08e43d2d
29 error worker crash: Assertion `Index < Length && "Invalid index!"' failed 137_0x09a747cd58ca52fd71746a81391cd7cafad90a67, 137_0x0ac79f4b335a16df57ba3561cfb47ba8c29bdd6e, 137_0x0b88bcb0266898c2671aa4dbb4597fa2d6a45b51
20 error No matching declaration found after argument-dependent lookup. 11155111_0x5d1927cfd1d9e9cecdbd68d121312055079d41e2, 11155111_0x809b86dd2e0d6449f6e8bafb1b6122b7124aea39, 11155111_0xbe8a300f661d958bd7ea1f1e07cc93469f267319
9 panic fixed array size fits u64: Overflow(N, N, N) @ src/type.rs 42161_0x1e53209eb4099988b106be22edb29192212ad8b7, 42161_0x2f49bcf6ae5f77fb8b4823b27579b63df2d2353b, 42161_0x50eb3d05d2c463949de9238d419385594f7adb97
6 panic not implemented: unsupported member access: getConversionRate @ expression/member.rs 11155111_0x35f798cb977a5ad1b6c434efd9b9ba6e5f7ab54d, 11155111_0x55a06e39523e8faace4ee655355087e369be4e84, 11155111_0x74fb49b38fb40d105752a068b16cad17d3ea92da
5 error Member 'clone' not found or not visible after argument-dependent lookup. 1_0x4845c9d8c05fe6e2e4aaefff958106b6f6158795, 1_0xde6cacaa399cad26d6fee8fa7dc478545fe48150, 8453_0x75c487b4d9039a018217fe3041cb4f4489e7fb1d
4 error Member 'interfaceId' not found or not visible after argument-dependent lookup. 137_0x3240589a1f866cb0db2f0ca65090983b75034d3a, 56_0x7ff77a23d42a3ff80358f58138a93d6bcfac15a2, 8453_0x342585edfbc856f47673c1e407dd5661a4b32bcf
3 abort Assertion `isaLLVM::LLVMStructType(addr.getType())' failed 11155111_0xf623d0ab080d753e3876b72057b415452ea5f2d2, 11155111_0xfbacb46ab55eb2d0beea1276d5d401379f3e6d1f, 11155111_0xfbf641397b210a43ca67e59ebaa4856a56881b9f
3 error LLVM error: Stackification failed for '_placeInLevelMatrix(address,address,uint8)_3165' function. It is recursive and has stack too deep errors. Consider refact 56_0xe45fc295347ccf7dab32347be4bd37d13a0a93be, 56_0xfdaba2bdb4aef7e6bc413fbc3e9800e8b7b8e933, 97_0xbf9c141a7db849a62be61e1191fb3725413ae581
2 error LLVM error: Stackification failed for '_placeInLevelMatrix(address,address,uint8)_3640' function. It is recursive and has stack too deep errors. Consider refact 56_0xd365458623e766304ffdef2eab590c363a2948d6, 97_0x1a9ea54ad1cf25ee0489c0c9996d5d8db166f524
2 error No unique declaration found for 'initialize'. 421614_0x1cf7c8e232cee49c592487986c5fdbdac2dbadde, 42161_0x5bcb52965004561ed4d78366a36d0b1b11f6cca4
2 panic index out of bounds: the len is N but the index is N @ expression/mod.rs 11155111_0x342e4f69ee7dc4f88d52f425d52e88053c614e2c, 11155111_0x54dff0a992fefd8595862d615821253d59e91e87
1 error LLVM error: Stackification failed for '_releaseMatrixIncome(address,uint8)_3049' function. It is recursive and has stack too deep errors. Consider refactoring i 56_0xcee18bb5958317b226ca08e7ef264c99dca56f58
1 error Member 'allow' not found or not visible after argument-dependent lookup. 11155111_0x143e12035c5ce6642c8f444732b2b285f18bed71

Failures shared with the baseline (baseline diagnostics) (189)

kinds: error 189

count kind signature examples
174 error runtime code LLVM IR generator: This contract cannot be compiled due to a combination of a memory-unsafe assembly block and a stack-too-deep error. 100_0x239d413a6ac5322d3ccaaaf43e34045bdacd7e74, 100_0x27ebdb6cefd590feaf79b20f6e77bf79dc3c04fe, 100_0x325afb837204d46a3d4158ded26a8be2681761b5
7 error LLVM error: Stackification failed for '_createAndPlaceSlot_rt_547' function. It is recursive and has stack too deep errors. Consider refactoring it to use a non 97_0x8a5577fbd87bae4551b275f409d3285df69836be, 97_0x9e2e5aa409d9d6db58e206c52a83e7513e9d22ac, 97_0xa0dcfd55389c6aece6ded5130c0df195f36191c8
5 error Undeclared identifier. 11155111_0x0dcf68b5416eaaf96e896809a33c404f30a06c7c, 11155111_0x74908cc7dc1416fc57fd4ef65d849020b18253b8, 11155111_0xc1335a802c09380d26202f0178464c3ef4e48fce
1 error LLVM error: Stackification failed for 'fun__placeInLevelMatrix_2414' function. It is recursive and has stack too deep errors. Consider refactoring it to use a n 11155111_0x3a37206d0b789561e9c8bd5151ece8a8318b4fe3
1 error LLVM error: Stackification failed for 'fun__placeInLevelMatrix_2778' function. It is recursive and has stack too deep errors. Consider refactoring it to use a n 97_0x6183dc1cfe56178bbb698c1db4b5aed3aabf8720
1 error Member "UnexpectedInitSelector" not found or not visible after argument-dependent lookup in type(contract ParkERC1967Proxy). 56_0x69d890f783a6662e8554c8f71d1daeda1baddda2

@nebasuke

nebasuke commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

Run 1 (d35b135 = #671 + harness, run 33651331011): 66,142 / 112,587 ok (58.75%), 46,320 Slang-only failures, 125 shared with 0.1.8 (the known stack-too-deep gates), 0 timeouts. Single-file contracts pass at 96%, multi-file at 10%.

Seven signatures cover 99.4% of the failures (buckets are first-failure only, so fixing the top one will unmask more of the lower rows):

count cause pointer
26,286 contracts with a contract base are skipped without a diagnostic: exit 0, target missing from the output solx-slang/src/source_unit.rs inheritance TODO. Needs a loud error in the meantime.
371 same gap via factories: new Token() where Token inherits, so lld reports an undefined __datasize__$<hash>$__ goes away with the above
10,457 bytes memory indexing lowered as sol.fixed_bytes_index; verifier rejects !sol.string<Memory> index_access.rs, Type::ByteArray arm
7,541 non-relative imports, i.e. remappings not applied #684, merged into this branch for run 2
1,334 URL-style source names: Path::components() collapses //, so ../utils/Context.sol never matches its key (surfaces as "Identifier already declared") compilation_config.rs resolve_import; #684's tip commit may already cover it
151 LLVM abort Unexpected data location cast Sol dialect, solx-llvm
45 payable(x).transfer(uintN) / send without widening to 256 bits transfer/send lowering
42 slang binder panic, index out of bounds in p5_resolve_references/resolution.rs upstream slang rev 8948967

Long tail (≤ 27 each): LLVM worker assertion Index < Length, call{value:} in a value position, using L for T member calls, sol.and/sol.xor operand type mismatch.

The first report's signatures hid most of this (Sol pass pipeline failed for the verifier errors, one bucket per __datasize__ hash), so de79e8b moves bucketing into report.py over the recorded diagnostics; run 2's comment will show these buckets directly, with example contract ids.

@nebasuke nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 2, 2026
@nebasuke nebasuke mentioned this pull request Sep 2, 2026
Closed
@abinavpp
abinavpp force-pushed the app-slang-inline-assembly branch 2 times, most recently from 2d34148 to a98f2e6 Compare September 4, 2026 08:07
@abinavpp
abinavpp force-pushed the app-slang-inline-assembly branch 2 times, most recently from c237244 to 6d41503 Compare September 16, 2026 13:18
@hedgar2017
hedgar2017 force-pushed the app-slang-inline-assembly branch from 6d41503 to 96cebda Compare September 16, 2026 16:03
@abinavpp
abinavpp force-pushed the app-slang-inline-assembly branch 3 times, most recently from ed6c4df to 31ee803 Compare September 17, 2026 12:29
Base automatically changed from app-slang-inline-assembly to main September 17, 2026 19:59
@nebasuke
nebasuke force-pushed the slang-sourcify-sweep branch from 69d842a to b123248 Compare September 20, 2026 12:39
@nebasuke
nebasuke changed the base branch from main to az-slang-modifiers September 20, 2026 12:39
@nebasuke nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR ci:compile-benchmark Run hyperfine compilation benchmark for this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 20, 2026
@nebasuke
nebasuke force-pushed the slang-sourcify-sweep branch from 88b9e22 to 3dc23b2 Compare September 20, 2026 14:41
@nebasuke nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 20, 2026
This was referenced Sep 20, 2026
Merged
Merged
@hedgar2017
hedgar2017 force-pushed the az-slang-modifiers branch 7 times, most recently from 175a65e to d4479be Compare September 28, 2026 13:18
Base automatically changed from az-slang-modifiers to main September 28, 2026 14:42
@nebasuke
nebasuke force-pushed the slang-sourcify-sweep branch from 3dc23b2 to fa0180e Compare September 30, 2026 12:22
@nebasuke nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:compile-benchmark Run hyperfine compilation benchmark for this PR ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 30, 2026
@nebasuke
test(sweep): Sourcify corpus sweep harness for the Slang frontend
d6c5437 
Compiles the pinned Sourcify corpus (112,587 verified solc 0.8.34 contracts,
evmVersion >= cancun) through `solx --standard-json` and reports the outcome
table plus a ranked failure census. Each corpus record becomes one
standard-JSON input with sources inline and evmVersion, libraries,
remappings and viaIR passed through verbatim; candidate failures are re-run
with the released solc-frontend solx so they split into candidate-only and
both-fail. Failure kinds (error, panic, abort, no-bytecode, timeout) are
recorded separately because a frontend crash ends the whole standard-JSON
call.

CI: `.github/workflows/sourcify-sweep.yaml`, label-gated on
`ci:sourcify-sweep` or workflow_dispatch, 24 hosted shards that each extract
only their slice of the corpus, summarize job posts the report as a sticky
PR comment. Report-only. Ten corpus records are committed as fixtures so the
harness runs offline against the workspace build.
@nebasuke
test(sweep): bucket failures in report.py from recorded diagnostics
808a858 
The first full run showed the recorded first-line signature is too coarse:
`Sol pass pipeline failed` hid the MLIR verifier message in stderr,
`__datasize__$<hash>$__` link errors split one bug into hundreds of buckets,
and the shared-failure census showed the candidate's kind instead of the
baseline diagnostic that explains the contract. Bucketing now lives in
report.py (`refine`) over the raw material run.py records, so old results
re-render: MLIR verifier and worker-crash lines are pulled from stderr,
import failures split into non-relative / URL-style / relative, panics carry
their source location, hashes and literal `\n` are normalised. run.py records
which contracts did come out on a no-bytecode result.
@nebasuke
test(sweep): rewrite version pragmas to each leg's own language version
2d0925f 
The Slang frontend checks pragmas against Slang's latest version (0.8.36)
while the baseline is a 0.8.34 compiler, so any exact pin fails one leg
before parsing and the outcome classification is lost. With
--rewrite-pragmas each leg compiles the source with every version pragma
replaced by the version its own --version reports; the corpus stays the
verbatim Sourcify record. Pragmas have no effect on codegen.
@nebasuke
test(sweep): pin the combined solc 0.8.34 + 0.8.36 Sourcify corpus
1fee9e8 
Sourcify holds only 9,296 solc 0.8.36 compilations, too few for a census
on their own, so the corpus is the 0.8.34 release (byte-identical half)
plus the 6,530 unique 0.8.36 contracts, extracted with the same query and
layout. The frontend's 0.8.36 language version is handled by the per-leg
pragma rewrite, so the records stay verbatim.
@nebasuke
ci(sweep): build with plain cargo build after solc pipeline removal
052ead9 
#751 removed the build-slang alias and build-toolchain's solc input; the default build is the Slang pipeline now.
@nebasuke
nebasuke force-pushed the slang-sourcify-sweep branch from fa0180e to 052ead9 Compare September 30, 2026 22:57
@nebasuke nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 30, 2026
@nebasuke
ci(sweep): run on main pushes and labelled PR pushes
cbaa7bd 
A full sweep is cheap enough to run per merge, so main gets a census for
every push instead of only when someone labels a PR. On a PR the label
now keeps the sweep running on each push rather than needing a re-label.
Main runs are not cancelled by the next merge so each one is attributable.
@nebasuke
nebasuke force-pushed the slang-sourcify-sweep branch from cbaa7bd to 97f479e Compare October 4, 2026 12:44
@nebasuke
fix(sweep): keep the pragma rewrite out of comment text
b10aaba 
The pattern matched any text up to the next `;`, so a comment mentioning
"pragma solidity" lost the code after it, often an import. Both legs saw
the damaged source and 17 verified contracts landed in both-fail with
`Undeclared identifier`, hiding a frontend gap. Matching only
version-expression characters leaves those comments alone; real
pragmas, including ones split across lines, are still rewritten.
@nebasuke
fix(sweep): render the report on Python before 3.12
1046d39 
A backslash inside an f-string expression is a syntax error before
PEP 701, so report.py did not load on older local interpreters.
@nebasuke
ci(sweep): give each main push its own concurrency group
3a1e869 
GitHub keeps one pending run per group, so with a shared main group a
third push replaced the queued run and that commit got no census. The
shard timing comment now matches measured runs.
@nebasuke
nebasuke force-pushed the slang-sourcify-sweep branch from 97f479e to 3a1e869 Compare October 4, 2026 13:26
This was referenced Oct 4, 2026
Draft
Open
@nebasuke
nebasuke marked this pull request as ready for review October 4, 2026 14:21
@nebasuke
nebasuke requested a review from a team October 4, 2026 14:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ci:sourcify-sweep Run Sourcify corpus on this PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nebasuke