Repository navigation
Fix backpressure stall and row loss, strict-mode quote check, SWAR count; scan with Aemi kernels; gate CI - #1
Merged
Merged
Conversation
…tall nor drop rows decodeWithBackpressure parked its producer past the high-water mark and nothing ever woke it (recordConsumed had no caller): 3,200 of 20,000 rows, then a hang. Both streams used .bufferingNewest, which silently drops the oldest rows when the consumer is slower: 12,800 of 200,000 reached a slow consumer, and even a fast one lost rows in the benchmark. The producer Task also ignored stream termination. AsyncThrowingStream(unfolding:) runs the parser on the consumer's pull, so there is no queue, no controller and no background task. BackpressureController is removed; MemoryLimitConfiguration keeps its fields for source compatibility, and only batchSize still acts. Interleaved release A/B, 9 rounds, loaded machine: rows 304 -> 280 ms, batches 231 -> 238 ms (ranges overlap); the old stream delivered 97,369 of 100,000.
findByte flags the byte above a real match when it is 0x01 after the xor, so countNewlinesApprox counted an LF followed by 0x0B twice: 56 of 131 random inputs of length 0...130 disagreed with a byte-by-byte count. exactMatchMask carries no bit across a lane; Formal/Swar.lean proves it exact in all eight lanes (bv_decide) and proves the classic form over-reports. findByte keeps its cheaper form for the any-match and first-match callers, where it is exact, so the encoder hot path is unchanged.
deadwood 0.11.0 found SingleValueEncoder.allValues and reset, SIMDScanner.StructuralPosition.isDelimiter and the tab constant, and HardwareInfo.markdownSummary and jsonSummary unreferenced.
… test the optimized build The two SwiftStaticAnalysis jobs ran with continue-on-error and never failed a build; Scripts/analyzers.sh (the AemiJSON script, tools pinned by version and SHA-256, dolly against a baseline of 290 existing clones) replaces them. Scripts/formal.sh runs TLC on the backpressure protocol (the shipped-before design must still deadlock, its replacement must hold) and Lean on the SWAR proofs; each breaks when its model is broken. test-release runs swift test -c release --enable-testable-imports.
…, and scan fields with Aemi's kernels Strict mode rejects a quote in an unquoted field, but the 64-byte SIMD jump skipped the field's prefix without looking for quotes, so the check only worked in the last 64 bytes of the buffer: 20 rows of ab"cd,ef were accepted, 1 to 3 rows were rejected. The jump now also stops at a quote. The field scan is FieldScan: 16 bytes inline, then AemiKernels.firstIndexOfAny (NEON) or firstIndexOfByte (memchr). The previous scanner was scalar below 64 bytes. Hybrid probe, ns per field, previous scanner vs FieldScan: 2 B 4.4/4.0, 5 B 6.2/4.9, 10 B 9.8/7.1, 20 B 17.9/18.0, 40 B 34/22, 100 B 77/22, 400 B 310/36, 4,000 B 3,145/243; calling Aemi alone was 1.9x slower at 2 bytes, which is why the inline head stays. Benchmarks, interleaved release A/B, 7 rounds: Raw Parse 1M 87.3 -> 75.5 ms, 500-byte fields 22.5 -> 13.1, 10 KB fields 21.3 -> 3.8, 200 columns 6.0 -> 4.3, simple/complex/quoted/numeric/Unicode decodes 1.00-1.02x (ranges overlap). SIMDScanner.findNextStructural and findNextQuote are removed with their tests; FieldScanTests compares the scan against a byte-by-byte search at every length and position.
…then Aemi's kernel needsQuoting had two copies of the same loop (BufferedCSVWriter and SIMDScanner, which dolly reported) and the 64-byte path extracted its mask lane by lane. The probe, ns per call old / scalar-16 + Aemi / swar-64 + Aemi: 24 B 8.3 / 21.6 / 7.4, 40 B 9.5 / 20.4 / 8.7, 64 B 46.8 / 16.9 / 12.1, 400 B 306 / 37 / 30, 4,000 B 3,099 / 231 / 230; a 16-byte scalar head would have regressed 24-63 byte fields 2-3x, so the head is the SWAR test. Interleaved release A/B, 11 rounds: Encode 10K 14.8 -> 14.1 ms, 500-byte fields 23.1 -> 13.9, 100K sequential 130 -> 127, the other encode benchmarks 0.95-0.99x. FieldScanTests compares the new test with a byte-by-byte one at every length and position up to 140.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| run: | | ||
| sudo mv /tmp/swa /usr/local/bin/swa | ||
| sudo chmod +x /usr/local/bin/swa | ||
| curl -fsSL --retry 3 -o elan-init.sh https://raw.githubusercontent.com/leanprover/elan/227caca133724d5516bee25c2aeb3e609478f2d8/elan-init.sh |
Comment on lines
+55
to
+56
| curl -fsSL --retry 3 -o "$scratch/$name.tar.gz" \ | ||
| "https://github.com/g-cqd/$tool/releases/download/v$version/$name.tar.gz" \ |
Comment on lines
+29
to
+30
| curl -fsSL --retry 3 -o "$JAR.part" \ | ||
| "https://github.com/tlaplus/tlaplus/releases/download/v$TLA_VERSION/tla2tools.jar" \ |
swift-format from Swift 6.4 wants @testable imports sorted into one block; the Xcode 26.6 formatter that the Format Check job runs wants regular imports first. The pre-commit hook uses the newer one, so this commit skips it; CI is the gate.
At 50,000 rows and 20,000 iterations per row the debug run took over a minute and hit the time limit on CI.
…ock parallel test The slow-consumer tests decoded up to 100,000 rows and kept every core busy for about a second. On the 3-core CI runner that cut the parallel decode in the speed test to 0.2x of sequential (it is 1.31x there without them). A 100 KB row estimate puts the pause threshold at 8 rows, so 300 rows reach it; the four delivery tests still fail against the previous implementation.
Tens of thousands of #expect calls per test kept a core busy in the debug build, which the wall-clock parallel test on the 3-core runner feels.
Debug timing of the wall-clock parallel test changed because the sequential path got 5x faster there (387 -> 77 ms on the runner, probe branch output) while the parallel path did not (257 -> 196 ms). Release, where the number means something, is 55 vs 43 ms. CHANGELOG and .version for 1.0.1.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Correctness (each has a test that failed first)
decodeWithBackpressurehung past the high-water mark (3,200 of 20,000 rows, then nothing):recordConsumedhad no caller. TLC reproduces the deadlock (Formal/BackpressureOld.tla)..bufferingNewestand dropped rows for a slow consumer (12,800 of 200,000 delivered). They now pull rows on demand (AsyncThrowingStream(unfolding:));BackpressureControlleris removed.MemoryLimitConfigurationkeeps its fields; onlybatchSizeacts.countNewlinesApproxovercounted (SWAR borrow).Formal/Swar.leanproves the replacement exact.Performance (interleaved release A/B, no benchmark slower)
Raw parse 1M rows 87 -> 76 ms; 500-byte fields decode 22.5 -> 13.1 ms, encode 23.1 -> 13.9 ms; 10 KB fields 21.3 -> 3.8 ms. Field scanning is now
FieldScan: an inline head, thenAemiKernels.firstIndexOfAny/firstIndexOfByte. Aemi's kernel alone was 1.9x slower on 2-byte fields, which is why the head stays.CI
Gates on deadwood, dolly (baseline) and arcleak, re-checks the formal models (TLC + Lean), and runs the optimized test suite. Replaces the SwiftStaticAnalysis jobs that ran with
continue-on-error.Known: the five platform builds already fail on
main(-warnings-as-errorsvs-suppress-warningsunder Xcode 26.6); not touched here.AemiKernelsis now a dependency of theCSVCodertarget (it was a macro-only dependency).