Skip to content

7.0: managed ZIP writer, parallel record writes, checksum verification, and Excel-conformance fixes - #141

Merged
GabrielMarquezMatte merged 43 commits into
masterfrom
develop
Oct 14, 2026
Merged

GabrielMarquezMatte merged 43 commits into
masterfrom
develop

Conversation

@GabrielMarquezMatte

Copy link
Copy Markdown
Owner

What does this change?

Everything on develop since #140, completing the candidate for 7.0.0: 41 commits, 106 files.
The write path gets its own ZIP container and deflate encoder, records can be written on several
cores, the reader verifies ZIP checksums, and three bug hunts against real Excel fixed 17 cases
where the library produced or read something Excel disagrees with. The v6 → v7 port, including the
behavior changes below, is in docs/guide/migrating-to-v7.md.

The C ABI keeps every signature and XL_ABI_VERSION stays 6. xl_open_options gains one field in
what was tail padding, so its size is unchanged.

Managed ZIP writer

  • XLSX and XLSB packages are written by Writer/Zip/ZipPackageWriter instead of ZipArchive. It
    writes the same layout (patched local headers on seekable output, data descriptors otherwise, Zip64
    only when a field overflows).
  • CompressionLevel.Fastest, the default, uses a managed fixed-Huffman deflate encoder with a
    stored-block fallback and a CRC-32 that folds with PCLMULQDQ. Optimal and SmallestSize keep zlib
    inside the same container.
  • 50,000 rows by four columns into a MemoryStream (Ryzen 7 5700X): XLSX 10.43 → 9.11 ms, XLSB
    7.66 → 5.42 ms; with PrefetchWrite, XLSB 5.37 → 3.08 ms. File sizes stay within 0.2%.
  • A write that faults still leaves a central directory for the entries that completed. A workbook
    torn down mid-row with PrefetchWrite no longer risks a crash: the background writer is stopped
    before the package is abandoned.
  • New deflate and deflate-segments fuzz targets check the encoder against two inflaters.

Parallel record writes (new API)

  • WriteRecordsParallelAsync and WriteTableParallelAsync on XlsxSheetWriter and
    XlsbSheetWriter, with ParallelWriteOptions (DegreeOfParallelism, BatchRows). Records are cut
    into batches; each batch is formatted and compressed on a pool thread into a self-contained deflate
    segment, and segments are appended to the sheet in order.
  • 1,000,000 rows by four columns at Fastest: XLSX 187 ms serial, 50 ms with 4 batches in flight,
    36 ms with 8; XLSB 124 / 41 / 28 ms. The file grows by 0.18% (XLSX) or 0.04% (XLSB).
  • Memory is about 3 MB per batch in flight, whatever the sheet's size.
  • The row delegate runs on several threads at once. A failure or a cancellation ends the sheet and
    its workbook. With UseSharedStrings, string indexes depend on thread timing, and many distinct
    strings (tens of thousands) limit the speed-up.

ZIP checksums on read (behavior change)

  • Each ZIP entry of an XLSX or XLSB package is checked against its CRC-32. A mismatch throws
    InvalidDataException when the entry's last bytes are read. v6 checked only the entry's size, so
    a damaged sheet could read as plausible but wrong cells.
  • ExcelReaderOptions.VerifyZipChecksums (default true) turns it off; the bindings expose it as
    verify_zip_checksums.
  • It costs 2% to 4% of a read, and under 1% with PrefetchDecompression, where it runs on the
    background thread.

XLS writer

  • Numbers that fit are written as RK and MULRK records, and long text goes through a shared-string
    table, so it is stored once.
  • The worksheet substream follows the row-block layout Excel expects, with conformance tests on the
    record structure.

Fixes found by comparing against Excel 16

Readers

  • XLSX text with a line break reads with \n, not \r\n. Excel stores the break as a CR LF pair in
    the XML; XLSB and XLS already returned \n.
  • An XLSX formula whose text result begins or ends with whitespace no longer reads as empty
    (<v xml:space="preserve">).
  • An XLS formula filled down or entered as an array no longer loses the text result of its first cell.

Writers (each of these produced a file Excel refused or altered)

  • Sheet names with a leading or trailing apostrophe or a control character, column settings past the
    format's last column, a column width above 255, more than 200 distinct number formats and a number
    format longer than 255 characters now throw.
  • In XLSX, a carriage return, U+FFFE and U+FFFF in cell text are written escaped.
  • An XLSB sheet stops at 1,048,576 rows; it accepted one more.
  • Write<T> with a type that has no numeric form throws ArgumentException in every writer; XLSB
    and XLS threw InvalidCastException.

Typed parser

  • A numeric 0 or 1 binds to bool in XLSB and XLS, as it did in XLSX. A 0/1 flag column read from
    those formats came back all false.
  • TimeOnly binds from a text cell in every format, and numeric cells in XLSB and XLS bind to
    TimeSpan, other parsable types and enums as they do in XLSX.

Bindings

  • Timestamp text with Z or an offset no longer shifts by the machine's time zone when a CSV column
    is read as a timestamp.
  • Python: an int outside the signed 64-bit range raises OverflowError instead of being written
    as its low 64 bits, and Cell.as_date is right for January and February 1900.

Other

  • ExcelLimitExceededException.Message no longer says "bytes" for limits counted in rows, columns
    or entries.
  • The static PGO profile for NativeAOT is regenerated.

How was it verified?

  • Every writer fix was confirmed by opening the output in Excel 16 through COM; the reader fixes by
    comparing the library's cells with Excel's own values for workbooks Excel saved in XLSX, XLSB and
    XLS, in both date systems.
  • Parallel reads were compared with sequential reads on seven 60,000-row files at six degrees of
    parallelism; parallel writes were opened in Excel at 1,000,000 rows and checked with Python's
    zipfile.testzip().
  • The Python and Rust suites pass against a freshly published native library.

Known gaps

  • The C++ binding was not built or run locally; CI covers it.
  • The read tables in docs/performance/benchmarks.md predate checksum verification and are 2% to 4%
    optimistic for XLSX and XLSB.
  • The PGO trainer covers reads only, so the new write path has no static profile under NativeAOT.
  • Native error messages still carry resource keys (Arg_ParamName_Name) after the text.
  • Parallel record writes are not exposed through the C ABI.

GabrielMarquezMatte and others added 30 commits October 12, 2026 23:16
- Added new BIFF records for improved handling of Excel features, including LabelSst, CalcMode, and Print settings.
- Introduced XlsSharedStrings class to manage shared strings efficiently, ensuring long text is stored only once.
- Enhanced XlsSheetLayout to correctly structure worksheet substreams, including row blocks and cell data.
- Updated XlsWorkbookWriter to integrate shared strings and manage workbook size dynamically.
- Added comprehensive conformance tests to validate the structure of generated XLS files against Excel's requirements.
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…h reaches the output

Matches what ZipArchive.Dispose does on a faulted archive, which the writer failure tests pin.
AbandonAsync writes the directory with WriteAsync, seeks to the counted end first on seekable output, and releases the open entry inside the best-effort block.
A workbook disposed with a row left open abandoned the archive while the
PrefetchWrite consumer was still compressing into the open entry, so the
encoder's pooled buffers were returned under it (AccessViolationException).
The sheet's open-row guard throws before the sheet faults, so nothing joined
the consumer. AbandonZip now faults the active sheet first, which disposes
the offload stream and waits for its consumer.
…endian

- DeflateEncoder throws ObjectDisposedException instead of reading through
  unchecked refs into arrays it already returned to the pool.
- ZipEntryStream feeds one Write in pieces of at most 1 MB, flushing between
  them, so a huge write no longer holds its whole compressed output.
- The managed encoder is used only on little-endian; Fastest goes through
  zlib elsewhere.
- Drop the unused ZipEntryWriter.Flush and a computed-branch ternary.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clamp BatchRows to the row limit and build the runner inside the fault
path, so a failed rent no longer leaves the sheet stuck with an active
row. Guard AppendSegmentAsync against an ended sheet, pre-size detached
row buffers, replace call-branch ternaries, and document that a failed
or cancelled parallel call ends the sheet and its workbook.
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 93.21368% with 121 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.04%. Comparing base (bcc628f) to head (6c46480).

Files with missing lines Patch % Lines
...eader.Core/Writer/ParallelSheetWriterExtensions.cs 50.00% 24 Missing ⚠️
src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs 91.54% 15 Missing and 3 partials ⚠️
src/ExcelReader.Core/Writer/Zip/Crc32.cs 91.73% 9 Missing and 1 partial ⚠️
...rc/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs 91.17% 6 Missing and 3 partials ⚠️
src/ExcelReader.Core/Writer/Zip/BufferSink.cs 55.55% 8 Missing ⚠️
...lReader.Core/Writer/Internal/WriteOffloadStream.cs 64.70% 4 Missing and 2 partials ⚠️
...rc/ExcelReader.Core/Writer/Xls/XlsSharedStrings.cs 95.32% 1 Missing and 4 partials ⚠️
...rc/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs 93.15% 4 Missing and 1 partial ⚠️
.../ExcelReader.Core/Writer/Internal/CellFormatter.cs 94.20% 2 Missing and 2 partials ⚠️
...eader.Core/Writer/Parallel/ParallelRecordWriter.cs 91.48% 3 Missing and 1 partial ⚠️
... and 10 more
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #141      +/-   ##
==========================================
+ Coverage   90.58%   91.04%   +0.45%     
==========================================
  Files         232      251      +19     
  Lines       15981    17472    +1491     
  Branches     2934     3153     +219     
==========================================
+ Hits        14477    15907    +1430     
- Misses       1016     1073      +57     
- Partials      488      492       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GabrielMarquezMatte
GabrielMarquezMatte merged commit 114f5ae into master Oct 14, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants