Repository navigation
7.0: managed ZIP writer, parallel record writes, checksum verification, and Excel-conformance fixes - #141
Merged
Conversation
- Added new BIFF records for improved handling of Excel features, including LabelSst, CalcMode, and Print settings. - Introduced XlsSharedStrings class to manage shared strings efficiently, ensuring long text is stored only once. - Enhanced XlsSheetLayout to correctly structure worksheet substreams, including row blocks and cell data. - Updated XlsWorkbookWriter to integrate shared strings and manage workbook size dynamically. - Added comprehensive conformance tests to validate the structure of generated XLS files against Excel's requirements.
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…h reaches the output Matches what ZipArchive.Dispose does on a faulted archive, which the writer failure tests pin.
AbandonAsync writes the directory with WriteAsync, seeks to the counted end first on seekable output, and releases the open entry inside the best-effort block.
A workbook disposed with a row left open abandoned the archive while the PrefetchWrite consumer was still compressing into the open entry, so the encoder's pooled buffers were returned under it (AccessViolationException). The sheet's open-row guard throws before the sheet faults, so nothing joined the consumer. AbandonZip now faults the active sheet first, which disposes the offload stream and waits for its consumer.
…endian - DeflateEncoder throws ObjectDisposedException instead of reading through unchecked refs into arrays it already returned to the pool. - ZipEntryStream feeds one Write in pieces of at most 1 MB, flushing between them, so a huge write no longer holds its whole compressed output. - The managed encoder is used only on little-endian; Fastest goes through zlib elsewhere. - Drop the unused ZipEntryWriter.Flush and a computed-branch ternary.
…ter to handle edge cases
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clamp BatchRows to the row limit and build the runner inside the fault path, so a failed rent no longer leaves the sheet stuck with an active row. Guard AppendSegmentAsync against an ended sheet, pre-size detached row buffers, replace call-branch ternaries, and document that a failed or cancelled parallel call ends the sheet and its workbook.
…r buffers Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… XLS; TimeOnly from text
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #141 +/- ##
==========================================
+ Coverage 90.58% 91.04% +0.45%
==========================================
Files 232 251 +19
Lines 15981 17472 +1491
Branches 2934 3153 +219
==========================================
+ Hits 14477 15907 +1430
- Misses 1016 1073 +57
- Partials 488 492 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Everything on
developsince #140, completing the candidate for 7.0.0: 41 commits, 106 files.The write path gets its own ZIP container and deflate encoder, records can be written on several
cores, the reader verifies ZIP checksums, and three bug hunts against real Excel fixed 17 cases
where the library produced or read something Excel disagrees with. The v6 → v7 port, including the
behavior changes below, is in
docs/guide/migrating-to-v7.md.The C ABI keeps every signature and
XL_ABI_VERSIONstays 6.xl_open_optionsgains one field inwhat was tail padding, so its size is unchanged.
Managed ZIP writer
Writer/Zip/ZipPackageWriterinstead ofZipArchive. Itwrites the same layout (patched local headers on seekable output, data descriptors otherwise, Zip64
only when a field overflows).
CompressionLevel.Fastest, the default, uses a managed fixed-Huffman deflate encoder with astored-block fallback and a CRC-32 that folds with PCLMULQDQ.
OptimalandSmallestSizekeep zlibinside the same container.
MemoryStream(Ryzen 7 5700X): XLSX 10.43 → 9.11 ms, XLSB7.66 → 5.42 ms; with
PrefetchWrite, XLSB 5.37 → 3.08 ms. File sizes stay within 0.2%.torn down mid-row with
PrefetchWriteno longer risks a crash: the background writer is stoppedbefore the package is abandoned.
deflateanddeflate-segmentsfuzz targets check the encoder against two inflaters.Parallel record writes (new API)
WriteRecordsParallelAsyncandWriteTableParallelAsynconXlsxSheetWriterandXlsbSheetWriter, withParallelWriteOptions(DegreeOfParallelism,BatchRows). Records are cutinto batches; each batch is formatted and compressed on a pool thread into a self-contained deflate
segment, and segments are appended to the sheet in order.
Fastest: XLSX 187 ms serial, 50 ms with 4 batches in flight,36 ms with 8; XLSB 124 / 41 / 28 ms. The file grows by 0.18% (XLSX) or 0.04% (XLSB).
its workbook. With
UseSharedStrings, string indexes depend on thread timing, and many distinctstrings (tens of thousands) limit the speed-up.
ZIP checksums on read (behavior change)
InvalidDataExceptionwhen the entry's last bytes are read. v6 checked only the entry's size, soa damaged sheet could read as plausible but wrong cells.
ExcelReaderOptions.VerifyZipChecksums(defaulttrue) turns it off; the bindings expose it asverify_zip_checksums.PrefetchDecompression, where it runs on thebackground thread.
XLS writer
table, so it is stored once.
record structure.
Fixes found by comparing against Excel 16
Readers
\n, not\r\n. Excel stores the break as a CR LF pair inthe XML; XLSB and XLS already returned
\n.(
<v xml:space="preserve">).Writers (each of these produced a file Excel refused or altered)
format's last column, a column width above 255, more than 200 distinct number formats and a number
format longer than 255 characters now throw.
Write<T>with a type that has no numeric form throwsArgumentExceptionin every writer; XLSBand XLS threw
InvalidCastException.Typed parser
boolin XLSB and XLS, as it did in XLSX. A 0/1 flag column read fromthose formats came back all
false.TimeOnlybinds from a text cell in every format, and numeric cells in XLSB and XLS bind toTimeSpan, other parsable types and enums as they do in XLSX.Bindings
Zor an offset no longer shifts by the machine's time zone when a CSV columnis read as a timestamp.
intoutside the signed 64-bit range raisesOverflowErrorinstead of being writtenas its low 64 bits, and
Cell.as_dateis right for January and February 1900.Other
ExcelLimitExceededException.Messageno longer says "bytes" for limits counted in rows, columnsor entries.
How was it verified?
comparing the library's cells with Excel's own values for workbooks Excel saved in XLSX, XLSB and
XLS, in both date systems.
parallelism; parallel writes were opened in Excel at 1,000,000 rows and checked with Python's
zipfile.testzip().Known gaps
docs/performance/benchmarks.mdpredate checksum verification and are 2% to 4%optimistic for XLSX and XLSB.
Arg_ParamName_Name) after the text.