diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index bacdfde0..f84816c6 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -163,9 +163,9 @@ but because a row holding spans into a worker's buffer cannot cross to the consu That constraint also settles what an ordered `ref struct` enumerator would cost. Since a worker cannot convert a row it must hand onward, conversion would run on the consumer thread — and conversion is the -floor here: the `ref struct` aggregate path, which shares this projection code, costs 833.8 ms -single-threaded on the conversion-heavy corpus, against 272.5 ms for the typed path at dop 8. Such an -API would be ~3.1x slower than what already ships while adding nothing over the +floor here: the `ref struct` aggregate path, which shares this projection code, costs 453.5 ms +single-threaded on the conversion-heavy corpus, against 219.9 ms for the typed path at dop 8. Such an +API would be ~2.1x slower than what already ships while adding nothing over the sequential `ref struct` parse, so it was not built. Ordered delivery at full speed needs workers to convert into a per-chunk arena of packed fields that the consumer rehydrates; that remains unbuilt. diff --git a/README.md b/README.md index 313ac3de..110a9c35 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ errors to stderr, so `convert` is safe to pipe. - Each ZIP entry of an XLSX or XLSB package is checked against its CRC-32 as it is read, so a damaged file throws `InvalidDataException` instead of returning wrong cells. `ExcelReaderOptions.VerifyZipChecksums = false` turns the check off. - The XLSX writer emits a compact workbook with strings, numbers, booleans, dates, and blank cells; shared strings are opt-in. - The XLSB writer emits BIFF12 workbook parts inside the standard XLSB ZIP package; shared strings are opt-in. -- `WriteRecordsParallelAsync` and `WriteTableParallelAsync` format and compress batches of records on several cores and write them in order, for XLSX and XLSB. On a Ryzen 7 5700X a 1,000,000-row sheet writes in 46 ms as XLSX with 4 batches in flight and 31 ms with 8, against 172 ms serially; XLSB takes 31 ms and 21 ms against 115 ms. Memory stays at about 3 MB per batch in flight. See [Write records in parallel](docs/guide/writing.md#write-records-in-parallel-xlsxxlsb) and the [benchmarks](docs/performance/benchmarks.md#parallel-record-writes). +- `WriteRecordsParallelAsync` and `WriteTableParallelAsync` format and compress batches of records on several cores and write them in order, for XLSX and XLSB. On a Ryzen 7 5700X a 1,000,000-row sheet writes in 44 ms as XLSX with 4 batches in flight and 28 ms with 8, against 165 ms serially; XLSB takes 30 ms and 19 ms against 114 ms. Memory stays at about 3 MB per batch in flight. See [Write records in parallel](docs/guide/writing.md#write-records-in-parallel-xlsxxlsb) and the [benchmarks](docs/performance/benchmarks.md#parallel-record-writes). - The XLS writer buffers records in memory and assembles the OLE container at `EndAsync`; choose it when write throughput matters more than peak allocation. - `Excel.EncryptPackage`/`EncryptPackageAsync` wrap a written XLSX/XLSB package in agile ECMA-376 encryption; see [Encrypted workbooks](docs/guide/encryption.md). diff --git a/cpp/README.md b/cpp/README.md index 33aa3951..3d21b6bb 100644 --- a/cpp/README.md +++ b/cpp/README.md @@ -166,6 +166,27 @@ for (std::thread &thread : threads) A cursor, a typed reader and an Arrow stream are each used by one thread at a time, and keep working after the workbook is destroyed. A `Sheet` does not: take what you need from it first. +### Tables + +`tables()` lists the workbook's Excel tables ("Format as Table" ranges) as `xl::TableInfo`: name, the +sheet holding it, and its 0-based range, header and totals rows included. `table(index)` and +`table_by_name(name)` (ignoring case) return an `xl::Sheet` bounded to the table, so every read works +on it and starts at the table's header row: + +```cpp +for (const xl::TableInfo &table : *workbook->tables()) +{ + std::println("{} on sheet {}: rows {}..{}", table.name, table.sheet, table.first_row, table.last_row); +} +if (auto sales = workbook->table_by_name("Sales").value()) +{ + auto rows = xl::parse(*sales); +} +``` + +Cells keep the column indexes of the sheet holding the table, and the table's `name()` is that +sheet's. A table is read on one thread whatever the degree of parallelism asks for. + ## Reading rows one at a time `Sheet::rows()` returns a cursor over that sheet. Each row borrows a buffer the cursor @@ -381,7 +402,7 @@ For XLSX and XLSB, `degree_of_parallelism` formats and compresses batches of `ba several threads: `0` uses every processor, `1` (the default) writes serially, and `batch_rows = 0` keeps the library default of 8,192. The columns must not change while the write runs. With `use_shared_strings` on, two parallel runs give the same content in files that differ byte for byte. -XLS, CSV and `xl::WriterHandle` ignore both fields. +XLS, CSV and `xl::WriterHandle`'s per-cell writes ignore both fields. ```cpp xl::write_columns("big.xlsx", columns, {.degree_of_parallelism = 0}); @@ -412,6 +433,26 @@ if (auto done = writer.finish(); !done) { /* done.error().message */ } Every call returns `std::expected`; one out of order fails instead of corrupting the file. +To write several sheets from whole tables, call the handle's `write_sheet` or `write_columns` once +per sheet. They take what the free functions take, plus the sheet's name, and write it in one call +instead of one per cell, using the handle's `degree_of_parallelism` and `batch_rows`. Between sheets +they mix with `start_sheet`..`end_sheet`; while a sheet is open they fail. + +```cpp +auto writer = xl::WriterHandle::open("report.xlsx", {.degree_of_parallelism = 0}).value(); +writer.write_sheet("Sales", sales); // a range of ExcelMapper rows +writer.write_columns("Stock", columns); // xl::ColumnRefs, such as an xl::Table's +if (auto done = writer.finish(); !done) { /* done.error().message */ } +``` + +An `xl::TableOptions` as the last argument writes the sheet as an Excel table, with filter buttons and +a table style (XLSX and XLSB only, every column named). The name must be unique in the workbook; the +style defaults to `TableStyleMedium2`, and `.style = std::nullopt` leaves the table unstyled: + +```cpp +writer.write_sheet("Sales", sales, xl::TableOptions{.name = "Sales", .style = "TableStyleLight9"}); +``` + ## Aggregating a CSV in parallel `xl::aggregate_csv` folds a CSV file or buffer across worker threads. Each partition starts from diff --git a/cpp/tests/write.cpp b/cpp/tests/write.cpp index 3db2cf95..cd54d93d 100644 --- a/cpp/tests/write.cpp +++ b/cpp/tests/write.cpp @@ -741,6 +741,138 @@ static int test_writer_handle_class_to_memory_round_trip() return 0; } +static std::vector read_cells(const xl::Workbook &workbook, int32_t sheet) +{ + std::vector rows; + auto cursor = workbook.sheet(sheet).rows(); + if (!cursor.has_value()) + { + return rows; + } + for (auto row = cursor->next_row(); row.has_value(); row = cursor->next_row()) + { + std::string line; + for (auto cell : *row) + { + line += (line.empty() ? "" : "|") + std::string(cell.value); + } + rows.push_back(std::move(line)); + } + return rows; +} + +static int test_writer_handle_writes_whole_sheets_beside_streamed_ones() +{ + const std::vector sales{{"widget", "w"}, {"gadget", "g"}}; + const std::vector offsets{0, 1, 2, 3, 4, 5}; + const std::vector blob{'a', 'b', 'c', 'd', 'e'}; + const std::vector values{10, 20, 30, 40, 50}; + const std::array stock{xl::string_column("name", offsets, blob), xl::i64_column("qty", values)}; + + for (const int32_t format : {XL_FORMAT_XLSX, XL_FORMAT_XLSB}) + { + for (const int32_t degree : {1, 4}) + { + std::vector bytes; + auto handle = xl::WriterHandle::open(bytes, {.format = format, .degree_of_parallelism = degree, .batch_rows = 2}); + CHECK(handle.has_value(), "WriterHandle::open over a vector must succeed"); + + CHECK(handle->write_sheet("Vendas", sales).has_value(), "write_sheet must succeed"); + CHECK(handle->start_sheet("Resumo").has_value(), "start_sheet must succeed"); + CHECK(handle->start_row().has_value(), "start_row must succeed"); + CHECK(handle->write(std::string_view("total")).has_value(), "write must succeed"); + CHECK(handle->end_row().has_value(), "end_row must succeed"); + CHECK(handle->end_sheet().has_value(), "end_sheet must succeed"); + CHECK(handle->write_columns("Estoque", stock).has_value(), "write_columns must succeed"); + CHECK(handle->finish().has_value(), "finish must succeed"); + + auto workbook = xl::Workbook::open(bytes, {.format = format}); + CHECK(workbook.has_value(), "the bytes finish wrote must open"); + CHECK(workbook->sheet(0).name() == "Vendas", "write_sheet's name must reach the bytes"); + CHECK(workbook->sheet(1).name() == "Resumo", "start_sheet's name must reach the bytes"); + CHECK(workbook->sheet(2).name() == "Estoque", "write_columns's name must reach the bytes"); + CHECK(read_cells(*workbook, 0) == (std::vector{"name|view", "widget|w", "gadget|g"}), + "write_sheet must write a header and every row"); + CHECK(read_cells(*workbook, 1) == (std::vector{"total"}), "the streamed sheet must sit between them"); + CHECK(read_cells(*workbook, 2) == (std::vector{"name|qty", "a|10", "b|20", "c|30", "d|40", "e|50"}), + "write_columns must write a header and every row in order"); + } + } + return 0; +} + +static int test_writer_handle_write_columns_fails_while_a_sheet_is_open() +{ + const std::vector values{1}; + const std::array columns{xl::i64_column("qty", values)}; + std::vector bytes; + auto handle = xl::WriterHandle::open(bytes, {.format = XL_FORMAT_XLSX}); + CHECK(handle.has_value(), "WriterHandle::open over a vector must succeed"); + + CHECK(handle->start_sheet("Aberta").has_value(), "start_sheet must succeed"); + CHECK(!handle->write_columns("Tabela", columns).has_value(), "write_columns must fail while a sheet is open"); + CHECK(handle->end_sheet().has_value(), "end_sheet must succeed"); + CHECK(handle->write_columns("Tabela", columns).has_value(), "write_columns must succeed once the sheet is ended"); + CHECK(handle->finish().has_value(), "finish must succeed"); + + auto workbook = xl::Workbook::open(bytes, {.format = XL_FORMAT_XLSX}); + CHECK(workbook.has_value() && workbook->sheet(1).name() == "Tabela", "the table must be the second sheet"); + return 0; +} + +static int test_writer_handle_writes_excel_tables_the_workbook_reads_back() +{ + const std::vector sales{{"widget", "w"}, {"gadget", "g"}}; + const std::vector offsets{0, 1, 2, 3}; + const std::vector blob{'a', 'b', 'c'}; + const std::vector values{10, 20, 30}; + const std::array stock{xl::string_column("name", offsets, blob), xl::i64_column("qty", values)}; + + for (const int32_t format : {XL_FORMAT_XLSX, XL_FORMAT_XLSB}) + { + std::vector bytes; + auto handle = xl::WriterHandle::open(bytes, {.format = format, .degree_of_parallelism = 4, .batch_rows = 1}); + CHECK(handle.has_value(), "WriterHandle::open over a vector must succeed"); + CHECK(handle->start_sheet("Notas").has_value(), "start_sheet must succeed"); + CHECK(handle->end_sheet().has_value(), "end_sheet must succeed"); + CHECK(handle->write_sheet("Vendas", sales, xl::TableOptions{.name = "Sales"}).has_value(), + "write_sheet as a table must succeed"); + CHECK(handle->write_columns("Estoque", stock, xl::TableOptions{.name = "Stock", .style = std::nullopt, .row_stripes = false}) + .has_value(), + "write_columns as a table must succeed"); + CHECK(handle->finish().has_value(), "finish must succeed"); + + auto workbook = xl::Workbook::open(bytes, {.format = format}); + CHECK(workbook.has_value(), "the bytes finish wrote must open"); + auto tables = workbook->tables(); + CHECK(tables.has_value() && tables->size() == 2, "both tables must be listed"); + CHECK((*tables)[0].name == "Sales" && (*tables)[1].name == "Stock", "tables keep their names and order"); + const xl::TableInfo &first = (*tables)[0]; + CHECK(first.index == 0 && first.sheet == 1 && first.first_row == 0 && first.last_row == 2 && first.first_column == 0 && + first.last_column == 1 && first.header_row_count == 1 && first.totals_row_count == 0, + "the table's range must cover its header and both rows"); + + auto found = workbook->table_by_name("sales"); + CHECK(found.has_value() && found->has_value(), "table_by_name must ignore case"); + CHECK((*found)->name() == "Vendas", "a table's sheet reports the sheet holding it"); + CHECK(read_cells(*workbook, (*found)->index()) == (std::vector{"name|view", "widget|w", "gadget|g"}), + "the table must read its header and rows"); + CHECK(read_cells(*workbook, workbook->table(1).index()) == (std::vector{"name|qty", "a|10", "b|20", "c|30"}), + "table() must read the second table"); + + auto missing = workbook->table_by_name("missing"); + CHECK(missing.has_value() && !missing->has_value(), "an unknown table name finds nothing"); + CHECK(!workbook->table(2).rows().has_value(), "a table index past the end fails at the first read"); + } + + std::vector xls; + auto handle = xl::WriterHandle::open(xls, {.format = XL_FORMAT_XLS}); + CHECK(handle.has_value(), "WriterHandle::open over a vector must succeed"); + CHECK(!handle->write_sheet("Vendas", sales, xl::TableOptions{.name = "Sales"}).has_value(), "XLS has no tables"); + CHECK(handle->write_sheet("Vendas", sales).has_value(), "a plain sheet must still be written"); + return 0; +} + static int test_writer_handle_class_rejects_auto_for_memory() { std::vector bytes; @@ -833,6 +965,18 @@ static int test_a_valid_empty_string_replaces_the_default() int main() { + if (test_writer_handle_writes_whole_sheets_beside_streamed_ones() != 0) + { + return 1; + } + if (test_writer_handle_write_columns_fails_while_a_sheet_is_open() != 0) + { + return 1; + } + if (test_writer_handle_writes_excel_tables_the_workbook_reads_back() != 0) + { + return 1; + } if (test_narrow_field_rejects_a_value_it_cannot_hold() != 0) { return 1; diff --git a/docs/guide/reading.md b/docs/guide/reading.md index 4a9b038c..3beed016 100644 --- a/docs/guide/reading.md +++ b/docs/guide/reading.md @@ -128,7 +128,7 @@ A callback that writes to a database or sends messages must be idempotent on a C [`ExcelParseException.Row`](xref:ExcelReader.Core.Parser.ExcelParseException.Row) on `ParseAsync` counts the rows emitted so far. On `ForEachAsync` and `AggregateAsync` it is the row a sequential read reports, whether or not the read was partitioned. -See the [parallel sheet benchmarks](../performance/benchmarks.md#parallel-sheet-reading) and the [parallel CSV benchmarks](../performance/benchmarks.md#parallel-csv) for what this buys. On a real-data XLSX the sheet stage drops from 28 ms (prefetched, sequential) to about 17 ms, and on XLSB from 17 ms to about 13 ms; the floor is inflate, which runs on one thread. CSV has no such floor: its `AggregateAsync` at dop 4 is ~1.8x faster than the typed path while allocating ~450x less, with zero garbage collections. +See the [parallel sheet benchmarks](../performance/benchmarks.md#parallel-sheet-reading) and the [parallel CSV benchmarks](../performance/benchmarks.md#parallel-csv) for what this buys. On a real-data XLSX the sheet stage drops from 24 ms (prefetched, sequential) to about 17 ms, and on XLSB from 17 ms to about 13 ms; the floor is inflate, which runs on one thread. CSV has no such floor: its `AggregateAsync` at dop 4 is ~2.0x faster than the typed path while allocating ~630x less, with zero garbage collections. Properties need setters. A get-only property is skipped during binding and silently receives nothing. @@ -356,15 +356,15 @@ Measured across both read benchmarks (see [Real data reads](../performance/bench | Workload | Default | `PrefetchDecompression = true` | Gain | |---|---:|---:|---:| -| XLSX, real data | 41.2 ms | 27.8 ms | 32% | -| XLSM, real data | 43.5 ms | 28.4 ms | 35% | -| XLSB, real data | 25.8 ms | 16.7 ms | 35% | -| XLSX, string-heavy | 33.1 ms | 23.0 ms | 31% | -| XLSB, string-heavy | 32.6 ms | 19.7 ms | 40% | +| XLSX, real data | 37.2 ms | 23.7 ms | 36% | +| XLSM, real data | 38.0 ms | 23.1 ms | 39% | +| XLSB, real data | 24.7 ms | 16.6 ms | 33% | +| XLSX, string-heavy | 30.0 ms | 20.0 ms | 33% | +| XLSB, string-heavy | 33.6 ms | 19.5 ms | 42% | The gain tracks how much of a read is decompression rather than parsing, so it is smallest -on XLSX real data (about a third) and largest on string-heavy XLSB (40%). Allocations rise from the producer task and the -pooled decompression buffers — on the real-data corpus, roughly 7 KB to 24 KB for XLSX +on XLSB real data and string-heavy XLSX (about a third) and largest on string-heavy XLSB (42%). Allocations rise from the producer task and the +pooled decompression buffers — on the real-data corpus, roughly 7 KB to 23 KB for XLSX and 7 KB to 15 KB for XLSB — and neither path triggers a garbage collection. Do **not** enable it for concurrent server workloads: a caller already reading many files diff --git a/docs/guide/writing.md b/docs/guide/writing.md index 26e6e90e..43c3e838 100644 --- a/docs/guide/writing.md +++ b/docs/guide/writing.md @@ -259,12 +259,12 @@ await sheet.EndAsync(); [`XlsbWriterOptions`](xref:ExcelReader.Core.Writer.Xlsb.XlsbWriterOptions) has the same property. XLS and CSV are uncompressed, so they have nothing to overlap and do not offer it. -Writing a 50,000-row workbook (`WriteBenchmark`, both figures from one run): +Writing a 50,000-row workbook (`WriteBenchmark`): | Workload | Default | `PrefetchWrite = true` | Gain | |---|---:|---:|---:| -| XLSX | 8.527 ms | 5.245 ms | 38% | -| XLSB | 5.786 ms | 3.234 ms | 44% | +| XLSX | 8.191 ms | 5.351 ms | 35% | +| XLSB | 5.462 ms | 3.282 ms | 40% | Allocations are unchanged (4.02 MB either way) — the background writer hands over buffers the row writer already owns rather than copying them. @@ -328,13 +328,13 @@ Writing 1,000,000 rows of four columns (text, integer, date, number) to a `Memor | Write | XLSX | XLSB | |---|---:|---:| -| Serial | 171.57 ms | 115.39 ms | -| Serial, `PrefetchWrite` | 104.92 ms | 60.02 ms | -| Parallel, 1 batch in flight | 165.22 ms | 107.81 ms | -| Parallel, 2 | 89.69 ms | 57.94 ms | -| Parallel, 4 | 46.29 ms | 30.62 ms | -| Parallel, 8 | 30.56 ms | 21.04 ms | -| Parallel, 16 | 27.93 ms | 18.25 ms | +| Serial | 165.08 ms | 113.61 ms | +| Serial, `PrefetchWrite` | 102.18 ms | 62.39 ms | +| Parallel, 1 batch in flight | 164.23 ms | 114.03 ms | +| Parallel, 2 | 84.48 ms | 58.53 ms | +| Parallel, 4 | 44.27 ms | 30.04 ms | +| Parallel, 8 | 27.54 ms | 19.48 ms | +| Parallel, 16 | 21.64 ms | 15.37 ms | The file comes out 0.18% (XLSX) or 0.04% (XLSB) larger, because each batch is compressed on its own. As with `PrefetchWrite`, this is for single-file batch work: a server already writing many files at diff --git a/docs/performance/benchmarks.md b/docs/performance/benchmarks.md index d58161b9..42a0a6f9 100644 --- a/docs/performance/benchmarks.md +++ b/docs/performance/benchmarks.md @@ -5,9 +5,9 @@ Throwaway numbers go stale; these are regenerated from the benchmark suite in ## Benchmarks -Benchmarks were run with BenchmarkDotNet v0.15.8 on Windows 10 (22H2), AMD Ryzen 7 5700X, .NET 10.0.12 (SDK 10.0.401). Generated-data benchmarks use 50,000 rows, except the string-heavy reads, which use 65,536. Raw results: [GitHub Pages benchmark dashboard](https://gabrielmarquezmatte.github.io/ExcelReader/dev/bench/) — `tests/ExcelReader.Benchmarks/BenchmarkDotNet.Artifacts/` is `.gitignore`-excluded and never actually reaches GitHub, so a link to it 404s for every reader. +Benchmarks were run with BenchmarkDotNet v0.16.0-preview.1 on Windows 10 (22H2), AMD Ryzen 7 5700X, .NET 10.0.13 (built with SDK 11.0.100-rc.2, targeting `net10.0`). Generated-data benchmarks use 50,000 rows, except the string-heavy reads, which use 65,536. Raw results: [GitHub Pages benchmark dashboard](https://gabrielmarquezmatte.github.io/ExcelReader/dev/bench/) — `tests/ExcelReader.Benchmarks/BenchmarkDotNet.Artifacts/` is `.gitignore`-excluded and never actually reaches GitHub, so a link to it 404s for every reader. -Every BenchmarkDotNet table is a `--job Short` run (1 launch, 3 warmup and 3 measured iterations) on an otherwise idle machine. Rows whose 99.9% confidence interval came out wider than 25% of the mean were run again with `--job Medium` (2 launches, 15 iterations) and carry those figures instead: the parallel CSV tables, the CSV cell-by-cell read, the Sylvan XLS read, the Sylvan string-heavy reads, the string-heavy XLSX matched-work read, the generated-workbook matched-work read, `DataTable.Load`, the large encrypted read and the `InternStrings` typed parse. The cold-start rows set their own launch count, and the file-size and working-set figures under the parallel record writes, the Python/C++/Rust figures and the native-library comparison come from other harnesses that were not re-run. The large-sheet and parallel-write tables are `--job Medium` runs. +Every BenchmarkDotNet table is a `--job Short` run (1 launch, 3 warmup and 3 measured iterations) on an otherwise idle machine. Methods whose 99.9% confidence interval came out wider than 10% of the mean on any row were run again with `--job Medium` (2 launches, 15 iterations) and carry those figures instead: the parallel CSV tables (except Sep's sequential rows), the large-sheet tables, the parallel sheet read, the serial, `PrefetchWrite`, 2-batch XLSB and 16-batch XLSX parallel writes, the XLS and XLSX rows of the XLS writer comparison, the XLS typed record write, the XLSB `PrefetchWrite` write, the XLSX real-data matched-work read, the string-heavy XLSX matched-work read, the interned XLSB matched-work read and the Sylvan XLSX string-heavy read, the stored whole read, the inferred-schema Arrow conversion, the decoded native read-all and the batched native typed reads. The read and typed-parse rows of the XLSX and XLSB tables, the generated-workbook matched-work read, the `ref struct` parsing table, the ADO.NET bridge table and the encrypted-workbook table come from later `--job Short --launchCount 5` runs instead: Tiered PGO settles each process into a different code shape, which moved these rows by up to ±10% between single-launch runs. The cold-start rows set their own launch count, and the file-size and working-set figures under the parallel record writes, the managed deflate and CRC-32 figures, the Python/C++/Rust figures and the native-library comparison come from other harnesses that were not re-run. **Benchmark methodology.** In the Excel-format tables below, the **"Cell-by-cell read"** rows (including the per-format rows in "Real data reads" / "String-heavy reads") read ExcelReader's `cell.Value` — a zero-copy `ReadOnlySpan`, no decode or allocation — against each competitor's own idiomatic read call. For Sylvan.Data.Excel, that's the ADO.NET-style `GetString(i)`, which is forced to materialize a UTF-16 `string`; its API has no zero-copy accessor, so it cannot avoid that cost. These rows are therefore not matched work: part of the reported gap is "we parse faster" and part is "we skipped an allocation you were never offered a way to skip." Each affected benchmark class also has a `*_Materialized` sibling (calling `cell.GetString()`, the same UTF-16 materialization Sylvan pays) so the matched-work number is measurable — see `tests/ExcelReader.Benchmarks/Shared/BenchmarkAccumulators.cs`. **Those results are published under [Matched-work reads](#matched-work-reads-_materialized), and reading them is the honest way to judge the comparison:** treat the Excel cell-by-cell ratios in the tables below as an upper bound, not a like-for-like number. The CSV cell-by-cell rows are matched work: Sep and Sylvan.Data.Csv both expose span accessors (`row[i].Span`, `GetFieldSpan(i)`), and the benchmarks use them. The **"Typed row parsing"** / **"Typed record writing"** rows are unaffected — both sides already materialize real objects/strings there, so those comparisons are matched work as published. @@ -15,57 +15,57 @@ Every BenchmarkDotNet table is a `--job Short` run (1 launch, 3 warmup and 3 mea Compares ExcelReader against established XLSX libraries on the same generated workbook shape. -| Scenario | ExcelReader | Sylvan | OfficeIMO | SpreadCheetah | -|---|---:|---:|---:|---:| -| Cell-by-cell read | 6.780 ms, 3.63 KB | 35.084 ms, 1.89 MB | 14.447 ms, 3.51 MB | - | -| Cell-by-cell read async | 6.953 ms, 3.70 KB | - | - | - | -| Typed row parsing | 10.979 ms, 3.87 MB | 56.677 ms, 10.48 MB | 11.802 ms, 2.36 MB | - | -| Typed row parsing async | 10.958 ms, 3.87 MB | 58.753 ms, 10.48 MB | - | - | -| Typed row parsing, shared strings | 9.896 ms, 2.29 MB | - | - | - | -| Typed row parsing, `InternStrings` | 11.58 ms, 2.30 MB | - | - | - | -| Workbook writing | 8.527 ms, 4.02 MB | - | 18.100 ms, 4.04 MB | 15.393 ms, 15.84 MB | -| Workbook writing, shared strings | 8.416 ms, 4.06 MB | - | - | - | +| Scenario | ExcelReader | Sylvan | ExcelDataReader | OfficeIMO | SpreadCheetah | +|---|---:|---:|---:|---:|---:| +| Cell-by-cell read | 7.098 ms, 3.44 KB | 37.056 ms, 1.89 MB | 87.075 ms, 32.88 MB | 13.892 ms, 3.51 MB | - | +| Cell-by-cell read async | 7.428 ms, 3.51 KB | - | - | - | - | +| Typed row parsing | 10.913 ms, 3.87 MB | 60.050 ms, 10.48 MB | - | 11.790 ms, 2.36 MB | - | +| Typed row parsing async | 10.223 ms, 3.87 MB | 58.884 ms, 10.48 MB | - | - | - | +| Typed row parsing, shared strings | 7.737 ms, 2.29 MB | - | - | - | - | +| Typed row parsing, `InternStrings` | 10.633 ms, 2.30 MB | - | - | - | - | +| Workbook writing | 8.191 ms, 4.02 MB | - | - | 18.162 ms, 4.04 MB | 14.828 ms, 15.84 MB | +| Workbook writing, shared strings | 7.981 ms, 4.03 MB | - | - | - | - | -ExcelReader is ~5.2x faster than Sylvan for raw XLSX reads, allocating ~534x less, and ~2.1x faster than OfficeIMO. For typed parsing it is ~5.2x faster than Sylvan (~5.4x async against async) and ~1.1x faster than OfficeIMO, which allocates less (2.36 MB against 3.87 MB) because it always interns strings. `ExcelReaderOptions.InternStrings` is off by default; turned on, ExcelReader allocates 2.30 MB at 11.58 ms, against OfficeIMO's 11.93 ms in the same `--job Medium` run. For XLSX writing, ExcelReader is ~1.8x faster than SpreadCheetah while allocating ~3.9x less memory, and ~2.1x faster than OfficeIMO. The OfficeIMO rows are OfficeIMO.Excel 3.4.5: that release made its XLSX read and typed parse several times faster than the 3.4.4 figures published before (106.861 ms and 107.805 ms). +ExcelReader is ~5.2x faster than Sylvan for raw XLSX reads, allocating ~564x less, ~12.3x faster than ExcelDataReader, and ~2.0x faster than OfficeIMO. For typed parsing it is ~5.5x faster than Sylvan (~5.8x async against async) and ~1.1x faster than OfficeIMO, which allocates less (2.36 MB against 3.87 MB) because it always interns strings. `ExcelReaderOptions.InternStrings` is off by default; turned on, ExcelReader allocates 2.30 MB at 10.633 ms, ~10% faster than OfficeIMO in the same run. For XLSX writing, ExcelReader is ~1.8x faster than SpreadCheetah while allocating ~3.9x less memory, and ~2.2x faster than OfficeIMO. The OfficeIMO rows are OfficeIMO.Excel 3.5.1, and the ExcelDataReader rows ExcelDataReader 3.9.0. OfficeIMO.Excel's typed parsing is hand-mapped from its `OpenDataReader`, because `RowsAs` needs an `r` attribute on every row and cell, which the spec makes optional and ExcelReader's writer omits. -Reading a shared-strings XLSX workbook with typed parsing is ~10% faster than the inline-string sheet above (9.896 ms vs. 10.979 ms) and allocates ~41% less (2.29 MB vs. 3.87 MB) — each distinct string decodes once into the shared-string cache instead of once per cell occurrence. +Reading a shared-strings XLSX workbook with typed parsing is ~29% faster than the inline-string sheet above (7.737 ms vs. 10.913 ms) and allocates ~41% less (2.29 MB vs. 3.87 MB) — each distinct string decodes once into the shared-string cache instead of once per cell occurrence. ### XLSB (BIFF12) -| Scenario | ExcelReader | OfficeIMO | -|---|---:|---:| -| Cell-by-cell read | 4.881 ms, 3.89 KB | 10.928 ms, 5.12 MB | -| Cell-by-cell read async | 5.220 ms, 3.96 KB | - | -| Typed row parsing | 7.851 ms, 3.87 MB | - | -| Typed row parsing async | 8.014 ms, 3.87 MB | - | -| Workbook writing | 5.786 ms, 4.02 MB | - | -| Workbook writing, shared strings | 5.734 ms, 4.05 MB | - | +| Scenario | ExcelReader | ExcelDataReader | OfficeIMO | +|---|---:|---:|---:| +| Cell-by-cell read | 4.942 ms, 3.67 KB | 28.106 ms, 18.06 MB | 9.841 ms, 5.12 MB | +| Cell-by-cell read async | 5.071 ms, 3.74 KB | - | - | +| Typed row parsing | 7.375 ms, 3.87 MB | - | - | +| Typed row parsing async | 7.276 ms, 3.87 MB | - | - | +| Workbook writing | 5.462 ms, 4.02 MB | - | - | +| Workbook writing, shared strings | 5.521 ms, 4.03 MB | - | - | -XLSB is the fastest generated Excel format in these results: raw reads are ~1.4x faster than XLSX reads, typed parsing is ~1.4x faster than XLSX parsing, and writing is ~1.5x faster than XLSX writing. Against OfficeIMO's XLSB reader, ExcelReader is ~2.2x faster while allocating ~1,350x less. The XLSB writer is also ~2.7x faster than SpreadCheetah's XLSX writer on this benchmark while allocating ~75% less memory. +XLSB is the fastest generated Excel format in these results: raw reads are ~1.4x faster than their XLSX counterparts, and typed parsing and writing ~1.5x. Against OfficeIMO's XLSB reader, ExcelReader is ~2.0x faster while allocating ~1,430x less, and against ExcelDataReader's ~5.7x faster. The XLSB writer is also ~2.7x faster than SpreadCheetah's XLSX writer on this benchmark while allocating ~75% less memory. ### XLS (BIFF8) -| Scenario | ExcelReader | Sylvan | OfficeIMO | -|---|---:|---:|---:| -| Cell-by-cell read | 3.961 ms, 2.84 KB | 5.177 ms, 1,717.73 KB | 7.788 ms, 10,499.26 KB | -| Cell-by-cell read async | 3.253 ms, 2.91 KB | - | - | -| Workbook writing | 4.753 ms, 16.31 MB | - | - | +| Scenario | ExcelReader | Sylvan | ExcelDataReader | OfficeIMO | +|---|---:|---:|---:|---:| +| Cell-by-cell read | 3.529 ms, 2.84 KB | 5.199 ms, 1,717.73 KB | 38.718 ms, 43,522.77 KB | 7.629 ms, 10,499.44 KB | +| Cell-by-cell read async | 3.279 ms, 2.91 KB | - | - | - | +| Workbook writing | 4.736 ms, 16.31 MB | - | - | - | -ExcelReader is ~1.3x faster than Sylvan for generated XLS reads while allocating ~605x less memory, and ~2.0x faster than OfficeIMO. The XLS writer is ~1.8x faster than the XLSX writer in this benchmark (`XlsWriteBenchmark`, 4.753 ms vs 8.600 ms in the same run). Most of its 16.31 MB is the benchmark's pre-sized 16 MB destination `MemoryStream`; the typed-record table below measures it at 4.31 MB, close to the other formats. +ExcelReader is ~1.5x faster than Sylvan for generated XLS reads while allocating ~605x less memory, ~2.2x faster than OfficeIMO, and ~11x faster than ExcelDataReader. The XLS writer is ~1.7x faster than the XLSX writer in this benchmark (`XlsWriteBenchmark`, 4.736 ms vs 7.941 ms in the same run). Most of its 16.31 MB is the benchmark's pre-sized 16 MB destination `MemoryStream`; the typed-record table below measures it at 4.31 MB, close to the other formats. ### CSV | Scenario | ExcelReader | Sep | Sylvan.Data.Csv | |---|---:|---:|---:| -| Cell-by-cell read | 2.499 ms, 888 B | 7.827 ms, 3.93 KB | 4.044 ms, 37.78 KB | -| Cell-by-cell read async | 2.935 ms, 960 B | - | - | -| Typed row parsing | 4.889 ms, 3.86 MB | 8.736 ms, 3.87 MB | 12.161 ms, 10.95 MB | -| Typed row parsing async | 4.915 ms, 3.86 MB | - | - | -| Row writing | 4.551 ms, 4.00 MB | 6.880 ms, 4.01 MB | 6.983 ms, 4.04 MB | +| Cell-by-cell read | 2.986 ms, 888 B | 8.005 ms, 3.93 KB | 4.034 ms, 37.78 KB | +| Cell-by-cell read async | 2.907 ms, 960 B | - | - | +| Typed row parsing | 4.618 ms, 3.86 MB | 8.438 ms, 3.87 MB | 11.703 ms, 10.95 MB | +| Typed row parsing async | 4.876 ms, 3.86 MB | - | - | +| Row writing | 4.634 ms, 4.00 MB | 6.861 ms, 4.01 MB | 6.925 ms, 4.04 MB | -For raw CSV reads, all three libraries read text through spans. ExcelReader is ~3.1x faster than Sep while allocating ~4.5x less, and ~1.6x faster than Sylvan.Data.Csv while allocating ~44x less. For typed CSV parsing (the more common case — building actual records), ExcelReader is ~1.8x faster than Sep and ~2.5x faster than Sylvan.Data.Csv, with the lowest allocation of the group. For CSV writing, ExcelReader is ~1.5x faster than both Sep and Sylvan.Data.Csv, which are tied; the ~4 MB shown across all three is primarily the benchmark's pre-sized destination `MemoryStream`, not per-row writer state. +For raw CSV reads, all three libraries read text through spans. ExcelReader is ~2.7x faster than Sep while allocating ~4.5x less, and ~1.4x faster than Sylvan.Data.Csv while allocating ~44x less. For typed CSV parsing (the more common case — building actual records), ExcelReader is ~1.8x faster than Sep and ~2.5x faster than Sylvan.Data.Csv, with the lowest allocation of the group. For CSV writing, ExcelReader is ~1.5x faster than both Sep and Sylvan.Data.Csv, which are tied; the ~4 MB shown across all three is primarily the benchmark's pre-sized destination `MemoryStream`, not per-row writer state. ### Parallel CSV @@ -73,34 +73,34 @@ For raw CSV reads, all three libraries read text through spans. ExcelReader is ~ | Dop | Conversion-heavy, typed | Conversion-heavy, struct + `InternStrings` | Narrow ints, typed | Conversion-heavy, `ref struct` aggregate | Narrow ints, `ref struct` aggregate | |---:|---:|---:|---:|---:|---:| -| 1 | 611.6 ms, 671.42 MB | 543.9 ms, 1.62 MB | 525.5 ms, 245.76 MB | 475.8 ms, 1.20 MB | 473.4 ms, 1.27 MB | -| 2 | 477.9 ms, 679.58 MB | 431.9 ms, 10.40 MB | 401.5 ms, 254.06 MB | 275.2 ms, 0.90 MB | 266.3 ms, 0.91 MB | -| 4 | 281.3 ms, 679.64 MB | 243.6 ms, 10.83 MB | 234.4 ms, 254.06 MB | 149.7 ms, 1.09 MB | 149.2 ms, 1.07 MB | -| 8 | 226.4 ms, 679.59 MB | 186.1 ms, 11.63 MB | 207.0 ms, 253.92 MB | 104.8 ms, 1.27 MB | 101.8 ms, 1.25 MB | -| 16 | 396.0 ms, 679.61 MB | 181.0 ms, 13.60 MB | 220.1 ms, 254.51 MB | 85.4 ms, 1.31 MB | 87.0 ms, 1.32 MB | +| 1 | 612.4 ms, 671.42 MB | 561.2 ms, 1.64 MB | 512.1 ms, 245.76 MB | 453.5 ms, 1.21 MB | 410.6 ms, 1.20 MB | +| 2 | 472.5 ms, 680.04 MB | 447.5 ms, 10.86 MB | 399.0 ms, 254.48 MB | 263.3 ms, 0.91 MB | 245.3 ms, 0.92 MB | +| 4 | 268.9 ms, 680.10 MB | 254.9 ms, 11.33 MB | 224.3 ms, 254.56 MB | 136.1 ms, 1.07 MB | 130.5 ms, 1.09 MB | +| 8 | 219.9 ms, 680.10 MB | 197.2 ms, 12.21 MB | 193.7 ms, 254.52 MB | 92.2 ms, 1.27 MB | 93.3 ms, 1.26 MB | +| 16 | 419.6 ms, 680.12 MB | 171.3 ms, 14.17 MB | 255.5 ms, 255.02 MB | 68.1 ms, 1.38 MB | 70.5 ms, 1.34 MB | -The typed path scales ~2.2x by dop 4 on both corpora (611.6 to 281.3 ms conversion-heavy, 525.5 to 234.4 ms narrow), then flattens out: narrow ints reach 207.0 ms at dop 8 and no better at dop 16, and the conversion-heavy corpus gets *slower* past dop 8 (396.0 ms at dop 16 against 226.4 ms at dop 8, standard deviation 14.1 ms). Its ~680 MB is one materialized row object and two strings per record, and at dop 16 that costs 42,000 Gen0 plus 41,000 Gen1 collections. +The typed path scales ~2.3x by dop 4 on both corpora (612.4 to 268.9 ms conversion-heavy, 512.1 to 224.3 ms narrow), then flattens out: narrow ints reach 193.7 ms at dop 8 and slow to 255.5 ms at dop 16, and the conversion-heavy corpus gets *slower* past dop 8 (419.6 ms at dop 16 against 219.9 ms at dop 8, standard deviation 105.4 ms). Its ~680 MB is one materialized row object and two strings per record, and at dop 16 that costs 42,000 Gen0 plus 41,000 Gen1 collections. -The struct + `InternStrings` column parses the same rows into a `struct` with `ExcelReaderOptions.InternStrings` on, so there is no row object and each repeated region or country name is one cached string. It never slows down: 186.1 ms at dop 8, 181.0 ms at dop 16, 2.2x faster than the class at dop 16 while allocating ~50x less (13.60 MB, no garbage collections at dop 16). What remains is per-worker caches and the ordered merge's buffers. The interning win depends on the data: the corpus has 4 regions and 6 countries, so every string is a cache hit. Text columns with mostly distinct values (names, ids, free text) allocate a string per cell either way, and on those the struct alone removes only the row object. +The struct + `InternStrings` column parses the same rows into a `struct` with `ExcelReaderOptions.InternStrings` on, so there is no row object and each repeated region or country name is one cached string. It never slows down: 197.2 ms at dop 8, 171.3 ms at dop 16, 2.4x faster than the class at dop 16 while allocating ~48x less (14.17 MB, under one Gen0 collection per run). What remains is per-worker caches and the ordered merge's buffers. The interning win depends on the data: the corpus has 4 regions and 6 countries, so every string is a cache hit. Text columns with mostly distinct values (names, ids, free text) allocate a string per cell either way, and on those the struct alone removes only the row object. -The aggregate columns fold each record into a per-partition accumulator through a `ref struct` model, so text columns stay as `ReadOnlySpan` and no row object is ever materialized. They scale better than the typed path — ~4.5x (conversion-heavy) and ~4.7x (narrow) at dop 8, ~5.6x and ~5.4x at dop 16 — because they are not fighting the allocator: at dop 4 the conversion-heavy aggregate is ~1.9x faster than typed while allocating ~620x less (1.09 MB against 679.64 MB), with **zero** garbage collections at every degree. The remaining allocation is per-partition bookkeeping, not per-row. +The aggregate columns fold each record into a per-partition accumulator through a `ref struct` model, so text columns stay as `ReadOnlySpan` and no row object is ever materialized. They scale better than the typed path — ~4.9x (conversion-heavy) and ~4.4x (narrow) at dop 8, ~6.7x and ~5.8x at dop 16 — because they are not fighting the allocator: at dop 4 the conversion-heavy aggregate is ~2.0x faster than typed while allocating ~630x less (1.07 MB against 680.10 MB), with **zero** garbage collections at every degree. The remaining allocation is per-partition bookkeeping, not per-row. -The typed path's slowdown past dop 8 is the garbage collector. Every degree allocates the same ~680 MB and runs 42 gen0 collections, but at dop 16 each of them also collects gen1 (41 gen1 collections against 24 at dop 8), and the pauses add up to 100-200 ms against 39 ms. In a console harness on the 4.3 million row corpus, with the default workstation GC, dop 16 took 365 ms; with `DOTNET_gcServer=1` it took 182 ms, faster than dop 8. Not allocating per row, as the struct + `InternStrings` and aggregate columns do, avoids it without changing the GC mode. +The typed path's slowdown past dop 8 is the garbage collector. Every degree allocates the same ~680 MB and runs 42 gen0 collections, but at dop 16 each of them also collects gen1 (41 gen1 collections against 26 at dop 8), and the pauses add up to 100-200 ms against 39 ms. In a console harness on the 4.3 million row corpus, with the default workstation GC, dop 16 took 365 ms; with `DOTNET_gcServer=1` it took 182 ms, faster than dop 8. Not allocating per row, as the struct + `InternStrings` and aggregate columns do, avoids it without changing the GC mode. -Type conversion is the floor under all of them. A single-threaded pass that allocates nothing still costs 475.8 ms, so parsing bytes into `DateTime`/`decimal`/`int` — not allocation and not I/O — is what the parallelism is actually buying down. +Type conversion is the floor under all of them. A single-threaded pass that allocates nothing still costs 453.5 ms, so parsing bytes into `DateTime`/`decimal`/`int` — not allocation and not I/O — is what the parallelism is actually buying down. Against [Sep](https://github.com/nietras/Sep)'s own `ParallelEnumerate` (all cores; 8,000,000 narrow rows, 3,000,000 conversion-heavy rows): | Corpus, row type | ExcelReader sequential | ExcelReader parallel | Sep sequential | Sep parallel | |---|---:|---:|---:|---:| -| Narrow ints, class | 492.3 ms, 244.15 MB | 227.7 ms, 254.44 MB | 525.1 ms, 244.15 MB | 460.4 ms, 245.89 MB | -| Narrow ints, struct | 397.2 ms, 10.32 KB | 158.0 ms, 11.58 MB | 494.2 ms, 9.34 KB | 263.1 ms, 2.16 MB | -| Conversion-heavy, class | 418.9 ms, 467.31 MB | 286.4 ms, 474.33 MB | 831.0 ms, 467.31 MB | 717.6 ms, 468.34 MB | -| Conversion-heavy, struct + pooled strings | 375.6 ms, 11.37 KB | 121.8 ms, 10.84 MB | 803.5 ms, 10.77 KB | 413.5 ms, 7.76 MB | +| Narrow ints, class | 474.9 ms, 244.15 MB | 269.1 ms, 255.02 MB | 507.5 ms, 244.15 MB | 433.0 ms, 245.46 MB | +| Narrow ints, struct | 392.0 ms, 13.93 KB | 162.0 ms, 12.15 MB | 476.2 ms, 9.34 KB | 251.7 ms, 2.81 MB | +| Conversion-heavy, class | 407.6 ms, 467.31 MB | 353.3 ms, 474.68 MB | 835.3 ms, 467.31 MB | 596.5 ms, 468.41 MB | +| Conversion-heavy, struct + pooled strings | 377.3 ms, 14.55 KB | 119.6 ms, 11.28 MB | 857.6 ms, 10.77 KB | 392.7 ms, 9.33 MB | -The struct rows give both libraries the same help: the row is a `struct`, and repeated text is pooled, through `ExcelReaderOptions.InternStrings` on ExcelReader's side and `SepToString.PoolPerCol()` (sequential) or `PoolPerColThreadSafe()` (parallel) on Sep's. Sep's parallel struct rows are from a Medium job, since the Short run spread too much (StdDev 47.6 ms on narrow ints); the conversion-heavy one still has a 30.4 ms standard deviation. +The struct rows give both libraries the same help: the row is a `struct`, and repeated text is pooled, through `ExcelReaderOptions.InternStrings` on ExcelReader's side and `SepToString.PoolPerCol()` (sequential) or `PoolPerColThreadSafe()` (parallel) on Sep's. Every row except Sep's sequential ones is from a Medium job, since the Short run spread too much; Sep's conversion-heavy parallel struct row still has a 35.1 ms standard deviation, and ExcelReader's conversion-heavy parallel class row 29.4 ms. -Sequentially, ExcelReader's typed parser is ~1.1x faster than Sep on narrow ints (492.3 ms vs. 525.1 ms), and ~2.0x faster on the conversion-heavy corpus, where decimal parsing dominates. In parallel ExcelReader is ~2.0x (narrow) and ~2.5x (conversion-heavy) faster than Sep's, while yielding rows in file order. With struct rows and pooled strings both libraries speed up and stop collecting, and ExcelReader stays ahead: ~1.2x and ~2.1x sequentially, ~1.7x and ~3.4x in parallel. The class rows gain less from parallelism (286.4 ms against 418.9 ms sequential on the conversion-heavy corpus) because they are bound by the allocator, like the typed dop 16 row above: 28 Gen1 collections per run; the struct rows run 3.1x faster in parallel on the same corpus. As in the table above, pooling pays off here because the conversion-heavy corpus repeats 4 regions and 6 countries. +Sequentially, ExcelReader's typed parser is ~1.1x faster than Sep on narrow ints (474.9 ms vs. 507.5 ms), and ~2.0x faster on the conversion-heavy corpus, where decimal parsing dominates. In parallel ExcelReader is ~1.6x (narrow) and ~1.7x (conversion-heavy) faster than Sep's, while yielding rows in file order. With struct rows and pooled strings both libraries speed up and stop collecting, and ExcelReader stays ahead: ~1.2x and ~2.3x sequentially, ~1.6x and ~3.3x in parallel. The class rows gain less from parallelism (353.3 ms against 407.6 ms sequential on the conversion-heavy corpus) because they are bound by the allocator, like the typed dop 16 row above: 28 Gen1 collections per run; the struct rows run 3.2x faster in parallel on the same corpus. As in the table above, pooling pays off here because the conversion-heavy corpus repeats 4 regions and 6 countries. ### Parallel sheet reading @@ -108,12 +108,12 @@ Sequentially, ExcelReader's typed parser is ~1.1x faster than Sep on narrow ints | Dop | XLSX | XLSB | |---:|---:|---:| -| 1 | 52.76 ms, 8.02 MB | 33.82 ms, 8.02 MB | -| 2 | 36.63 ms, 8.13 MB | 25.35 ms, 8.07 MB | -| 4 | 16.64 ms, 8.11 MB | 13.25 ms, 8.07 MB | -| 8 | 16.73 ms, 8.16 MB | 13.25 ms, 8.09 MB | +| 1 | 46.07 ms, 8.02 MB | 29.86 ms, 8.02 MB | +| 2 | 29.36 ms, 8.13 MB | 22.49 ms, 8.07 MB | +| 4 | 16.69 ms, 8.12 MB | 13.28 ms, 8.07 MB | +| 8 | 16.79 ms, 8.17 MB | 13.18 ms, 8.09 MB | -At dop 4 XLSX reads ~3.2x faster than sequentially and XLSB ~2.6x. Dop 2 is a single parse worker running alongside the inflating thread. Past dop 4 nothing improves: the one inflating thread is the floor (~14 ms for XLSX, ~9 ms for XLSB, see [Inflate](#inflate), plus the XLSB record walk that finds row boundaries). Allocation barely moves with dop (8.02 to 8.16 MB): it is the typed model's strings. +At dop 4 XLSX reads ~2.8x faster than sequentially and XLSB ~2.2x. Dop 2 is a single parse worker running alongside the inflating thread. Past dop 4 nothing improves: the one inflating thread is the floor (~14 ms for XLSX, ~9 ms for XLSB, see [Inflate](#inflate), plus the XLSB record walk that finds row boundaries). Allocation barely moves with dop (8.02 to 8.17 MB): it is the typed model's strings. ### Large sheets @@ -121,31 +121,31 @@ At dop 4 XLSX reads ~3.2x faster than sequentially and XLSB ~2.6x. Dop 2 is a si | XLSX, Dop | Aggregate | Typed parse | |---:|---:|---:| -| 1 | 215.91 ms, 77.25 MB | 243.27 ms, 77.25 MB | -| 2 | 169.84 ms, 77.62 MB | 185.00 ms, 106.81 MB | -| 4 | 64.81 ms, 77.58 MB | 74.01 ms, 106.75 MB | -| 8 | 51.11 ms, 77.52 MB | 55.44 ms, 106.78 MB | -| 16 | 44.51 ms, 77.94 MB | 57.32 ms, 107.11 MB | +| 1 | 223.16 ms, 77.25 MB | 249.45 ms, 77.25 MB | +| 2 | 163.11 ms, 77.62 MB | 190.50 ms, 106.81 MB | +| 4 | 59.25 ms, 77.57 MB | 73.08 ms, 106.75 MB | +| 8 | 45.08 ms, 77.64 MB | 55.79 ms, 106.78 MB | +| 16 | 44.48 ms, 77.95 MB | 55.15 ms, 107.11 MB | | XLSB, Dop | Aggregate | Typed parse | |---:|---:|---:| -| 1 | 160.91 ms, 77.25 MB | 189.08 ms, 77.25 MB | -| 2 | 119.50 ms, 77.53 MB | 127.86 ms, 99.69 MB | -| 4 | 48.76 ms, 77.46 MB | 58.04 ms, 99.59 MB | -| 8 | 46.92 ms, 77.61 MB | 53.51 ms, 99.74 MB | -| 16 | 49.19 ms, 77.88 MB | 54.30 ms, 100.00 MB | +| 1 | 157.36 ms, 77.25 MB | 170.55 ms, 77.25 MB | +| 2 | 108.82 ms, 77.53 MB | 118.56 ms, 99.69 MB | +| 4 | 47.71 ms, 77.48 MB | 56.25 ms, 99.62 MB | +| 8 | 49.58 ms, 77.61 MB | 54.14 ms, 99.76 MB | +| 16 | 49.46 ms, 77.87 MB | 53.82 ms, 100.03 MB | | CSV, Dop | Aggregate | Typed parse | Typed parse, struct + `InternStrings` | |---:|---:|---:|---:| -| 1 | 434.80 ms, 2.84 KB | 541.20 ms, 623.08 MB | 504.24 ms, 10.86 KB | -| 2 | 220.11 ms, 6.79 KB | 369.71 ms, 626.88 MB | 311.99 ms, 4.44 MB | -| 4 | 148.27 ms, 10.61 KB | 201.27 ms, 626.70 MB | 163.29 ms, 4.67 MB | -| 8 | 95.41 ms, 18.32 KB | 169.04 ms, 626.46 MB | 138.99 ms, 5.29 MB | -| 16 | 69.28 ms, 34.10 KB | 250.57 ms, 626.59 MB | 119.77 ms, 7.33 MB | +| 1 | 418.65 ms, 1.99 KB | 509.94 ms, 623.07 MB | 506.93 ms, 14.05 KB | +| 2 | 204.59 ms, 11.58 KB | 325.09 ms, 626.87 MB | 323.26 ms, 4.46 MB | +| 4 | 115.50 ms, 10.94 KB | 200.87 ms, 626.68 MB | 178.60 ms, 4.70 MB | +| 8 | 84.80 ms, 18.88 KB | 165.18 ms, 626.40 MB | 139.31 ms, 5.33 MB | +| 16 | 61.33 ms, 35.14 KB | 216.44 ms, 626.55 MB | 122.48 ms, 7.40 MB | The XLSX and XLSB aggregates and typed parses each materialize a `Record` and its name string per row, which is where their ~77-107 MB comes from; the CSV aggregate binds a `ref struct` and allocates kilobytes, while the CSV typed parse allocates ~626 MB at every degree, one row object and two strings per record. The struct + `InternStrings` parse drops both (the corpus repeats 4 regions and 6 countries, see [Parallel CSV](#parallel-csv)) and keeps a few MB of per-worker caches and merge buffers. -XLSX and XLSB reach ~3.3x at dop 4 and then barely improve, as in the real-data table above (the XLSX aggregate gets to 4.9x at dop 16, the XLSB aggregate to 3.4x at dop 8). The CSV aggregate keeps scaling because nothing but the parser runs on the workers: 4.6x at dop 8 and 6.3x at dop 16. The typed CSV parse reaches 3.2x at dop 8 and then loses 48% at dop 16 to the collector (standard deviation 15.2 ms), while the struct + `InternStrings` parse keeps going to 4.2x at dop 16, 2.1x faster than the class. The aggregate is still the fastest, ~1.7x ahead of the struct parse at dop 16, because it never builds the row at all. +XLSX and XLSB reach ~3.0-3.8x at dop 4. The XLSB aggregate stops there (3.3x), as in the real-data table above, while the XLSX aggregate improves to 5.0x at dop 8 and then stops. The CSV aggregate keeps scaling because nothing but the parser runs on the workers: 4.9x at dop 8 and 6.8x at dop 16. The typed CSV parse reaches 3.1x at dop 8 and then loses 31% at dop 16 to the collector, while the struct + `InternStrings` parse keeps going to 4.1x at dop 16, 1.8x faster than the class. The aggregate is still the fastest, ~2.0x ahead of the struct parse at dop 16, because it never builds the row at all. ### Real data reads @@ -153,13 +153,13 @@ This benchmark reads a real workbook exported in multiple formats. | Format | ExcelReader | ExcelReader, prefetch | Sylvan | |---|---:|---:|---:| -| XLSX | 41.206 ms, 7.38 KB | 27.823 ms, 23.54 KB | 191.883 ms, 644.25 KB | -| XLSM | 43.502 ms, 7.38 KB | 28.364 ms, 23.55 KB | 197.799 ms, 644.33 KB | -| XLSB | 25.775 ms, 6.94 KB | 16.722 ms, 15.28 KB | 29.536 ms, 338.54 KB | -| XLS | 10.750 ms, 12.02 KB | n/a | 18.124 ms, 185.90 KB | -| CSV | 4.992 ms, 888 B | n/a | 4.482 ms, 40.26 KB | +| XLSX | 37.237 ms, 7.20 KB | 23.663 ms, 23.34 KB | 195.758 ms, 644.27 KB | +| XLSM | 37.999 ms, 7.20 KB | 23.061 ms, 23.35 KB | 199.923 ms, 644.35 KB | +| XLSB | 24.746 ms, 6.72 KB | 16.642 ms, 15.09 KB | 29.421 ms, 338.55 KB | +| XLS | 11.196 ms, 12.02 KB | n/a | 18.221 ms, 185.90 KB | +| CSV | 4.949 ms, 888 B | n/a | 4.502 ms, 40.26 KB | -On this real-data workload, ExcelReader is ~4.7x faster than Sylvan for XLSX and ~4.5x for XLSM, ~1.1x faster for XLSB, and ~1.7x faster for XLS — allocating ~87x less for XLSX and XLSM, ~49x less for XLSB and ~15x less for XLS. On CSV, where both sides read text through spans, Sylvan.Data.Csv is ~11% faster, while ExcelReader allocates ~46x less. The prefetch column is the opt-in [`PrefetchDecompression`](../guide/reading.md#prefetch-decompression-xlsxxlsb) option; XLS and CSV are uncompressed, so it does not apply to them. +On this real-data workload, ExcelReader is ~5.3x faster than Sylvan for XLSX and XLSM, ~1.2x faster for XLSB, and ~1.6x faster for XLS — allocating ~90x less for XLSX and XLSM, ~50x less for XLSB and ~15x less for XLS. On CSV, where both sides read text through spans, Sylvan.Data.Csv is ~10% faster, while ExcelReader allocates ~46x less. The prefetch column is the opt-in [`PrefetchDecompression`](../guide/reading.md#prefetch-decompression-xlsxxlsb) option; XLS and CSV are uncompressed, so it does not apply to them. ### In-memory real-data reads @@ -167,18 +167,18 @@ The real-data benchmark also measures the in-memory path for workbook content lo | Method | Mean | StdDev | Allocated | |---|---:|---:|---:| -| Xlsx_ExcelReader_Memory | 41.625 ms | 0.079 ms | 7.10 KB | -| Xlsx_ExcelReader_Memory_Prefetch | 28.699 ms | 0.022 ms | 23.26 KB | -| Xlsm_ExcelReader_Memory | 41.559 ms | 0.042 ms | 7.10 KB | -| Xlsm_ExcelReader_Memory_Prefetch | 28.687 ms | 0.011 ms | 23.23 KB | -| Xlsb_ExcelReader_Memory | 24.465 ms | 0.008 ms | 8.61 KB | -| Xlsb_ExcelReader_Memory_Prefetch | 17.235 ms | 0.015 ms | 16.93 KB | -| Xls_ExcelReader_Memory | 10.574 ms | 0.026 ms | 11.88 KB | -| Csv_ExcelReader_Memory | 4.762 ms | 0.003 ms | 616 B | +| Xlsx_ExcelReader_Memory | 37.534 ms | 0.007 ms | 6.71 KB | +| Xlsx_ExcelReader_Memory_Prefetch | 23.609 ms | 0.043 ms | 22.83 KB | +| Xlsm_ExcelReader_Memory | 37.225 ms | 0.045 ms | 6.71 KB | +| Xlsm_ExcelReader_Memory_Prefetch | 23.467 ms | 0.052 ms | 22.78 KB | +| Xlsb_ExcelReader_Memory | 24.418 ms | 0.012 ms | 8.31 KB | +| Xlsb_ExcelReader_Memory_Prefetch | 15.382 ms | 0.047 ms | 16.58 KB | +| Xls_ExcelReader_Memory | 11.031 ms | 0.051 ms | 11.87 KB | +| Csv_ExcelReader_Memory | 4.744 ms | 0.004 ms | 616 B | -`Csv_ExcelReader_Memory` is both faster (4.762 ms vs. 4.992 ms for `Csv_ExcelReader`) and allocates less (616 B vs. 888 B) than its stream twin. +`Csv_ExcelReader_Memory` is both faster (4.744 ms vs. 4.949 ms for `Csv_ExcelReader`) and allocates less (616 B vs. 888 B) than its stream twin. -`Xls_ExcelReader_Memory` is ~2% faster than its stream twin (10.574 ms vs. 10.750 ms for `Xls_ExcelReader`), with near-identical allocation (11.88 KB vs. 12.02 KB). In earlier runs the in-memory path was ~33% faster, thanks to `BiffCursor` caching the enclosing contiguous sector run; the stream path has since mostly caught up. +`Xls_ExcelReader_Memory` is ~1% faster than its stream twin (11.031 ms vs. 11.196 ms for `Xls_ExcelReader`), with near-identical allocation (11.87 KB vs. 12.02 KB). In earlier runs the in-memory path was ~33% faster, thanks to `BiffCursor` caching the enclosing contiguous sector run; the stream path has since mostly caught up. ### String-heavy reads @@ -186,22 +186,22 @@ The real-data corpus above is mostly numbers and dates — its shared-string tab | Format | ExcelReader | ExcelReader, prefetch | Sylvan | |---|---:|---:|---:| -| XLSX | 33.14 ms, 2.18 MB | 22.97 ms, 2.21 MB | 176.85 ms, 17.41 MB | -| XLSB | 32.56 ms, 2.18 MB | 19.68 ms, 2.20 MB | 59.78 ms, 17.38 MB | +| XLSX | 30.00 ms, 2.18 MB | 20.01 ms, 2.21 MB | 176.47 ms, 17.41 MB | +| XLSB | 33.56 ms, 2.18 MB | 19.52 ms, 2.20 MB | 58.80 ms, 17.38 MB | -Both formats handle this well: ~5.3x and ~1.8x faster than Sylvan for XLSX and XLSB respectively, at roughly ~8x less memory in both cases, with no garbage collections in either configuration. XLSB's shared-string path previously materialized its table eagerly (~27 MB here); `ParseSharedStreaming` brought it in line with the XLSX streaming/pooling path, cutting allocation by ~12x on this workload. +Both formats handle this well: ~5.9x and ~1.8x faster than Sylvan for XLSX and XLSB respectively, at roughly ~8x less memory in both cases, with no garbage collections in either configuration. XLSB's shared-string path previously materialized its table eagerly (~27 MB here); `ParseSharedStreaming` brought it in line with the XLSX streaming/pooling path, cutting allocation by ~12x on this workload. `XlsxSharedStringHotPathBenchmark` splits the XLSX read into stages on the same workbook. The stored rows re-zip it without compression, so they show what parsing costs once inflate is gone: | Stage | Mean | Allocated | |---|---:|---:| -| Whole read | 33.06 ms | 2.18 MB | -| Whole read, prefetch | 23.00 ms | 2.21 MB | -| Shared-string table only (open + first row) | 9.37 ms | 747 KB | -| Whole read, stored | 20.69 ms | 2.18 MB | -| Shared-string table only, stored | 3.35 ms | 746 KB | +| Whole read | 30.01 ms | 2.18 MB | +| Whole read, prefetch | 19.56 ms | 2.21 MB | +| Shared-string table only (open + first row) | 9.22 ms | 747 KB | +| Whole read, stored | 17.65 ms | 2.18 MB | +| Shared-string table only, stored | 3.34 ms | 746 KB | -About 37% of the default read is inflate: without it the whole read drops from 33.06 ms to 20.69 ms, and the shared-string table from 9.37 ms to 3.35 ms. What remains, ~17 ms, is the worksheet scan. +About 41% of the default read is inflate: without it the whole read drops from 30.01 ms to 17.65 ms, and the shared-string table from 9.22 ms to 3.34 ms. What remains, ~14 ms, is the worksheet scan. ### Inflate @@ -209,9 +209,9 @@ About 37% of the default read is inflate: without it the whole read drops from 3 | Entry | `DeflateStream` | `InflateStream` | Ratio | |---|---:|---:|---:| -| XLSX `sheet1.xml` (6.7 MB → 34.4 MB) | 22.232 ms, 280 B | 13.523 ms, 200 B | 0.61 | -| XLSB `sheet1.bin` (3.7 MB → 15.9 MB) | 12.695 ms, 280 B | 9.147 ms, 200 B | 0.72 | -| String-heavy `sharedStrings.xml` | 10.375 ms, 280 B | 5.915 ms, 200 B | 0.57 | +| XLSX `sheet1.xml` (6.7 MB → 34.4 MB) | 22.244 ms, 280 B | 13.463 ms, 200 B | 0.61 | +| XLSB `sheet1.bin` (3.7 MB → 15.9 MB) | 12.600 ms, 280 B | 9.040 ms, 200 B | 0.72 | +| String-heavy `sharedStrings.xml` | 10.360 ms, 280 B | 5.763 ms, 200 B | 0.56 | Without prefetch a read is inflate plus parse, so the faster inflate converts directly; with prefetch the two overlap and the read is bounded by the slower stage, which is why the prefetch columns above moved little. @@ -250,15 +250,15 @@ The encoder's limits are deliberate: | Write | XLSX | vs serial | XLSB | vs serial | |---|---:|---:|---:|---:| -| Serial | 171.57 ms, 16.11 KB | 1.0x | 115.39 ms, 13.42 KB | 1.0x | -| Serial, `PrefetchWrite` | 104.92 ms, 18.43 KB | 1.6x | 60.02 ms, 16.25 KB | 1.9x | -| Parallel, 1 batch in flight | 165.22 ms, 48.09 KB | 1.0x | 107.81 ms, 45.48 KB | 1.1x | -| Parallel, 2 | 89.69 ms, 83.40 KB | 1.9x | 57.94 ms, 81.92 KB | 2.0x | -| Parallel, 4 | 46.29 ms, 79.39 KB | 3.7x | 30.62 ms, 77.87 KB | 3.8x | -| Parallel, 8 | 30.56 ms, 74.61 KB | 5.6x | 21.04 ms, 70.78 KB | 5.5x | -| Parallel, 16 | 27.93 ms, 72.66 KB | 6.1x | 18.25 ms, 66.78 KB | 6.3x | +| Serial | 165.08 ms, 16.09 KB | 1.0x | 113.61 ms, 13.42 KB | 1.0x | +| Serial, `PrefetchWrite` | 102.18 ms, 18.35 KB | 1.6x | 62.39 ms, 15.64 KB | 1.8x | +| Parallel, 1 batch in flight | 164.23 ms, 16.15 KB | 1.0x | 114.03 ms, 13.48 KB | 1.0x | +| Parallel, 2 | 84.48 ms, 82.67 KB | 2.0x | 58.53 ms, 81.12 KB | 1.9x | +| Parallel, 4 | 44.27 ms, 79.71 KB | 3.7x | 30.04 ms, 78.45 KB | 3.8x | +| Parallel, 8 | 27.54 ms, 74.09 KB | 6.0x | 19.48 ms, 70.47 KB | 5.8x | +| Parallel, 16 | 21.64 ms, 79.04 KB | 7.6x | 15.37 ms, 67.00 KB | 7.4x | -It scales because a write is bound by building rows, not by deflate: `PrefetchWrite` already takes deflate off the calling thread and stops at 1.6x to 1.9x, so compressing chunks in parallel would add nothing on top of it, while formatting the rows in parallel does. Past dop 8 the gain is small, because the machine has 8 physical cores. None of the rows triggers a garbage collection, and every write allocates under 85 KB for 1,000,000 rows. +It scales because a write is bound by building rows, not by deflate: `PrefetchWrite` already takes deflate off the calling thread and stops at 1.6x to 1.8x, so compressing chunks in parallel would add nothing on top of it, while formatting the rows in parallel does. Past dop 8 the gain shrinks (6.0x to 7.6x for XLSX, 5.8x to 7.4x for XLSB), because the machine has 8 physical cores. None of the rows triggers a garbage collection, and every write allocates under 85 KB for 1,000,000 rows. Each batch is compressed with no reference to the one before it, which costs 0.18% of file size for XLSX (15,481,372 B to 15,508,760 B) and 0.04% for XLSB (16,863,131 B to 16,869,306 B). The process's peak working set rose by 12 MB over the serial write with 4 batches in flight and by 23 MB with 8, about 3 MB per batch, and does not depend on the number of rows. The file sizes and working sets come from a console harness, because BenchmarkDotNet does not report them. @@ -272,30 +272,30 @@ Real-data workbook, per format: | Format | Mean | StdDev | Allocated | |---|---:|---:|---:| -| XLSX | 44.42 ms | 0.04 ms | 16.85 KB | -| XLSM | 44.96 ms | 0.06 ms | 16.85 KB | -| XLSB | 26.61 ms | 0.01 ms | 16.41 KB | -| XLS | 12.31 ms | 0.04 ms | 21.48 KB | -| CSV | 18.63 ms | 0.18 ms | 35.71 MB | +| XLSX | 39.13 ms | 0.08 ms | 16.66 KB | +| XLSM | 39.48 ms | 0.04 ms | 16.66 KB | +| XLSB | 27.93 ms | 0.01 ms | 16.19 KB | +| XLS | 12.73 ms | 0.02 ms | 21.48 KB | +| CSV | 18.95 ms | 0.09 ms | 35.71 MB | Generated 50,000-row XLSX workbook: | Scenario | Mean | StdDev | Allocated | |---|---:|---:|---:| -| Cell-by-cell read, materialized | 8.43 ms | 0.15 ms | 1.58 MB | +| Cell-by-cell read, materialized | 8.42 ms | 0.24 ms | 1.58 MB | String-heavy workbook (65,536 rows, ~190,000 distinct shared strings): | Format | Mean | StdDev | Gen0 | Gen1 | Gen2 | Allocated | |---|---:|---:|---:|---:|---:|---:| -| XLSX | 50.93 ms | 1.98 ms | 1,100.0 | 1,000.0 | 300.0 | 15.25 MB | -| XLSB | 53.12 ms | 0.67 ms | 1,100.0 | 1,000.0 | 300.0 | 15.25 MB | +| XLSX | 50.77 ms | 0.30 ms | 1,100.0 | 1,000.0 | 300.0 | 15.25 MB | +| XLSB | 54.01 ms | 0.24 ms | 1,100.0 | 1,000.0 | 300.0 | 15.25 MB | Reading the allocation columns against the tables above gives the honest shape of the tradeoff: - **CSV real data:** 35.71 MB materialized against 888 B for the span-based read of the same file. Every CSV field is a distinct string, so nothing dedupes — this is where zero-copy reading earns its keep outright. -- **XLSB real data:** 16.41 KB, essentially cheap. That corpus repeats a small set of values, so the shared-string table dedupes and the reader's string cache materializes each distinct value once. -- **String-heavy XLSX:** 15.25 MB materialized, against **Sylvan's 17.41 MB on the same workload** — doing matched work here, ExcelReader allocates ~12% *less* than Sylvan and is still ~3.5x faster (50.93 ms vs. 176.85 ms). The same allocation holds for XLSB (15.25 MB vs. Sylvan's 17.38 MB, also ~12% less), where matched-work time is ~13% ahead of Sylvan (53.12 ms vs. 59.78 ms). This was previously an inversion (ExcelReader allocated ~1.75x *more* than Sylvan here): the per-reader shared-string dedup cache was an unpresized `Dictionary`, and at ~190,000 distinct values its resize/rehash churn (several of the largest resizes landing on the LOH) accounted for the entire gap — the strings themselves were never the problem, since both readers retain the same ~190,000 distinct instances. Replacing the dictionary with a `string?[]` indexed by shared-string index (sized exactly from the table's known count, no resizing) cut the allocation in half and cut wall-clock time by 14-16% on this benchmark too, since the churn was costing cycles, not just memory. +- **XLSB real data:** 16.19 KB, essentially cheap. That corpus repeats a small set of values, so the shared-string table dedupes and the reader's string cache materializes each distinct value once. +- **String-heavy XLSX:** 15.25 MB materialized, against **Sylvan's 17.41 MB on the same workload** — doing matched work here, ExcelReader allocates ~12% *less* than Sylvan and is still ~3.5x faster (50.77 ms vs. 176.47 ms). The same allocation holds for XLSB (15.25 MB vs. Sylvan's 17.38 MB, also ~12% less), where matched-work time is ~8% ahead of Sylvan (54.01 ms vs. 58.80 ms). This was previously an inversion (ExcelReader allocated ~1.75x *more* than Sylvan here): the per-reader shared-string dedup cache was an unpresized `Dictionary`, and at ~190,000 distinct values its resize/rehash churn (several of the largest resizes landing on the LOH) accounted for the entire gap — the strings themselves were never the problem, since both readers retain the same ~190,000 distinct instances. Replacing the dictionary with a `string?[]` indexed by shared-string index (sized exactly from the table's known count, no resizing) cut the allocation in half and cut wall-clock time by 14-16% on this benchmark too, since the churn was costing cycles, not just memory. The takeaway is not that one column beats the other: it is that ExcelReader's headline read numbers come from a zero-copy path competitors do not expose, and when it does the same work as them, the gap narrows — and here, with the dedup cache fixed, no longer inverts even at high shared-string cardinality. @@ -305,12 +305,12 @@ The takeaway is not that one column beats the other: it is that ExcelReader's he | Format | Mean | Allocated | |---|---:|---:| -| XLSX | 8.978 ms | 4.02 MB | -| XLSB | 6.170 ms | 4.02 MB | -| XLS | 6.167 ms | 4.31 MB | -| CSV | 4.688 ms | 4.00 MB | +| XLSX | 8.381 ms | 4.02 MB | +| XLSB | 5.838 ms | 4.02 MB | +| XLS | 6.182 ms | 4.31 MB | +| CSV | 4.814 ms | 4.00 MB | -Relative ordering: CSV at the front, XLS and XLSB close behind it, then XLSX — the record-mapping layer adds little overhead over hand-written cell-by-cell writes (XLSB 6.170 ms vs. 5.786 ms, XLSX 8.978 ms vs. 8.527 ms). +Relative ordering: CSV at the front, XLSB and XLS close behind it, then XLSX — the record-mapping layer adds little overhead over hand-written cell-by-cell writes (XLSB 5.838 ms vs. 5.462 ms, XLSX 8.381 ms vs. 8.191 ms). ### Native and Arrow string writes @@ -318,10 +318,10 @@ Relative ordering: CSV at the front, XLS and XLSB close behind it, then XLSX — | Scenario | Mean | Allocated | |---|---:|---:| -| Native string columns → XLSX | 15.598 ms | 15.93 KB | -| Native string columns → CSV | 4.963 ms | 416 B | -| Arrow `StringArray` → XLSX | 21.837 ms | 15.98 KB | -| Styled rows → XLSX | 6.318 ms | 16.10 KB | +| Native string columns → XLSX | 14.275 ms | 15.93 KB | +| Native string columns → CSV | 5.525 ms | 416 B | +| Arrow `StringArray` → XLSX | 20.674 ms | 15.94 KB | +| Styled rows → XLSX | 5.817 ms | 16.10 KB | Text arrives through `IRowWriter.WriteUtf8`, which the XLSX and CSV writers copy through as bytes whenever the text needs no escaping, so none of these allocate per cell: the ~16 KB is per-workbook ZIP and part setup, which CSV does not have. @@ -331,11 +331,11 @@ A `ref struct` model (see [Parse into a ref struct](../guide/parsing.md#parse-in | Target | Mean | Allocated | |---|---:|---:| -| `class` (`ExcelParser`) | 10.98 ms | 3.87 MB | -| `struct` (`ExcelParser`) | 10.32 ms | 1.58 MB | -| `ref struct` + span binding (`ExcelParser`) | 9.67 ms | 4.17 KB | +| `class` (`ExcelParser`) | 10.91 ms | 3.87 MB | +| `struct` (`ExcelParser`) | 10.68 ms | 1.58 MB | +| `ref struct` + span binding (`ExcelParser`) | 9.02 ms | 3.98 KB | -Parsing into a `ref struct` with a `ReadOnlySpan` text column removes essentially all per-row allocation — ~99.9% less than the `class` baseline — and is ~6% faster than the `struct` target, since there's no per-row `string` allocation for the text column. It is not AOT/trim-safe (reflection-based, same tradeoff as `ExcelParser.FromAttributes`). It can be consumed with `foreach` or `await foreach` but not through `IEnumerable`/`IAsyncEnumerable`/LINQ — a `ref struct` element can't be boxed through those interfaces. +Parsing into a `ref struct` with a `ReadOnlySpan` text column removes essentially all per-row allocation — ~99.9% less than the `class` baseline — and is ~16% faster than the `struct` target, since there's no per-row `string` allocation for the text column. It is not AOT/trim-safe (reflection-based, same tradeoff as `ExcelParser.FromAttributes`). It can be consumed with `foreach` or `await foreach` but not through `IEnumerable`/`IAsyncEnumerable`/LINQ — a `ref struct` element can't be boxed through those interfaces. ### ADO.NET bridge (`IDataReader`) @@ -343,13 +343,13 @@ Parsing into a `ref struct` with a `ReadOnlySpan` text column removes esse | Access | Mean | Ratio | Allocated | |---|---:|---:|---:| -| Raw rows (`row[i].Value`, baseline) | 6.545 ms | 1.00 | 3.67 KB | -| `GetBytes` (text column only) | 6.766 ms | 1.03 | 4.37 KB | -| `GetValue` (every column) | 9.215 ms | 1.41 | 5.01 MB | -| Typed getters (`GetString`/`GetInt32`/`GetDateTime`/`GetDouble`) | 8.954 ms | 1.37 | 1.58 MB | -| `DataTable.Load` | 62.767 ms | 9.59 | 22.84 MB | +| Raw rows (`row[i].Value`, baseline) | 6.542 ms | 1.00 | 3.48 KB | +| `GetBytes` (text column only) | 6.983 ms | 1.07 | 4.18 KB | +| `GetValue` (every column) | 9.568 ms | 1.46 | 5.01 MB | +| Typed getters (`GetString`/`GetInt32`/`GetDateTime`/`GetDouble`) | 9.357 ms | 1.43 | 1.58 MB | +| `DataTable.Load` | 62.94 ms | 9.62 | 22.84 MB | -The bridge costs ~1.4x over reading rows directly. `GetValue` boxes every value (the 5 MB), and the typed getters allocate only the text column's `string`s. `GetBytes` copies UTF-8 without making a `string`, so it stays within ~3% of the raw read. `DataTable.Load` is dominated by `DataTable`'s own row storage: the same values cost 9.215 ms to read through `GetValue` alone. +The bridge costs ~1.4x over reading rows directly. `GetValue` boxes every value (the 5 MB), and the typed getters allocate only the text column's `string`s. `GetBytes` copies UTF-8 without making a `string`, so it allocates nothing per row and costs ~7% over the raw read. `DataTable.Load` is dominated by `DataTable`'s own row storage: the same values cost 9.568 ms to read through `GetValue` alone. ### Arrow conversion @@ -357,12 +357,12 @@ The bridge costs ~1.4x over reading rows directly. `GetValue` boxes every value | Scenario | Mean | Allocated | |---|---:|---:| -| CSV, typed, explicit schema | 11.94 ms | 8.13 MB | -| CSV, typed, inferred schema | 12.51 ms | 14.89 MB | -| XLSB, typed, explicit schema | 16.40 ms | 8.14 MB | -| CSV, 8 text columns, explicit schema | 22.22 ms | 16.27 MB | +| CSV, typed, explicit schema | 12.26 ms | 8.13 MB | +| CSV, typed, inferred schema | 12.48 ms | 14.89 MB | +| XLSB, typed, explicit schema | 16.88 ms | 8.14 MB | +| CSV, 8 text columns, explicit schema | 22.31 ms | 16.27 MB | -On the same file, letting `Excel.InferSchema` guess the schema costs ~5% more time (12.51 ms vs. 11.94 ms) and ~1.8x the allocation of passing an explicit `ExcelColumnSchema[]`. The allocation grows with the sheet, because the batch holds every row at once and there is no streaming variant. +On the same file, letting `Excel.InferSchema` guess the schema costs ~2% more time (12.48 ms vs. 12.26 ms) and ~1.8x the allocation of passing an explicit `ExcelColumnSchema[]`. The allocation grows with the sheet, because the batch holds every row at once and there is no streaming variant. ### Encrypted workbooks @@ -370,16 +370,16 @@ On the same file, letting `Excel.InferSchema` guess the schema costs ~5% more ti | Scenario | Mean | Allocated | |---|---:|---:| -| Small, plain | 0.215 ms | 8.91 KB | -| Small, encrypted, stream | 26.414 ms | 38.13 KB | -| Small, encrypted, stream, `VerifyEncryptedIntegrity` | 25.961 ms | 40.29 KB | -| Small, encrypted, in memory | 25.546 ms | 53.28 KB | -| Small, encrypted, open only | 25.244 ms | 32.52 KB | -| 300,000 rows, plain | 36.634 ms | 3.67 KB | -| 300,000 rows, encrypted | 63.432 ms | 29.59 KB | -| 300,000 rows, encrypted, open only | 25.605 ms | 28.92 KB | +| Small, plain | 0.208 ms | 8.72 KB | +| Small, encrypted, stream | 26.133 ms | 37.93 KB | +| Small, encrypted, stream, `VerifyEncryptedIntegrity` | 26.505 ms | 40.05 KB | +| Small, encrypted, in memory | 25.552 ms | 52.98 KB | +| Small, encrypted, open only | 26.099 ms | 32.37 KB | +| 300,000 rows, plain | 37.640 ms | 3.48 KB | +| 300,000 rows, encrypted | 64.103 ms | 29.41 KB | +| 300,000 rows, encrypted, open only | 25.471 ms | 28.72 KB | -Nearly all of the cost is a fixed ~25 ms paid when the workbook is opened: deriving the key through 100,000 SHA-512 iterations, which the format requires so that passwords are slow to brute-force. It does not depend on file size (open-only is 25.24 ms for 15 KB and 25.61 ms for 300,000 rows). After that, decrypting while reading costs ~3% over a plain read: 63.43 ms − 25.61 ms = 37.83 ms, against 36.63 ms unencrypted. On the small fixture, integrity verification and the in-memory path stay within ~3% of the plain encrypted read, inside the spread of the key derivation itself. The fixture is too small to show the extra pass that verification makes over a large stream. +Nearly all of the cost is a fixed ~25 ms paid when the workbook is opened: deriving the key through 100,000 SHA-512 iterations, which the format requires so that passwords are slow to brute-force. It does not depend on file size (open-only is 26.10 ms for 15 KB and 25.47 ms for 300,000 rows). After that, decrypting while reading costs ~3% over a plain read: 64.10 ms − 25.47 ms = 38.63 ms, against 37.64 ms unencrypted. On the small fixture, integrity verification and the in-memory path stay within ~3% of the plain encrypted read, inside the spread of the key derivation itself. The fixture is too small to show the extra pass that verification makes over a large stream. ### Cold start @@ -387,8 +387,8 @@ First use of `ExcelParser.FromAttributes`/`ExcelRecordLayout.FromAttributes`) | 46.64 ms | 20.91 KB | -| First fluent parse (`ExcelParser.Build`) | 42.14 ms | 23.84 KB | -| First fluent parse (`BuildWithAttributeFallback`) | 49.63 ms | 25.97 KB | +| First typed parse (`ExcelParser.FromAttributes`) | 46.05 ms | 20.72 KB | +| First fluent parse (`ExcelParser.Build`) | 41.57 ms | 23.66 KB | +| First fluent parse (`BuildWithAttributeFallback`) | 49.10 ms | 25.78 KB | -`Build` is ~10% faster than the reflection-based parser here; `BuildWithAttributeFallback` is slower than reflection (~6%, with a 0.4 ms standard deviation), since it runs the same `TypeMapper.GetInfo()` path plus the fluent build on top. +`Build` is ~10% faster than the reflection-based parser here; `BuildWithAttributeFallback` is slower than reflection (~7%, with a 0.5 ms standard deviation), since it runs the same `TypeMapper.GetInfo()` path plus the fluent build on top. Run the benchmarks locally: @@ -414,25 +414,25 @@ dotnet run --project tests/ExcelReader.Benchmarks/ExcelReader.Benchmarks.csproj | Entry point | Shape handed back | Mean | Managed allocated | |---|---|---:|---:| -| `xl_rows_next` | one row per call, serialized into a caller buffer | 37.72 ms | 3.71 MB | -| `xl_rows_read_all_blob` | whole sheet in one buffer | 40.31 ms | 21.06 MB | -| `xl_rows_read_all_decoded` | whole sheet as native row structs | 57.21 ms | 5.71 MB | -| `xl_parse_typed` | typed native columns | 34.30 ms | 3.71 MB | -| `xl_parse_arrow` | Arrow C Data Interface arrays | 35.19 ms | 3.72 MB | +| `xl_rows_next` | one row per call, serialized into a caller buffer | 38.42 ms | 3.71 MB | +| `xl_rows_read_all_blob` | whole sheet in one buffer | 41.82 ms | 21.06 MB | +| `xl_rows_read_all_decoded` | whole sheet as native row structs | 52.05 ms | 5.71 MB | +| `xl_parse_typed` | typed native columns | 34.84 ms | 3.71 MB | +| `xl_parse_arrow` | Arrow C Data Interface arrays | 36.02 ms | 3.72 MB | The allocation column is **managed memory only**. `MemoryDiagnoser` cannot see the native blocks that `xl_rows_read_all_decoded`, `xl_parse_typed` and `xl_parse_arrow` return their data in. -Row by row, `xl_rows_next` costs ~1.5x the managed in-memory read of the same file (37.72 ms vs. 24.465 ms for `Xlsb_ExcelReader_Memory` under [In-memory real-data reads](#in-memory-real-data-reads)). Every cell crosses the ABI as text, so each binary XLSB number is formatted to UTF-8 on the way out. `xl_rows_read_all_blob` adds ~7% over it, because the whole sheet is buffered before it is copied out. `xl_rows_read_all_decoded` adds ~52%, because it allocates a native block per row. `xl_parse_typed` and `xl_parse_arrow` are slightly faster than a row-by-row read (~7–9%) and return columnar native buffers with numbers still binary, so they are the path to use for bulk typed loads. +Row by row, `xl_rows_next` costs ~1.6x the managed in-memory read of the same file (38.42 ms vs. 24.418 ms for `Xlsb_ExcelReader_Memory` under [In-memory real-data reads](#in-memory-real-data-reads)). Every cell crosses the ABI as text, so each binary XLSB number is formatted to UTF-8 on the way out. `xl_rows_read_all_blob` adds ~9% over it, because the whole sheet is buffered before it is copied out. `xl_rows_read_all_decoded` adds ~35%, because it allocates a native block per row. `xl_parse_typed` and `xl_parse_arrow` are slightly faster than a row-by-row read (~6–9%) and return columnar native buffers with numbers still binary, so they are the path to use for bulk typed loads. `ChunkedParseBenchmark` compares `xl_parse_typed` on a whole sheet against draining the same sheet in batches through `xl_typed_reader_open`/`xl_typed_reader_next`. It uses the generated 50,000-row XLSX file, read from disk, with three nullable columns (string, int64, float64): | Mode | Mean | Managed allocated | |---|---:|---:| -| Whole sheet (`xl_parse_typed`) | 8.55 ms | 10.06 KB | -| Batches of 10,000 rows | 8.09 ms | 17.00 KB | -| Batches of 1,000 rows | 7.86 ms | 81.34 KB | +| Whole sheet (`xl_parse_typed`) | 8.76 ms | 9.87 KB | +| Batches of 10,000 rows | 8.83 ms | 16.82 KB | +| Batches of 1,000 rows | 8.14 ms | 81.15 KB | -Batching costs nothing in time: both batch sizes are slightly faster than one whole-sheet table (~5–8%). A caller can bound its native memory to one batch at a time without paying for it. +Batching costs nothing in time: batches of 10,000 rows land within 1% of one whole-sheet table, and batches of 1,000 rows are ~7% faster. A caller can bound its native memory to one batch at a time without paying for it. ### C++ and Rust bindings @@ -467,4 +467,3 @@ zsv is the one competitor ahead. The CSV row is the typed workload, where both s 14-column rows with owned strings. Just counting cell bytes, zsv is ~3.3x faster, and that gap is ExcelReader's C ABI, which serializes every row into a buffer, not its CSV parser. That suite is built with GCC (zsv does not compile under MSVC). Its caveats are in the C++ README. - diff --git a/python/README.md b/python/README.md index 4a9ff9fe..601718f6 100644 --- a/python/README.md +++ b/python/README.md @@ -71,6 +71,24 @@ generator that has already started keeps working after `workbook.close()`; one t raises at its first `next()`. `to_arrow_stream()` opens its stream at the call, so it works after `close()` even if never started. +### Tables + +`workbook.tables` lists the workbook's Excel tables ("Format as Table" ranges), with the same +lookups as `sheets`: index, slice or name, ignoring case. A `Table` is a `Sheet` bounded to the +table's range, so every read works on it and starts at the table's header row: + +```python +with excelreader.open_workbook("report.xlsx") as workbook: + sales = workbook.tables["Sales"] + print(sales.name, sales.sheet.name, sales.first_row, sales.last_row) + frame = sales.to_pandas(sales.infer_schema()) +``` + +Rows and columns are zero-based and include the header and totals rows; cells keep the column +indexes of the sheet holding the table. `header_row_count` is 0 for a table whose header row is +turned off, and `totals_row_count` is 1 when it shows a totals row. A table is read on one thread +whatever `parallelism` asks for. + ### Formats `open_workbook` sniffs XLS/XLSX/XLSB by file signature. CSV has no signature, so it is chosen by the @@ -291,10 +309,40 @@ content in files that differ byte for byte. xls and csv ignore both fields. write_workbook("big.xlsx", table, types, options=WriteOptions(degree_of_parallelism=0)) ``` -**Phase-1 limits, stated plainly:** a single sheet only (no multi-sheet workbooks); the whole table -must already be in memory (no streaming/chunked writes); no styling beyond the temporal number -formats `xl_write_typed` applies to DATE/TIME/TIMESTAMP columns. `format="auto"` is not accepted — -sniffing reads a file's existing signature bytes, and a file being created has none. +**Phase-1 limits, stated plainly:** `write_workbook` writes a single sheet (for several, see +`write_sheet` below); the whole table must already be in memory (no streaming/chunked writes); no +styling beyond the temporal number formats `xl_write_typed` applies to DATE/TIME/TIMESTAMP columns. +`format="auto"` is not accepted — sniffing reads a file's existing signature bytes, and a file being +created has none. + +### Several sheets + +`open_writer` returns a `SheetWriter`, whose `write_sheet` writes a whole table as one sheet in a +single native call. It takes the same `data` and `types` as `write_workbook`, so each sheet can be a +`TypedTable` or a pandas, polars or pyarrow frame: + +```python +from excelreader import WriteOptions, open_writer + +with open_writer("report.xlsx", options=WriteOptions(degree_of_parallelism=0)) as writer: + writer.write_sheet("Sales", sales_df) + writer.write_sheet("Stock", table, types) +``` + +Parallelism and the other options come from `open_writer` and apply to every sheet; +`WriteOptions.sheet_name` does not apply, since each call names its sheet. `write_sheet` raises +`ExcelReaderError` while a sheet started with `start_sheet` is open; between sheets, the two styles +mix freely. + +`table=TableOptions(...)` writes the sheet as an Excel table, with filter buttons and a table style +(xlsx and xlsb only). The name must be unique in the workbook; the style defaults to +`TableStyleMedium2`, and `style=None` leaves the table unstyled: + +```python +from excelreader import TableOptions + +writer.write_sheet("Sales", sales_df, table=TableOptions("Sales", style="TableStyleLight9", column_stripes=True)) +``` ## Writing a sheet row by row diff --git a/python/src/excelreader/__init__.py b/python/src/excelreader/__init__.py index ed6e12f4..28d70bcf 100644 --- a/python/src/excelreader/__init__.py +++ b/python/src/excelreader/__init__.py @@ -1,6 +1,6 @@ """Python bindings for ExcelReader.""" -from excelreader.reader import Sheet, Sheets, Workbook, open_workbook +from excelreader.reader import Sheet, Sheets, Table, Tables, Workbook, open_workbook from excelreader.sources import Source from excelreader.types import ( Cell, @@ -13,6 +13,7 @@ PasswordRequiredError, SheetVisibility, StringColumn, + TableOptions, TypedTable, WriteOptions, ) @@ -34,6 +35,9 @@ "Sheets", "Source", "StringColumn", + "Table", + "TableOptions", + "Tables", "TypedTable", "Workbook", "WriteOptions", diff --git a/python/src/excelreader/_native.py b/python/src/excelreader/_native.py index ca9aa581..5e09ad7d 100644 --- a/python/src/excelreader/_native.py +++ b/python/src/excelreader/_native.py @@ -15,7 +15,7 @@ from typing import TYPE_CHECKING if TYPE_CHECKING: - from excelreader.types import OpenOptions, WriteOptions + from excelreader.types import OpenOptions, TableOptions, WriteOptions XL_OK = 0 XL_EOF = -1 @@ -28,6 +28,17 @@ XL_ABI_VERSION = 6 +XL_TABLE_NO_STYLE = 1 +XL_TABLE_NO_ROW_STRIPES = 2 +XL_TABLE_COLUMN_STRIPES = 4 +XL_TABLE_FIRST_COLUMN = 8 +XL_TABLE_LAST_COLUMN = 16 + + +def table_sheet(table: int) -> int: + """XL_TABLE_SHEET: the `sheet` value that makes a read call read table `table`.""" + return -2 - table + XL_SHEET_VISIBLE = 0 XL_SHEET_HIDDEN = 1 XL_SHEET_VERY_HIDDEN = 2 @@ -351,6 +362,51 @@ def default_write_options() -> NativeWriteOptions: return options +class NativeTableInfo(ctypes.Structure): + """Mirrors xl_table_info.""" + + _fields_ = [ + ("sheet", ctypes.c_int32), + ("first_row", ctypes.c_int32), + ("last_row", ctypes.c_int32), + ("first_column", ctypes.c_int32), + ("last_column", ctypes.c_int32), + ("header_row_count", ctypes.c_int32), + ("totals_row_count", ctypes.c_int32), + ] + + +class NativeSheetOptions(ctypes.Structure): + """Mirrors xl_sheet_options.""" + + _fields_ = [ + ("struct_size", ctypes.c_int32), + ("table_name_len", ctypes.c_int32), + ("table_name", ctypes.c_char_p), + ("table_style_len", ctypes.c_int32), + ("table_style", ctypes.c_char_p), + ("table_flags", ctypes.c_int32), + ] + + +def to_native_sheet_options(table: TableOptions | None) -> NativeSheetOptions | None: + if table is None: + return None + raw = NativeSheetOptions(struct_size=ctypes.sizeof(NativeSheetOptions)) + name = table.name.encode("utf-8") + raw.table_name, raw.table_name_len = name, len(name) + flags = 0 if table.row_stripes else XL_TABLE_NO_ROW_STRIPES + flags |= XL_TABLE_COLUMN_STRIPES if table.column_stripes else 0 + flags |= XL_TABLE_FIRST_COLUMN if table.first_column else 0 + flags |= XL_TABLE_LAST_COLUMN if table.last_column else 0 + flags |= XL_TABLE_NO_STYLE if table.style is None else 0 + raw.table_flags = flags + if table.style is not None: + style = table.style.encode("utf-8") + raw.table_style, raw.table_style_len = style, len(style) + return raw + + def to_native_write_options(options: WriteOptions) -> NativeWriteOptions: """Converts a public `WriteOptions` into the raw ABI struct. @@ -469,6 +525,14 @@ def _bind(lib: ctypes.CDLL) -> ctypes.CDLL: lib.xl_sheet_visibility_at.restype = c_int lib.xl_sheet_index.argtypes = [p_void, p_bytes, c_int, p_int] lib.xl_sheet_index.restype = c_int + lib.xl_table_count.argtypes = [p_void, p_int] + lib.xl_table_count.restype = c_int + lib.xl_table_info_at.argtypes = [p_void, c_int, ctypes.POINTER(NativeTableInfo)] + lib.xl_table_info_at.restype = c_int + lib.xl_table_name_at.argtypes = [p_void, c_int, p_bytes, c_int, p_int] + lib.xl_table_name_at.restype = c_int + lib.xl_table_index.argtypes = [p_void, p_bytes, c_int, p_int] + lib.xl_table_index.restype = c_int lib.xl_rows_open.argtypes = [p_void, c_int, pp_void] lib.xl_rows_open.restype = c_int lib.xl_rows_close.argtypes = [p_void] @@ -573,6 +637,12 @@ def _bind(lib: ctypes.CDLL) -> ctypes.CDLL: lib.xl_end_sheet.argtypes = [ctypes.c_void_p] lib.xl_end_sheet.restype = ctypes.c_int32 + lib.xl_write_sheet.argtypes = [ + ctypes.c_void_p, ctypes.POINTER(ctypes.c_uint8), ctypes.c_int32, + ctypes.POINTER(NativeColumnSpec), ctypes.POINTER(NativeTable), ctypes.POINTER(NativeSheetOptions), + ] + lib.xl_write_sheet.restype = ctypes.c_int32 + lib.xl_close_write_handle.argtypes = [ctypes.c_void_p] lib.xl_close_write_handle.restype = ctypes.c_int32 diff --git a/python/src/excelreader/reader.py b/python/src/excelreader/reader.py index 161f2602..8e159865 100644 --- a/python/src/excelreader/reader.py +++ b/python/src/excelreader/reader.py @@ -124,6 +124,12 @@ def sheets(self) -> Sheets: """ return Sheets(self) + @property + def tables(self) -> Tables: + """The workbook's Excel tables: `len()`, iteration, and lookup by index, slice or name + (ignoring case). Each reads like a `Sheet`, bounded to the table's range.""" + return Tables(self) + @property def is_date1904(self) -> bool: flag = ctypes.c_int32() @@ -419,6 +425,103 @@ def infer_schema(self, *, header_row: int = 1, sample_size: int = 100, parse_tex self._lib.xl_free_schema(ctypes.byref(schema)) +class Tables(Sequence["Table"]): + """A workbook's Excel tables, in workbook order. Listing them opens nothing.""" + + def __init__(self, workbook: Workbook) -> None: + self._workbook = workbook + + def __len__(self) -> int: + workbook = self._workbook + count = ctypes.c_int32() + _check(workbook._lib.xl_table_count(workbook._require_handle(), ctypes.byref(count))) + return count.value + + @overload + def __getitem__(self, key: SupportsIndex | str) -> Table: ... + @overload + def __getitem__(self, key: slice) -> list[Table]: ... + def __getitem__(self, key: SupportsIndex | str | slice) -> Table | list[Table]: # type: ignore[override] + workbook = self._workbook + if isinstance(key, str): + encoded = key.encode("utf-8") + index = ctypes.c_int32() + _check(workbook._lib.xl_table_index(workbook._require_handle(), encoded, len(encoded), ctypes.byref(index))) + if index.value < 0: + raise KeyError(key) + return Table(workbook, index.value) + count = len(self) + if isinstance(key, slice): + return [Table(workbook, index) for index in range(*key.indices(count))] + index = operator.index(key) + if index < 0: + index += count + if not 0 <= index < count: + raise IndexError(f"table index {key} is out of range; the workbook has {count} tables") + return Table(workbook, index) + + +class Table(Sheet): + """One Excel table of a `Workbook`. Every `Sheet` read works on it and reads only the table's + range, header row first; cells keep the column indexes of the sheet holding it. + + Rows and columns are zero-based and include the header and totals rows. + """ + + def __init__(self, workbook: Workbook, index: int) -> None: + super().__init__(workbook, _native.table_sheet(index)) + self._table = index + + def __repr__(self) -> str: + return f"Table(index={self._table})" + + @property + def index(self) -> int: + """Zero-based position among the workbook's tables.""" + return self._table + + @property + def name(self) -> str: + raw, length = _fill_buffer(self._lib.xl_table_name_at, self._workbook._require_handle(), self._table, initial=256) + return raw[:length].decode("utf-8") + + @property + def sheet(self) -> Sheet: + """The sheet that holds the table.""" + return Sheet(self._workbook, self._info().sheet) + + @property + def first_row(self) -> int: + return self._info().first_row + + @property + def last_row(self) -> int: + return self._info().last_row + + @property + def first_column(self) -> int: + return self._info().first_column + + @property + def last_column(self) -> int: + return self._info().last_column + + @property + def header_row_count(self) -> int: + """1, or 0 when the table's header row is turned off.""" + return self._info().header_row_count + + @property + def totals_row_count(self) -> int: + """1 when the table shows a totals row, otherwise 0.""" + return self._info().totals_row_count + + def _info(self) -> _native.NativeTableInfo: + info = _native.NativeTableInfo() + _check(self._lib.xl_table_info_at(self._workbook._require_handle(), self._table, ctypes.byref(info))) + return info + + def _decode_row(blob: bytes, length: int, date1904: bool) -> list[Cell]: count = struct.unpack_from(" Any: return (ctypes.c_char * len(blob)).from_buffer_copy(blob) +def _byte_pointer(buffer: Any) -> Any: + # ctypes.cast(buffer, ...) puts buffer in a reference cycle, so the copy would outlive `keepalive` until a gc pass. + return ctypes.cast(ctypes.addressof(buffer), ctypes.POINTER(ctypes.c_uint8)) + + def _native_column(column: Any, column_type: ColumnType, validity: bytes | None, row_count: int, keepalive: list[Any]) -> _native.NativeColumn: if isinstance(column, StringColumn): offsets = _copy(_checked_view(column.offsets, _SIGNED_FORMATS, 4, row_count + 1, "a STRING column's offsets")) @@ -88,9 +93,9 @@ def _native_column(column: Any, column_type: ColumnType, validity: bytes | None, return _native.NativeColumn( type=int(ColumnType.STRING), length=row_count, - values=ctypes.cast(offsets, ctypes.c_void_p), + values=ctypes.addressof(offsets), validity=_validity_pointer(validity, row_count, keepalive), - data=ctypes.cast(data, ctypes.POINTER(ctypes.c_uint8)) if data_len else None, + data=_byte_pointer(data) if data_len else None, data_len=data_len, ) @@ -102,7 +107,7 @@ def _native_column(column: Any, column_type: ColumnType, validity: bytes | None, return _native.NativeColumn( type=int(column_type), length=row_count, - values=ctypes.cast(values, ctypes.c_void_p), + values=ctypes.addressof(values), validity=_validity_pointer(validity, row_count, keepalive), data=None, data_len=0, @@ -118,7 +123,7 @@ def _validity_pointer(validity: bytes | None, row_count: int, keepalive: list[An ) buffer = (ctypes.c_uint8 * len(validity)).from_buffer_copy(validity) keepalive.append(buffer) - return ctypes.cast(buffer, ctypes.POINTER(ctypes.c_uint8)) + return _byte_pointer(buffer) def _marshal( @@ -181,16 +186,9 @@ def write_workbook( returns that it took only part; the object is not closed. The column names become the header row. `options` sets the sheet name and CSV dialect; see - `WriteOptions`. + `WriteOptions`. For several sheets, use `SheetWriter.write_sheet`. """ - if isinstance(data, TypedTable): - if types is None: - raise TypeError("types is required when data is a TypedTable: one ColumnType per column") - table = data - else: - if types is not None: - raise TypeError("types must be None unless data is a TypedTable; Arrow data carries its own types") - table, types = _table_from_arrow(_arrow_table(data)) + table, types = _typed_table(data, types) if isinstance(destination, (str, os.PathLike)): _write_file(Path(destination), table, types, format, options) return @@ -205,6 +203,16 @@ def write_workbook( _write_all(destination, _write_memory(table, types, format, options)) +def _typed_table(data: Any, types: list[ColumnType] | None) -> tuple[TypedTable, list[ColumnType]]: + if isinstance(data, TypedTable): + if types is None: + raise TypeError("types is required when data is a TypedTable: one ColumnType per column") + return data, types + if types is not None: + raise TypeError("types must be None unless data is a TypedTable; Arrow data carries its own types") + return _table_from_arrow(_arrow_table(data)) + + def _write_all(destination: IO[bytes], data: bytes) -> None: """Calls `write()` until it has taken all of `data`. A `write()` that returns None is taken to have written everything, as files that predate the `int` return convention do. diff --git a/python/src/excelreader/writer_handle.py b/python/src/excelreader/writer_handle.py index 1891f765..e590fda4 100644 --- a/python/src/excelreader/writer_handle.py +++ b/python/src/excelreader/writer_handle.py @@ -15,8 +15,8 @@ from excelreader import _native from excelreader.reader import _check -from excelreader.types import ColumnType, WriteOptions -from excelreader.writer import _write_all +from excelreader.types import ColumnType, TableOptions, WriteOptions +from excelreader.writer import _marshal, _typed_table, _write_all _EPOCH_DATE = datetime.date(1970, 1, 1) _EPOCH_DATETIME = datetime.datetime(1970, 1, 1) @@ -43,9 +43,9 @@ class SheetWriter: """A streaming workbook writer. Use as a context manager; closing finalizes the workbook, and for a destination with `write()` hands it the finished bytes. - The required call order is `start_sheet` -> `write_row` per row -> `end_sheet`. Calling out of - order raises `ExcelReaderError`, and the writer stays usable: fix the order and continue, or - close to discard. + The required call order is `start_sheet` -> `write_row` per row -> `end_sheet`, or one + `write_sheet` per whole table. Calling out of order raises `ExcelReaderError`, and the writer + stays usable: fix the order and continue, or close to discard. """ __slots__ = ("_handle", "_lib", "_destination") @@ -67,6 +67,29 @@ def start_sheet(self, name: str) -> None: def end_sheet(self) -> None: _check(self._lib.xl_end_sheet(self._require_handle())) + def write_sheet( + self, name: str, data: Any, types: list[ColumnType] | None = None, *, table: TableOptions | None = None + ) -> None: + """Writes `data` as a whole sheet named `name`, in one native call instead of one per cell. + + `data` and `types` follow `write_workbook`'s rules: a `TypedTable` with one `ColumnType` per + column, or Arrow-compatible data (pyarrow, pandas, polars) with `types` None. The column names + become the header row. With `table`, the sheet is written as an Excel table (xlsx/xlsb only). + Parallelism and the other options come from `open_writer`; `WriteOptions.sheet_name` does not + apply. Raises `ExcelReaderError` while a sheet started with `start_sheet` is open. + """ + typed, types = _typed_table(data, types) + specs, native_table, _, keepalive = _marshal(typed, types, None) + pointer, length = _encode(name) + options = _native.to_native_sheet_options(table) + _check( + self._lib.xl_write_sheet( + self._require_handle(), pointer, length, specs, ctypes.byref(native_table), + None if options is None else ctypes.byref(options), + ) + ) + del keepalive + def write_row(self, values: Sequence[Any]) -> None: """Writes one row, one cell per value, its type inferred from the value. diff --git a/python/tests/test_tables.py b/python/tests/test_tables.py new file mode 100644 index 00000000..360bbb0e --- /dev/null +++ b/python/tests/test_tables.py @@ -0,0 +1,105 @@ +from __future__ import annotations + +import io +import zipfile +from array import array + +import pytest + +from excelreader import ( + ColumnSpec, + ColumnType, + ExcelReaderError, + StringColumn, + TableOptions, + TypedTable, + WriteOptions, + open_workbook, + open_writer, +) + +_SCHEMA = [ColumnSpec(ColumnType.STRING, name="name"), ColumnSpec(ColumnType.I64, name="qty")] +_TYPES = [spec.type for spec in _SCHEMA] + + +def _table(names: list[str], quantities: list[int]) -> TypedTable: + data = "".join(names).encode() + offsets = array("i", [0]) + for name in names: + offsets.append(offsets[-1] + len(name.encode())) + return TypedTable( + row_count=len(names), + names=["name", "qty"], + columns=[StringColumn(offsets, data), array("q", quantities)], + validity=[None, None], + ) + + +def _workbook_bytes(extension: str, options: WriteOptions | None = None) -> bytes: + buffer = io.BytesIO() + with open_writer(buffer, extension, options=options) as writer: + writer.start_sheet("Notas") + writer.write_row(["antes"]) + writer.end_sheet() + writer.write_sheet("Vendas", _table(["widget", "gadget"], [3, 7]), _TYPES, table=TableOptions("Sales")) + writer.write_sheet( + "Estoque", + _table(["a", "b", "c"], [10, 20, 30]), + _TYPES, + table=TableOptions("Stock", style=None, row_stripes=False), + ) + return buffer.getvalue() + + +@pytest.mark.parametrize("extension", ["xlsx", "xlsb"]) +@pytest.mark.parametrize("degree", [1, 4]) +def test_tables_written_by_write_sheet_read_back(extension, degree): + payload = _workbook_bytes(extension, WriteOptions(degree_of_parallelism=degree, batch_rows=1)) + + with open_workbook(payload, format=extension) as workbook: + tables = workbook.tables + assert len(tables) == 2 + assert [table.name for table in tables] == ["Sales", "Stock"] + + sales = tables["sales"] + assert sales == tables[0] + assert sales.index == 0 + assert sales.sheet.name == "Vendas" + assert (sales.first_row, sales.last_row, sales.first_column, sales.last_column) == (0, 2, 0, 1) + assert (sales.header_row_count, sales.totals_row_count) == (1, 0) + assert [[cell.value for cell in row] for row in sales.rows()] == [["name", "qty"], ["widget", "3"], ["gadget", "7"]] + + stock = tables[-1].parse(_SCHEMA) + assert list(stock.columns[0]) == ["a", "b", "c"] + assert list(stock.columns[1]) == [10, 20, 30] + + with pytest.raises(KeyError): + tables["missing"] + with pytest.raises(IndexError): + tables[2] + + +def test_table_options_reach_the_table_part(): + with zipfile.ZipFile(io.BytesIO(_workbook_bytes("xlsx"))) as package: + parts = {name: package.read(name).decode() for name in package.namelist() if name.startswith("xl/tables/")} + + sales = next(xml for xml in parts.values() if 'name="Sales"' in xml) + stock = next(xml for xml in parts.values() if 'name="Stock"' in xml) + assert '= rows // 8] + finally: + gc.set_debug(0) + gc.garbage.clear() + gc.enable() + + assert leaked == [] diff --git a/python/tests/test_writer_handle.py b/python/tests/test_writer_handle.py index 4078ad99..cf572db8 100644 --- a/python/tests/test_writer_handle.py +++ b/python/tests/test_writer_handle.py @@ -4,7 +4,15 @@ import pytest -from excelreader import CellType, ColumnSpec, ColumnType, open_workbook, open_writer +from excelreader import ( + CellType, + ColumnSpec, + ColumnType, + ExcelReaderError, + WriteOptions, + open_workbook, + open_writer, +) def test_round_trips_typed_cells(tmp_path): @@ -284,3 +292,93 @@ def test_an_integer_beyond_64_bits_is_refused_not_wrapped(tmp_path, value): writer.write_row(["fine", value]) writer.write_row([2**63 - 1, -(2**63)]) writer.end_sheet() + + +_QTY_SCHEMA = [ColumnSpec(ColumnType.STRING, name="name"), ColumnSpec(ColumnType.I64, name="qty", nullable=True)] +_QTY_TYPES = [spec.type for spec in _QTY_SCHEMA] + + +def _qty_table(tmp_path, rows): + source = tmp_path / "source.csv" + source.write_text("name,qty\n" + "".join(f"{name},{qty}\n" for name, qty in rows), encoding="utf-8") + with open_workbook(source) as workbook: + return workbook.sheets[0].parse(_QTY_SCHEMA) + + +def _parsed(sheet): + table = sheet.parse(_QTY_SCHEMA) + return list(table.columns[0]), [ + table.columns[1][i] if table.validity[1] is None or table.validity[1][i >> 3] >> (i & 7) & 1 else None + for i in range(table.row_count) + ] + + +@pytest.mark.parametrize( + ("extension", "options"), + [ + ("xlsx", None), + ("xlsb", None), + ("xlsx", WriteOptions(degree_of_parallelism=4, batch_rows=2)), + ("xlsb", WriteOptions(degree_of_parallelism=4, batch_rows=2)), + ], +) +def test_write_sheet_writes_several_tables_beside_streamed_rows(tmp_path, extension, options): + sales = _qty_table(tmp_path, [("widget", 3), ("gadget", 7)]) + stock = _qty_table(tmp_path, [(f"item{i}", "" if i == 2 else i * 10) for i in range(7)]) + path = tmp_path / f"multi.{extension}" + + with open_writer(path, options=options) as writer: + writer.write_sheet("Vendas", sales, _QTY_TYPES) + writer.start_sheet("Resumo") + writer.write_row(["name", "qty"]) + writer.write_row(["total", 10]) + writer.end_sheet() + writer.write_sheet("Estoque", stock, _QTY_TYPES) + + with open_workbook(path) as workbook: + assert [sheet.name for sheet in workbook.sheets] == ["Vendas", "Resumo", "Estoque"] + assert _parsed(workbook.sheets[0]) == (["widget", "gadget"], [3, 7]) + assert _parsed(workbook.sheets[1]) == (["total"], [10]) + assert _parsed(workbook.sheets[2]) == ( + [f"item{i}" for i in range(7)], + [0, 10, None, 30, 40, 50, 60], + ) + + +def test_write_sheet_raises_while_a_streamed_sheet_is_open_and_the_writer_stays_usable(tmp_path): + table = _qty_table(tmp_path, [("widget", 3)]) + path = tmp_path / "open.xlsx" + + with open_writer(path) as writer: + writer.start_sheet("Streamed") + with pytest.raises(ExcelReaderError, match="already open"): + writer.write_sheet("Table", table, _QTY_TYPES) + writer.end_sheet() + writer.write_sheet("Table", table, _QTY_TYPES) + + with open_workbook(path) as workbook: + assert [sheet.name for sheet in workbook.sheets] == ["Streamed", "Table"] + + +def test_write_sheet_rejects_a_typed_table_without_types(tmp_path): + table = _qty_table(tmp_path, [("widget", 3)]) + with open_writer(tmp_path / "untyped.xlsx") as writer: + with pytest.raises(TypeError, match="types"): + writer.write_sheet("Data", table) + writer.write_sheet("Data", table, _QTY_TYPES) + + +def test_write_sheet_writes_arrow_data_to_a_bytesio(): + import io + + pa = pytest.importorskip("pyarrow") + + buffer = io.BytesIO() + with open_writer(buffer, "xlsx") as writer: + writer.write_sheet("A", pa.table({"name": ["widget"], "qty": [3]})) + writer.write_sheet("B", pa.RecordBatch.from_pydict({"name": ["gadget", "gizmo"], "qty": [7, 9]})) + + with open_workbook(buffer.getvalue(), format="xlsx") as workbook: + assert [sheet.name for sheet in workbook.sheets] == ["A", "B"] + assert _parsed(workbook.sheets[0]) == (["widget"], [3]) + assert _parsed(workbook.sheets[1]) == (["gadget", "gizmo"], [7, 9]) diff --git a/rust/excelreader/README.md b/rust/excelreader/README.md index 4dc6833c..768ee325 100644 --- a/rust/excelreader/README.md +++ b/rust/excelreader/README.md @@ -148,6 +148,31 @@ let tables = std::thread::scope(|scope| { # Ok::<(), excelreader::Error>(()) ``` +### Tables + +`Workbook::tables` lists the workbook's Excel tables ("Format as Table" ranges) as `TableInfo`: +name, the sheet holding it, and its zero-based range, header and totals rows included. +`Workbook::table(index)` and `table_by_name(name)` (ignoring case) return a `Sheet` bounded to the +table, so every read works on it and starts at the table's header row: + +```rust,no_run +# use excelreader::{ReadOptions, Workbook}; +# #[derive(Default, excelreader::ExcelMapper)] +# struct Sale { #[excel(name = "Units")] units: i64 } +let workbook = Workbook::open("report.xlsx")?; +for table in workbook.tables()? { + println!("{} on sheet {}: rows {}..={}", table.name, table.sheet, table.first_row, table.last_row); +} +if let Some(sales) = workbook.table_by_name("Sales")? { + let rows = sales.parse::(&ReadOptions::new())?; + println!("{} sales", rows.len()); +} +# Ok::<(), excelreader::Error>(()) +``` + +Cells keep the column indexes of the sheet holding the table, and the table's `Sheet::name` is that +sheet's. A table is read on one thread whatever the degree of parallelism asks for. + ### Encrypted workbooks `OpenOptions::password` unlocks a password-protected OOXML workbook (.xlsx/.xlsb/.xlsm): @@ -342,7 +367,8 @@ does not recognize is an error - a file being created has no signature bytes to For XLSX and XLSB, `degree_of_parallelism` formats and compresses batches of `batch_rows` rows on several threads: `0` uses every processor, `1` writes serially, and leaving it unset is serial. Unset `batch_rows` keeps the library default of 8,192. With shared strings on, two parallel runs -give the same content in files that differ byte for byte. XLS, CSV and `WriterHandle` ignore both. +give the same content in files that differ byte for byte. XLS, CSV and `WriterHandle`'s per-cell +writes ignore both. ```rust,no_run use excelreader::{write_columns, Column, ColumnData, WriteOptions}; @@ -397,6 +423,29 @@ returns `Err` rather than corrupting output. A handle from `new` writes the whol writer once, in `finish`. Dropping a handle without finishing it closes it on a best-effort basis, discarding any error, and writes nothing to a writer. +To write several sheets from whole tables, call the handle's `write_sheet` or `write_columns` once +per sheet. They take what the free functions take, plus the sheet's name, and write it in one call +instead of one per cell, using the handle's `degree_of_parallelism` and `batch_rows`. Between sheets +they mix with `start_sheet..end_sheet`; while a sheet is open they return `Err`. + +```rust,no_run +use excelreader::{WriteOptions, WriterHandle}; +# #[derive(Default, excelreader::ExcelMapper)] +# struct Row { #[excel(name = "Name")] name: String } +# let sales = vec![Row::default()]; +# let stock = vec![Row::default()]; + +let mut handle = WriterHandle::create("report.xlsx", &WriteOptions::new().degree_of_parallelism(0))?; +handle.write_sheet("Sales", &sales, None)?; +handle.write_sheet("Stock", &stock, None)?; +handle.finish()?; +# Ok::<(), excelreader::Error>(()) +``` + +Passing `Some(&TableOptions::new("Sales"))` writes the sheet as an Excel table, with filter buttons +and a table style (XLSX and XLSB only, every column named). The name must be unique in the +workbook; the style defaults to `TableStyleMedium2`, and `.style(None)` leaves the table unstyled. + ## Reading rows one at a time `Sheet::rows` returns a cursor over that sheet. Each row borrows a buffer the cursor diff --git a/rust/excelreader/excelreader.def b/rust/excelreader/excelreader.def index 454ceb9e..18e44e32 100644 --- a/rust/excelreader/excelreader.def +++ b/rust/excelreader/excelreader.def @@ -11,6 +11,10 @@ EXPORTS xl_is_date1904 xl_sheet_visibility_at xl_sheet_index + xl_table_count + xl_table_info_at + xl_table_name_at + xl_table_index xl_rows_open xl_rows_close xl_rows_next @@ -47,6 +51,7 @@ EXPORTS xl_write_null xl_end_row xl_end_sheet + xl_write_sheet xl_close_write_handle xl_write_handle_bytes xl_csv_aggregate_file diff --git a/rust/excelreader/src/aggregate.rs b/rust/excelreader/src/aggregate.rs index c54a03da..8ce0fc79 100644 --- a/rust/excelreader/src/aggregate.rs +++ b/rust/excelreader/src/aggregate.rs @@ -149,6 +149,15 @@ impl Shared<'_, A, F> { } PANIC_STATUS } + + /// Runs one user callback: its error code passes through and a panic is stored as above. + fn guard(&self, body: impl FnOnce() -> Result<(), i32>) -> i32 { + match catch_unwind(AssertUnwindSafe(body)) { + Ok(Ok(())) => XL_OK, + Ok(Err(code)) => code, + Err(payload) => self.store_panic(payload), + } + } } /// # Safety @@ -187,13 +196,9 @@ where let shared = unsafe { &*(user_data as *const Shared<'_, A, F>) }; let accumulator = unsafe { &mut (*(state as *mut State)).value }; let row = unsafe { &*row }; - match catch_unwind(AssertUnwindSafe(|| { + shared.guard(|| { accumulator.accumulate(unsafe { RowRef::from_decoded(row.cells, row.cell_count) }) - })) { - Ok(Ok(())) => XL_OK, - Ok(Err(code)) => code, - Err(payload) => shared.store_panic(payload), - } + }) } /// # Safety @@ -210,11 +215,7 @@ where let shared = unsafe { &*(user_data as *const Shared<'_, A, F>) }; let accumulator = unsafe { &mut (*(acc as *mut State)).value }; let other = unsafe { &mut (*(next as *mut State)).value }; - match catch_unwind(AssertUnwindSafe(|| accumulator.combine(other))) { - Ok(Ok(())) => XL_OK, - Ok(Err(code)) => code, - Err(payload) => shared.store_panic(payload), - } + shared.guard(|| accumulator.combine(other)) } /// # Safety @@ -246,13 +247,7 @@ where { let shared = unsafe { &*(user_data as *const Shared<'_, A, F>) }; let accumulator = unsafe { &mut (*(state as *mut State)).value }; - match catch_unwind(AssertUnwindSafe(|| { - accumulator.accumulate(unsafe { RowRef::from_views(cells, count) }) - })) { - Ok(Ok(())) => XL_OK, - Ok(Err(code)) => code, - Err(payload) => shared.store_panic(payload), - } + shared.guard(|| accumulator.accumulate(unsafe { RowRef::from_views(cells, count) })) } /// # Safety @@ -269,11 +264,7 @@ where let shared = unsafe { &*(user_data as *const Shared<'_, A, F>) }; let accumulator = unsafe { &mut (*(acc as *mut State)).value }; let other = unsafe { &mut (*(next as *mut State)).value }; - match catch_unwind(AssertUnwindSafe(|| accumulator.combine(other))) { - Ok(Ok(())) => XL_OK, - Ok(Err(code)) => code, - Err(payload) => shared.store_panic(payload), - } + shared.guard(|| accumulator.combine(other)) } fn raw_view_aggregation(shared: &Shared<'_, A, F>) -> XlAggregation diff --git a/rust/excelreader/src/ffi.rs b/rust/excelreader/src/ffi.rs index 80a922e9..978a9fc2 100644 --- a/rust/excelreader/src/ffi.rs +++ b/rust/excelreader/src/ffi.rs @@ -3,6 +3,18 @@ use std::os::raw::{c_int, c_void}; +/// Native memory as a slice; empty when `ptr` is null or `len` is zero or negative. +/// +/// # Safety +/// `ptr` must be null or point to `len` live `E`s that outlive `'a`. +pub(crate) unsafe fn native_slice<'a, E>(ptr: *const E, len: impl TryInto) -> &'a [E] { + let len = len.try_into().unwrap_or(0); + if ptr.is_null() || len == 0 { + return &[]; + } + std::slice::from_raw_parts(ptr, len) +} + pub const XL_OK: i32 = 0; pub const XL_EOF: i32 = -1; pub const XL_BUFFER_TOO_SMALL: i32 = -2; @@ -42,6 +54,18 @@ pub const XL_SHEET_VISIBLE: i32 = 0; pub const XL_SHEET_HIDDEN: i32 = 1; pub const XL_SHEET_VERY_HIDDEN: i32 = 2; +/// `XL_TABLE_SHEET`: the `sheet` value that makes a read call read table `table`. +#[must_use] +pub const fn xl_table_sheet(table: i32) -> i32 { + -2 - table +} + +pub const XL_TABLE_NO_STYLE: i32 = 1; +pub const XL_TABLE_NO_ROW_STRIPES: i32 = 2; +pub const XL_TABLE_COLUMN_STRIPES: i32 = 4; +pub const XL_TABLE_FIRST_COLUMN: i32 = 8; +pub const XL_TABLE_LAST_COLUMN: i32 = 16; + pub const XL_FORMAT_AUTO: i32 = 0; pub const XL_FORMAT_XLS: i32 = 1; pub const XL_FORMAT_XLSX: i32 = 2; @@ -138,6 +162,31 @@ pub struct XlWriteOptions { pub batch_rows: i32, } +/// Mirrors `xl_table_info`. +#[repr(C)] +#[derive(Clone, Copy, Debug, Default)] +pub struct XlTableInfo { + pub sheet: i32, + pub first_row: i32, + pub last_row: i32, + pub first_column: i32, + pub last_column: i32, + pub header_row_count: i32, + pub totals_row_count: i32, +} + +/// Mirrors `xl_sheet_options`; field order is the C struct's, padding included. +#[repr(C)] +#[derive(Clone, Copy, Debug)] +pub struct XlSheetOptions { + pub struct_size: i32, + pub table_name_len: i32, + pub table_name: *const u8, + pub table_style_len: i32, + pub table_style: *const u8, + pub table_flags: i32, +} + #[repr(C)] pub struct XlColumnSpec { pub names: *const *const u8, @@ -325,6 +374,23 @@ extern "C" { pub fn xl_is_date1904(handle: *mut XlWorkbook, out_flag: *mut i32) -> c_int; + pub fn xl_table_count(handle: *mut XlWorkbook, out_count: *mut i32) -> c_int; + pub fn xl_table_info_at(handle: *mut XlWorkbook, index: i32, out_info: *mut XlTableInfo) -> c_int; + pub fn xl_table_name_at( + handle: *mut XlWorkbook, + index: i32, + buffer: *mut u8, + capacity: i32, + out_len: *mut i32, + ) -> c_int; + /// Matches ignoring case; a name that matches no table is `XL_OK` with `-1`. + pub fn xl_table_index( + handle: *mut XlWorkbook, + name: *const u8, + name_len: i32, + out_index: *mut i32, + ) -> c_int; + pub fn xl_parse_typed( handle: *mut XlWorkbook, sheet: i32, @@ -461,6 +527,14 @@ extern "C" { pub fn xl_end_row(handle: *mut XlWriterHandle) -> c_int; pub fn xl_end_sheet(handle: *mut XlWriterHandle) -> c_int; + pub fn xl_write_sheet( + handle: *mut XlWriterHandle, + name: *const u8, + name_len: i32, + specs: *const XlColumnSpec, + table: *const XlTable, + options: *const XlSheetOptions, + ) -> c_int; pub fn xl_close_write_handle(handle: *mut XlWriterHandle) -> c_int; /// Reads back everything written so far to a handle opened by `xl_open_write_handle_to_memory`. diff --git a/rust/excelreader/src/lib.rs b/rust/excelreader/src/lib.rs index fed98411..cad8f964 100644 --- a/rust/excelreader/src/lib.rs +++ b/rust/excelreader/src/lib.rs @@ -27,11 +27,11 @@ pub use source::Source; pub use temporal::{Date, Time, Timestamp}; pub use workbook::{ ColumnType, ExcelMapper, InferredColumn, Sheet, SheetVisibility, Table, TableBatches, - TableView, TableViewIter, TypedBatches, Workbook, + TableInfo, TableView, TableViewIter, TypedBatches, Workbook, }; pub use writer::{ encrypt_package, write_columns, write_columns_to, write_sheet, write_sheet_to, Column, - ColumnData, ExcelWriter, + ColumnData, ExcelWriter, TableOptions, }; pub use writer_handle::WriterHandle; diff --git a/rust/excelreader/src/rows.rs b/rust/excelreader/src/rows.rs index 257b9734..22461de4 100644 --- a/rust/excelreader/src/rows.rs +++ b/rust/excelreader/src/rows.rs @@ -3,8 +3,8 @@ use std::borrow::Cow; use crate::ffi::{ - self, XlCellView, XlRowCell, XL_CELL_BOOL, XL_CELL_DATE, XL_CELL_EMPTY, XL_CELL_ERROR, - XL_CELL_FORMULA, XL_CELL_NUMBER, XL_CELL_STRING, + self, native_slice, XlCellView, XlRowCell, XL_CELL_BOOL, XL_CELL_DATE, XL_CELL_EMPTY, + XL_CELL_ERROR, XL_CELL_FORMULA, XL_CELL_NUMBER, XL_CELL_STRING, }; use crate::{Error, ErrorKind}; @@ -125,15 +125,10 @@ impl<'a> RowRef<'a> { /// `cells` must point to `count` initialized `XlRowCell` values whose `value` pointers stay /// valid for `'a`. pub(crate) unsafe fn from_decoded(cells: *const XlRowCell, count: i32) -> RowRef<'a> { - let len = if count > 0 { count as usize } else { 0 }; - let slice = if len == 0 || cells.is_null() { - &[][..] - } else { - unsafe { std::slice::from_raw_parts(cells, len) } - }; + let slice = unsafe { native_slice(cells, count) }; RowRef { backing: RowBacking::Decoded(slice), - len, + len: slice.len(), } } @@ -143,12 +138,7 @@ impl<'a> RowRef<'a> { /// `cells` must point to `count` initialized `XlCellView` values whose `value` pointers stay /// valid for `'a`. pub(crate) unsafe fn from_views(cells: *const XlCellView, count: i32) -> RowRef<'a> { - let len = if count > 0 { count as usize } else { 0 }; - let slice = if len == 0 || cells.is_null() { - &[][..] - } else { - unsafe { std::slice::from_raw_parts(cells, len) } - }; + let slice = unsafe { native_slice(cells, count) }; RowRef { backing: RowBacking::Views(slice), len: slice.len(), @@ -274,11 +264,7 @@ fn cell_from_blob(blob: &[u8], offset: usize) -> Option<(CellRef<'_>, usize)> { fn cell_from_decoded(raw: &XlRowCell) -> Option> { let cell_type = CellType::from_raw(raw.cell_type)?; - let value = if raw.value.is_null() || raw.value_len <= 0 { - &[][..] - } else { - unsafe { std::slice::from_raw_parts(raw.value, raw.value_len as usize) } - }; + let value = unsafe { native_slice(raw.value, raw.value_len) }; Some(CellRef { column: raw.column, cell_type, @@ -289,11 +275,7 @@ fn cell_from_decoded(raw: &XlRowCell) -> Option> { fn cell_from_view(raw: &XlCellView) -> Option> { let cell_type = CellType::from_raw(raw.cell_type)?; - let value = if raw.value.is_null() || raw.value_len <= 0 { - &[][..] - } else { - unsafe { std::slice::from_raw_parts(raw.value, raw.value_len as usize) } - }; + let value = unsafe { native_slice(raw.value, raw.value_len) }; Some(CellRef { column: raw.column, cell_type, diff --git a/rust/excelreader/src/workbook.rs b/rust/excelreader/src/workbook.rs index 6148fb51..dffa92b2 100644 --- a/rust/excelreader/src/workbook.rs +++ b/rust/excelreader/src/workbook.rs @@ -1,6 +1,6 @@ use crate::ffi::{ - self, XlColumn, XlColumnSpec, XlInferredSchema, XlOpenOptions, XlTable, XlWorkbook, - XL_BUFFER_TOO_SMALL, XL_EOF, XL_OK, + self, native_slice, XlColumn, XlColumnSpec, XlInferredSchema, XlOpenOptions, XlTable, + XlWorkbook, XL_BUFFER_TOO_SMALL, XL_EOF, XL_OK, }; use crate::{ AllRows, Column, ColumnData, Date, Error, ErrorKind, OpenOptions, ReadOptions, RowCursor, @@ -12,11 +12,10 @@ use std::path::Path; pub(crate) fn last_error(code: i32) -> Error { unsafe { let mut len: i32 = 0; - let ptr = ffi::xl_last_error_ptr(&mut len); - let message = if ptr.is_null() || len <= 0 { + let bytes = native_slice(ffi::xl_last_error_ptr(&mut len), len); + let message = if bytes.is_empty() { "unknown error".to_string() } else { - let bytes = std::slice::from_raw_parts(ptr, len as usize); String::from_utf8_lossy(bytes).into_owned() }; Error::from_status(code, message) @@ -52,14 +51,9 @@ pub(crate) fn len_i32(len: usize, what: &str) -> Result { } /// Copies a native `XlBuffer` into an owned `Vec` and releases the native allocation via -/// `xl_free_buffer`. `buffer.data` may be null (an empty result), which `from_raw_parts` cannot -/// take even for a zero-length slice. +/// `xl_free_buffer`. `buffer.data` may be null (an empty result). pub(crate) fn buffer_to_vec(mut buffer: ffi::XlBuffer) -> Vec { - let bytes = if buffer.data.is_null() || buffer.len <= 0 { - Vec::new() - } else { - unsafe { std::slice::from_raw_parts(buffer.data, buffer.len as usize).to_vec() } - }; + let bytes = unsafe { native_slice(buffer.data, buffer.len) }.to_vec(); unsafe { ffi::xl_free_buffer(&mut buffer); } @@ -209,6 +203,57 @@ impl Workbook { })) } + /// Every Excel table, in workbook order. + pub fn tables(&self) -> Result, Error> { + let mut count: i32 = 0; + check(unsafe { ffi::xl_table_count(self.handle, &mut count) })?; + (0..count).map(|index| self.table_info(index)).collect() + } + + fn table_info(&self, index: i32) -> Result { + let mut raw = ffi::XlTableInfo::default(); + check(unsafe { ffi::xl_table_info_at(self.handle, index, &mut raw) })?; + let name = self.fill_string(|handle, buffer, capacity, out_len| unsafe { + ffi::xl_table_name_at(handle, index, buffer, capacity, out_len) + })?; + Ok(TableInfo { + name, + index, + sheet: raw.sheet, + first_row: raw.first_row, + last_row: raw.last_row, + first_column: raw.first_column, + last_column: raw.last_column, + header_row_count: raw.header_row_count, + totals_row_count: raw.totals_row_count, + }) + } + + /// Table `index` read as a sheet: every [`Sheet`] read covers only the table's range, header + /// row first, with the column indexes of the sheet holding it. Its [`Sheet::name`] and + /// [`Sheet::visibility`] are that sheet's. An index outside the workbook is an error here. + pub fn table(&self, index: i32) -> Result, Error> { + let mut raw = ffi::XlTableInfo::default(); + check(unsafe { ffi::xl_table_info_at(self.handle, index, &mut raw) })?; + Ok(Sheet { + workbook: self, + index: ffi::xl_table_sheet(index), + }) + } + + /// The table called `name`, compared without regard to case, read as in [`table`](Self::table). + /// `Ok(None)` when no table matches. + pub fn table_by_name(&self, name: &str) -> Result>, Error> { + let mut index: i32 = -1; + check(unsafe { + ffi::xl_table_index(self.handle, name.as_ptr(), len_i32(name.len(), "name")?, &mut index) + })?; + Ok((index >= 0).then_some(Sheet { + workbook: self, + index: ffi::xl_table_sheet(index), + })) + } + /// Shared two-pass buffer dance for the `xl_*` functions that write a UTF-8 name into a caller /// buffer and report the required capacity through `XL_BUFFER_TOO_SMALL`. fn fill_string( @@ -283,6 +328,26 @@ impl OpenOptions { } } +/// One Excel table's place in a workbook. Rows and columns are zero-based and include the header +/// and totals rows. Read its cells through [`Workbook::table`]. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct TableInfo { + pub name: String, + /// Position among the workbook's tables, as [`Workbook::table`] takes it. + pub index: i32, + /// Index of the sheet holding the table. + pub sheet: i32, + pub first_row: i32, + pub last_row: i32, + pub first_column: i32, + pub last_column: i32, + /// 1, or 0 when the table's header row is turned off. + pub header_row_count: i32, + /// 1 when the table shows a totals row, otherwise 0. + pub totals_row_count: i32, +} + /// How a sheet is shown in the application's tab bar. #[derive(Clone, Copy, Debug, PartialEq, Eq)] #[non_exhaustive] @@ -301,7 +366,8 @@ pub struct Sheet<'w> { } impl<'w> Sheet<'w> { - /// Zero-based position in the workbook. + /// Zero-based position in the workbook. A sheet from [`Workbook::table`] reports the table's + /// encoded index, which only read calls understand. #[must_use] pub fn index(&self) -> i32 { self.index @@ -443,24 +509,9 @@ impl<'w> Sheet<'w> { schema: &[InferredColumn], options: &ReadOptions, ) -> Result, Error> { - let specs = schema_specs(schema); - let mut reader: *mut ffi::XlTypedReader = std::ptr::null_mut(); - check(unsafe { - ffi::xl_typed_reader_open( - self.workbook.handle, - self.index, - specs.specs.as_ptr(), - specs.specs.len() as i32, - options.header_row, - options.batch_size, - &mut reader, - ) - })?; Ok(TableBatches { - reader, + batches: BatchReader::open(self, &schema_specs(schema).specs, options)?, schema: schema.to_vec(), - done: false, - _workbook: PhantomData, }) } @@ -471,23 +522,9 @@ impl<'w> Sheet<'w> { options: &ReadOptions, ) -> Result, Error> { let arena = build_specs::(); - let mut reader: *mut ffi::XlTypedReader = std::ptr::null_mut(); - check(unsafe { - ffi::xl_typed_reader_open( - self.workbook.handle, - self.index, - arena.specs.as_ptr(), - arena.specs.len() as i32, - options.header_row, - options.batch_size, - &mut reader, - ) - })?; Ok(TypedBatches { - reader, + batches: BatchReader::open(self, &arena.specs, options)?, bindings: arena.bindings, - done: false, - _workbook: PhantomData, }) } @@ -593,11 +630,7 @@ impl ColumnType { /// # Safety /// `schema` must be one `xl_infer_schema` returned `XL_OK` for, not yet passed to `xl_free_schema`. unsafe fn copy_inferred(schema: &XlInferredSchema) -> Result, Error> { - if schema.columns.is_null() || schema.column_count <= 0 { - return Ok(Vec::new()); - } - let specs = std::slice::from_raw_parts(schema.columns, schema.column_count as usize); - specs + native_slice(schema.columns, schema.column_count) .iter() .map(|spec| { let column_type = ColumnType::from_raw(spec.r#type).ok_or_else(|| { @@ -612,14 +645,8 @@ unsafe fn copy_inferred(schema: &XlInferredSchema) -> Result let name = if spec.name_count <= 0 || spec.names.is_null() || spec.name_lens.is_null() { None } else { - let name_ptr = *spec.names; - let name_len = *spec.name_lens; - if name_ptr.is_null() || name_len <= 0 { - None - } else { - let bytes = std::slice::from_raw_parts(name_ptr, name_len as usize); - Some(String::from_utf8_lossy(bytes).into_owned()) - } + let bytes = native_slice(*spec.names, *spec.name_lens); + (!bytes.is_empty()).then(|| String::from_utf8_lossy(bytes).into_owned()) }; Ok(InferredColumn { name, @@ -930,11 +957,7 @@ impl Table { /// Takes ownership of `table`, freeing it and failing if it does not have one column of a known /// type and of `row_count` rows per `schema` entry. fn new(mut table: XlTable, schema: &[InferredColumn], source: &str) -> Result { - let columns = if table.columns.is_null() { - &[][..] - } else { - unsafe { std::slice::from_raw_parts(table.columns, table.column_count.max(0) as usize) } - }; + let columns = unsafe { native_slice(table.columns, table.column_count) }; let consistent = table.row_count >= 0 && columns.len() == schema.len() && columns @@ -988,15 +1011,6 @@ impl Table { } } -/// # Safety -/// `ptr` must be null or point to `len` live `E`s that outlive `'a`. -unsafe fn native_slice<'a, E>(ptr: *const E, len: usize) -> &'a [E] { - if ptr.is_null() || len == 0 { - return &[]; - } - std::slice::from_raw_parts(ptr, len) -} - /// # Safety /// `raw` must be a column of a native table whose type [`Table::new`] checked. unsafe fn column_data(raw: &XlColumn) -> ColumnData<'_> { @@ -1037,22 +1051,48 @@ impl std::fmt::Debug for Table { } } -/// Iterator over the batches of one chunked typed read. Closes the native reader on `Drop`. -pub struct TypedBatches<'a, T: ExcelMapper> { +/// The native half of a chunked typed read: one `xl_typed_reader`, closed on `Drop`. +struct BatchReader<'a> { reader: *mut ffi::XlTypedReader, - bindings: Vec>, done: bool, _workbook: PhantomData<&'a Workbook>, } // SAFETY: a typed reader owns its native enumerator; ABI v6 lets it be used from any one thread // at a time. -unsafe impl Send for TypedBatches<'_, T> {} +unsafe impl Send for BatchReader<'_> {} -impl Iterator for TypedBatches<'_, T> { - type Item = Result, Error>; +impl<'a> BatchReader<'a> { + fn open( + sheet: &Sheet<'a>, + specs: &[XlColumnSpec], + options: &ReadOptions, + ) -> Result { + let mut reader: *mut ffi::XlTypedReader = std::ptr::null_mut(); + check(unsafe { + ffi::xl_typed_reader_open( + sheet.workbook.handle, + sheet.index, + specs.as_ptr(), + specs.len() as i32, + options.header_row, + options.batch_size, + &mut reader, + ) + })?; + Ok(BatchReader { + reader, + done: false, + _workbook: PhantomData, + }) + } - fn next(&mut self) -> Option { + /// Hands the next batch's table to `wrap`, which takes ownership of it. Stops after the first + /// error, native or from `wrap`. + fn next_with( + &mut self, + wrap: impl FnOnce(XlTable) -> Result, + ) -> Option> { if self.done { return None; } @@ -1068,68 +1108,58 @@ impl Iterator for TypedBatches<'_, T> { return Some(Err(last_error(status))); } - // ponytail: one small Vec of function pointers cloned per batch, not per row. - let view = TableView::new(table, self.bindings.clone(), "xl_typed_reader_next"); - self.done = view.is_err(); - Some(view) + let batch = wrap(table); + self.done = batch.is_err(); + Some(batch) } } -impl Drop for TypedBatches<'_, T> { +impl Drop for BatchReader<'_> { fn drop(&mut self) { unsafe { ffi::xl_typed_reader_close(self.reader) }; } } +/// Iterator over the batches of one chunked typed read. Closes the native reader on `Drop`. +pub struct TypedBatches<'a, T: ExcelMapper> { + batches: BatchReader<'a>, + bindings: Vec>, +} + +impl Iterator for TypedBatches<'_, T> { + type Item = Result, Error>; + + fn next(&mut self) -> Option { + let bindings = &self.bindings; + // ponytail: one small Vec of function pointers cloned per batch, not per row. + self.batches + .next_with(|table| TableView::new(table, bindings.clone(), "xl_typed_reader_next")) + } +} + /// Progress only - the reader handle is opaque. Needed for `expect_err` on a `Result`. impl std::fmt::Debug for TypedBatches<'_, T> { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("TypedBatches") .field("columns", &self.bindings.len()) - .field("done", &self.done) + .field("done", &self.batches.done) .finish_non_exhaustive() } } /// [`TypedBatches`]' counterpart for [`Sheet::parse_batches_with_schema`], yielding [`Table`]s. pub struct TableBatches<'a> { - reader: *mut ffi::XlTypedReader, + batches: BatchReader<'a>, schema: Vec, - done: bool, - _workbook: PhantomData<&'a Workbook>, } -// SAFETY: as for TypedBatches. -unsafe impl Send for TableBatches<'_> {} - impl Iterator for TableBatches<'_> { type Item = Result; fn next(&mut self) -> Option { - if self.done { - return None; - } - - let mut table = empty_table(); - let status = unsafe { ffi::xl_typed_reader_next(self.reader, &mut table) }; - if status == XL_EOF { - self.done = true; - return None; - } - if status != XL_OK { - self.done = true; - return Some(Err(last_error(status))); - } - - let batch = Table::new(table, &self.schema, "xl_typed_reader_next"); - self.done = batch.is_err(); - Some(batch) - } -} - -impl Drop for TableBatches<'_> { - fn drop(&mut self) { - unsafe { ffi::xl_typed_reader_close(self.reader) }; + let schema = &self.schema; + self.batches + .next_with(|table| Table::new(table, schema, "xl_typed_reader_next")) } } @@ -1137,7 +1167,7 @@ impl std::fmt::Debug for TableBatches<'_> { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("TableBatches") .field("columns", &self.schema.len()) - .field("done", &self.done) + .field("done", &self.batches.done) .finish_non_exhaustive() } } diff --git a/rust/excelreader/src/writer.rs b/rust/excelreader/src/writer.rs index 09f05976..bb5d78a7 100644 --- a/rust/excelreader/src/writer.rs +++ b/rust/excelreader/src/writer.rs @@ -128,6 +128,84 @@ impl<'a> Column<'a> { } } +/// Writes a sheet as an Excel table ("Format as Table"), through +/// [`WriterHandle::write_columns`](crate::WriterHandle::write_columns) or +/// [`write_sheet`](crate::WriterHandle::write_sheet). XLSX and XLSB only. +/// +/// ``` +/// use excelreader::TableOptions; +/// +/// let table = TableOptions::new("Sales").style(Some("TableStyleLight9")).column_stripes(true); +/// ``` +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct TableOptions { + name: String, + style: Option, + flags: i32, +} + +impl TableOptions { + /// A table called `name`: unique in the workbook, ignoring case, and valid as an Excel name. + /// Styled `TableStyleMedium2`, with banded rows. + #[must_use] + pub fn new(name: impl Into) -> Self { + TableOptions { + name: name.into(), + style: Some("TableStyleMedium2".to_string()), + flags: 0, + } + } + + /// A built-in table style such as `TableStyleLight9`, or `None` for no style. + #[must_use] + pub fn style(mut self, value: Option<&str>) -> Self { + self.style = value.map(str::to_string); + self + } + + /// Whether alternate rows are banded. Default `true`. + #[must_use] + pub fn row_stripes(self, value: bool) -> Self { + self.flag(ffi::XL_TABLE_NO_ROW_STRIPES, !value) + } + + /// Whether alternate columns are banded. Default `false`. + #[must_use] + pub fn column_stripes(self, value: bool) -> Self { + self.flag(ffi::XL_TABLE_COLUMN_STRIPES, value) + } + + /// Whether the first column gets the style's emphasis. Default `false`. + #[must_use] + pub fn first_column(self, value: bool) -> Self { + self.flag(ffi::XL_TABLE_FIRST_COLUMN, value) + } + + /// Whether the last column gets the style's emphasis. Default `false`. + #[must_use] + pub fn last_column(self, value: bool) -> Self { + self.flag(ffi::XL_TABLE_LAST_COLUMN, value) + } + + fn flag(mut self, bit: i32, set: bool) -> Self { + self.flags = if set { self.flags | bit } else { self.flags & !bit }; + self + } + + /// Borrows `self`'s strings, so the result must not outlive it. + pub(crate) fn to_raw(&self) -> Result { + let style_flag = if self.style.is_none() { ffi::XL_TABLE_NO_STYLE } else { 0 }; + Ok(ffi::XlSheetOptions { + struct_size: std::mem::size_of::() as i32, + table_name_len: len_i32(self.name.len(), "table name")?, + table_name: self.name.as_ptr(), + table_style_len: len_i32(self.style.as_ref().map_or(0, String::len), "table style")?, + table_style: self.style.as_ref().map_or(std::ptr::null(), |style| style.as_ptr()), + table_flags: self.flags | style_flag, + }) + } +} + fn invalid(message: String) -> Error { Error::new(ErrorKind::InvalidArgument, message) } @@ -192,15 +270,15 @@ fn validate(columns: &[Column<'_>]) -> Result { /// The ABI's column specs and table for `columns`, borrowing every buffer. `specs` and `table` /// point into the other three fields, so the whole value must outlive the FFI call. -struct RawTable { - specs: Vec, - table: XlTable, +pub(crate) struct RawTable { + pub(crate) specs: Vec, + pub(crate) table: XlTable, _columns: Vec, _names: Vec<*const u8>, _name_lens: Vec, } -fn raw_table(columns: &[Column<'_>]) -> Result { +pub(crate) fn raw_table(columns: &[Column<'_>]) -> Result { let row_count = validate(columns)?; let names: Vec<*const u8> = columns .iter() @@ -359,7 +437,7 @@ pub struct OwnedColumn { } impl OwnedColumn { - fn as_column(&self) -> Column<'_> { + pub(crate) fn as_column(&self) -> Column<'_> { let data = match &self.data { OwnedColumnData::Str { offsets, data } => ColumnData::Str { offsets, data }, OwnedColumnData::I64(v) => ColumnData::I64(v), diff --git a/rust/excelreader/src/writer_handle.rs b/rust/excelreader/src/writer_handle.rs index da518240..bac94fcf 100644 --- a/rust/excelreader/src/writer_handle.rs +++ b/rust/excelreader/src/writer_handle.rs @@ -4,13 +4,14 @@ //! instead of materializing a whole table up front, it writes directly as each call arrives - one //! sheet and one row open at a time. Call order mirrors the C ABI (see `xl_writer_handle` in //! `excelreader.h`): `create`/`new`, then per sheet `start_sheet`..`end_sheet`, each containing -//! `start_row`..`end_row` with one `write_*` call per cell in between, left to right, then -//! `finish`. A call out of order returns an `Err` rather than corrupting output, and the handle +//! `start_row`..`end_row` with one `write_*` call per cell in between, left to right, or one +//! `write_columns`/`write_sheet` per whole sheet, then `finish`. A call out of order returns an `Err` rather than corrupting output, and the handle //! stays usable afterward - fix the call order and continue, or let `Drop` discard it. use crate::ffi::{self, XlWriterHandle}; use crate::workbook::{buffer_to_vec, check, check_abi_version, len_i32, path_str}; -use crate::{ColumnType, Date, Error, Time, Timestamp, WriteOptions}; +use crate::writer::{raw_table, OwnedColumn}; +use crate::{Column, ColumnType, Date, Error, ExcelWriter, TableOptions, Time, Timestamp, WriteOptions}; use std::io::Write; use std::path::Path; @@ -148,6 +149,45 @@ impl<'w> WriterHandle<'w> { check(unsafe { ffi::xl_end_sheet(self.handle) }) } + /// Writes `columns` as a whole sheet named `name`, as [`write_columns`](crate::write_columns) + /// does, in one call instead of one per cell; with `table`, as an Excel table (XLSX/XLSB only, + /// every column named). Uses the parallelism set in the handle's [`WriteOptions`]; its sheet + /// name does not apply. Fails while a sheet started with [`start_sheet`](Self::start_sheet) is + /// open. + pub fn write_columns( + &mut self, + name: &str, + columns: &[Column<'_>], + table: Option<&TableOptions>, + ) -> Result<(), Error> { + let name_len = len_i32(name.len(), "name")?; + let raw = raw_table(columns)?; + let options = table.map(TableOptions::to_raw).transpose()?; + check(unsafe { + ffi::xl_write_sheet( + self.handle, + name.as_ptr(), + name_len, + raw.specs.as_ptr(), + &raw.table, + options.as_ref().map_or(std::ptr::null(), |o| o), + ) + }) + } + + /// Writes `rows` as a whole sheet named `name`, as [`write_sheet`](crate::write_sheet) does, + /// on top of [`write_columns`](Self::write_columns). + pub fn write_sheet( + &mut self, + name: &str, + rows: &[T], + table: Option<&TableOptions>, + ) -> Result<(), Error> { + let owned = T::to_columns(rows)?; + let borrowed: Vec> = owned.iter().map(OwnedColumn::as_column).collect(); + self.write_columns(name, &borrowed, table) + } + /// Completes the workbook and releases the handle. For a handle from [`new`](Self::new), the /// whole workbook is then written to its `io::Write` in one `write_all`. pub fn finish(mut self) -> Result<(), Error> { diff --git a/rust/excelreader/tests/tables.rs b/rust/excelreader/tests/tables.rs new file mode 100644 index 00000000..836de328 --- /dev/null +++ b/rust/excelreader/tests/tables.rs @@ -0,0 +1,144 @@ +//! Excel tables: written through `WriterHandle` with `TableOptions`, read back through +//! `Workbook::tables`/`table`/`table_by_name`. + +use excelreader::{ + Column, ColumnData, ExcelMapper, Format, OpenOptions, ReadOptions, TableInfo, TableOptions, + Workbook, WriteOptions, WriterHandle, +}; + +#[derive(Default, Debug, Clone, PartialEq, ExcelMapper)] +struct Item { + #[excel(name = "name")] + name: String, + #[excel(name = "qty")] + qty: i64, +} + +fn item(name: &str, qty: i64) -> Item { + Item { + name: name.to_string(), + qty, + } +} + +fn workbook_with_tables(format: Format, degree: i32) -> Vec { + let mut bytes = Vec::new(); + let options = WriteOptions::new() + .format(format) + .degree_of_parallelism(degree) + .batch_rows(1); + let mut handle = WriterHandle::new(&mut bytes, &options).expect("new must succeed"); + + handle.start_sheet("Notas").expect("start_sheet must succeed"); + handle.start_row().expect("start_row must succeed"); + handle.write_str(Some("antes")).expect("write_str must succeed"); + handle.end_row().expect("end_row must succeed"); + handle.end_sheet().expect("end_sheet must succeed"); + + let sales = [item("widget", 3), item("gadget", 7)]; + handle + .write_sheet("Vendas", &sales, Some(&TableOptions::new("Sales"))) + .expect("write_sheet as a table must succeed"); + + let offsets = [0i32, 1, 2, 3]; + let quantities = [10i64, 20, 30]; + let columns = [ + Column::new( + Some("name"), + ColumnData::Str { + offsets: &offsets, + data: b"abc", + }, + ), + Column::new(Some("qty"), ColumnData::I64(&quantities)), + ]; + let stock = TableOptions::new("Stock").style(None).row_stripes(false); + handle + .write_columns("Estoque", &columns, Some(&stock)) + .expect("write_columns as a table must succeed"); + handle.finish().expect("finish must succeed"); + bytes +} + +fn read_items(workbook: &Workbook, sheet: excelreader::Sheet<'_>) -> Vec { + let _ = workbook; + let table = sheet + .parse::(&ReadOptions::new()) + .expect("the table must parse"); + (0..table.len()) + .map(|row| { + let read = table.get(row).expect("the row must exist"); + item(&read.name, read.qty) + }) + .collect() +} + +#[test] +fn tables_written_through_the_handle_read_back() { + for (format, degree) in [ + (Format::Xlsx, 1), + (Format::Xlsb, 1), + (Format::Xlsx, 4), + (Format::Xlsb, 4), + ] { + let bytes = workbook_with_tables(format, degree); + let workbook = OpenOptions::new() + .format(format) + .open_bytes(&bytes) + .expect("the bytes finish wrote must open"); + + let tables = workbook.tables().expect("tables must list"); + let names: Vec<&str> = tables.iter().map(|t| t.name.as_str()).collect(); + assert_eq!(names, ["Sales", "Stock"], "{format:?} at degree {degree}"); + let sales: &TableInfo = &tables[0]; + assert_eq!( + ( + sales.index, + sales.sheet, + sales.first_row, + sales.last_row, + sales.first_column, + sales.last_column, + sales.header_row_count, + sales.totals_row_count + ), + (0, 1, 0, 2, 0, 1, 1, 0) + ); + + let by_name = workbook + .table_by_name("sales") + .expect("lookup must succeed") + .expect("the table must be found ignoring case"); + assert_eq!(by_name.name().expect("name must read"), "Vendas"); + assert_eq!( + read_items(&workbook, by_name), + [item("widget", 3), item("gadget", 7)] + ); + assert_eq!( + read_items(&workbook, workbook.table(1).expect("table 1 must exist")), + [item("a", 10), item("b", 20), item("c", 30)] + ); + + assert!(workbook.table(2).is_err(), "a missing table index must fail"); + assert!(workbook + .table_by_name("missing") + .expect("lookup must succeed") + .is_none()); + } +} + +#[test] +fn a_table_is_refused_outside_xlsx_and_xlsb() { + let mut bytes = Vec::new(); + let options = WriteOptions::new().format(Format::Xls); + let mut handle = WriterHandle::new(&mut bytes, &options).expect("new must succeed"); + let rows = [item("widget", 3)]; + let error = handle + .write_sheet("Vendas", &rows, Some(&TableOptions::new("Sales"))) + .expect_err("XLS has no tables"); + assert!(error.to_string().contains("XLSX or XLSB"), "{error}"); + handle + .write_sheet("Vendas", &rows, None) + .expect("a plain sheet must still be written"); + handle.finish().expect("finish must succeed"); +} diff --git a/rust/excelreader/tests/writer_handle.rs b/rust/excelreader/tests/writer_handle.rs index ff5f29eb..8f09e801 100644 --- a/rust/excelreader/tests/writer_handle.rs +++ b/rust/excelreader/tests/writer_handle.rs @@ -3,8 +3,8 @@ use excelreader::ReadOptions; use excelreader::{ - Date, ErrorKind, ExcelMapper, Format, OpenOptions, Time, Timestamp, Workbook, WriteOptions, - WriterHandle, + Column, ColumnData, Date, ErrorKind, ExcelMapper, Format, OpenOptions, Time, Timestamp, + Workbook, WriteOptions, WriterHandle, }; use std::path::PathBuf; use std::sync::atomic::{AtomicU32, Ordering}; @@ -238,3 +238,143 @@ fn writer_handle_rejects_out_of_order_calls() { } std::fs::remove_file(&path).ok(); } + +#[derive(Default, Debug, Clone, PartialEq, ExcelMapper)] +struct Item { + #[excel(name = "name")] + name: String, + #[excel(name = "qty")] + qty: Option, +} + +fn item(name: &str, qty: Option) -> Item { + Item { + name: name.to_string(), + qty, + } +} + +fn read_items(workbook: &Workbook, index: i32) -> (String, Vec) { + let sheet = workbook.sheet(index).expect("the sheet must exist"); + let table = sheet + .parse::(&ReadOptions::new()) + .expect("the sheet must parse back"); + let rows = (0..table.len()) + .map(|row| { + let read = table.get(row).expect("the row must exist"); + item(&read.name, read.qty) + }) + .collect(); + (sheet.name().expect("name must read back"), rows) +} + +#[test] +fn write_sheet_and_write_columns_write_whole_sheets_beside_streamed_ones() { + for (format, degree) in [ + (Format::Xlsx, 1), + (Format::Xlsb, 1), + (Format::Xlsx, 4), + (Format::Xlsb, 4), + ] { + let mut bytes = Vec::new(); + let options = WriteOptions::new() + .format(format) + .degree_of_parallelism(degree) + .batch_rows(2); + let mut handle = WriterHandle::new(&mut bytes, &options).expect("new must succeed"); + + let sales = [item("widget", Some(3)), item("gadget", Some(7))]; + handle + .write_sheet("Vendas", &sales, None) + .expect("write_sheet must succeed"); + + handle + .start_sheet("Resumo") + .expect("start_sheet must succeed"); + handle.start_row().expect("start_row must succeed"); + handle + .write_str(Some("name")) + .expect("write_str must succeed"); + handle + .write_str(Some("qty")) + .expect("write_str must succeed"); + handle.end_row().expect("end_row must succeed"); + handle.start_row().expect("start_row must succeed"); + handle + .write_str(Some("total")) + .expect("write_str must succeed"); + handle.write_i64(Some(10)).expect("write_i64 must succeed"); + handle.end_row().expect("end_row must succeed"); + handle.end_sheet().expect("end_sheet must succeed"); + + let offsets = [0i32, 1, 2, 3, 4, 5]; + let quantities = [10i64, 20, 0, 40, 50]; + let validity = [0b1_1011u8]; + let columns = [ + Column::new( + Some("name"), + ColumnData::Str { + offsets: &offsets, + data: b"abcde", + }, + ), + Column::new(Some("qty"), ColumnData::I64(&quantities)).validity(&validity), + ]; + handle + .write_columns("Estoque", &columns, None) + .expect("write_columns must succeed"); + handle.finish().expect("finish must succeed"); + + let workbook = OpenOptions::new() + .open_bytes(&bytes) + .expect("the bytes finish wrote must open"); + assert_eq!( + read_items(&workbook, 0), + ("Vendas".to_string(), sales.to_vec()) + ); + assert_eq!( + read_items(&workbook, 1), + ("Resumo".to_string(), vec![item("total", Some(10))]) + ); + assert_eq!( + read_items(&workbook, 2), + ( + "Estoque".to_string(), + vec![ + item("a", Some(10)), + item("b", Some(20)), + item("c", None), + item("d", Some(40)), + item("e", Some(50)), + ] + ), + "{format:?} at degree {degree}" + ); + } +} + +#[test] +fn write_sheet_fails_while_a_streamed_sheet_is_open_and_the_handle_stays_usable() { + let mut bytes = Vec::new(); + let options = WriteOptions::new().format(Format::Xlsx); + let mut handle = WriterHandle::new(&mut bytes, &options).expect("new must succeed"); + let rows = [item("widget", Some(3))]; + + handle + .start_sheet("Aberta") + .expect("start_sheet must succeed"); + let error = handle + .write_sheet("Tabela", &rows, None) + .expect_err("a whole sheet cannot start while another is open"); + assert!(error.to_string().contains("already open"), "{error}"); + handle.end_sheet().expect("end_sheet must succeed"); + handle + .write_sheet("Tabela", &rows, None) + .expect("write_sheet must succeed once the sheet is ended"); + handle.finish().expect("finish must succeed"); + + let workbook = OpenOptions::new() + .open_bytes(&bytes) + .expect("the bytes finish wrote must open"); + assert_eq!(read_items(&workbook, 1).0, "Tabela"); +} diff --git a/src/ExcelReader.Core/Crypto/CfbContainer.cs b/src/ExcelReader.Core/Crypto/CfbContainer.cs index a08dda89..0e29ebc8 100644 --- a/src/ExcelReader.Core/Crypto/CfbContainer.cs +++ b/src/ExcelReader.Core/Crypto/CfbContainer.cs @@ -524,7 +524,7 @@ private static long SectorOffset(int sector, int sectorSize) internal readonly record struct DirectoryEntry(string Name, byte ObjectType, int StartSector, long Size); - private sealed class CfbStreamView : Stream + private sealed class CfbStreamView : ReadOnlyStream { [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", Justification = "Borrowed, not owned.")] private readonly Stream _source; @@ -541,12 +541,8 @@ internal CfbStreamView(Stream source, int[] chain, int sectorSize, long length) Length = length; } - public override bool CanRead => true; - public override bool CanSeek => true; - public override bool CanWrite => false; - public override long Length { get; } public override long Position @@ -606,20 +602,6 @@ public override long Seek(long offset, SeekOrigin origin) return _position; } - public override void Flush() - { - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - protected override void Dispose(bool disposing) { if (!_disposed && disposing) diff --git a/src/ExcelReader.Core/Crypto/DecryptedPackageStream.cs b/src/ExcelReader.Core/Crypto/DecryptedPackageStream.cs index 24c9b84e..fc1c2338 100644 --- a/src/ExcelReader.Core/Crypto/DecryptedPackageStream.cs +++ b/src/ExcelReader.Core/Crypto/DecryptedPackageStream.cs @@ -6,7 +6,7 @@ namespace ExcelReader.Core.Crypto { - internal sealed class DecryptedPackageStream : Stream + internal sealed class DecryptedPackageStream : ReadOnlyStream { private const int SegmentSize = 4096; private const int PrefixSize = 8; @@ -94,12 +94,8 @@ private static long ReadDeclaredLength(Stream view, ExcelReaderOptions options) return declared; } - public override bool CanRead => true; - public override bool CanSeek => true; - public override bool CanWrite => false; - public override long Length => _length; public override long Position @@ -112,10 +108,6 @@ public override long Position } } - public override void Flush() - { - } - public override int Read(byte[] buffer, int offset, int count) { return Read(buffer.AsSpan(offset, count)); @@ -196,16 +188,6 @@ public override long Seek(long offset, SeekOrigin origin) return _position; } - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - private void EnsureSegment(int index) { if (_cachedSegment == index) diff --git a/src/ExcelReader.Core/Crypto/EncryptedPackageOpener.cs b/src/ExcelReader.Core/Crypto/EncryptedPackageOpener.cs index 522abf4b..2540db29 100644 --- a/src/ExcelReader.Core/Crypto/EncryptedPackageOpener.cs +++ b/src/ExcelReader.Core/Crypto/EncryptedPackageOpener.cs @@ -1,6 +1,5 @@ using System.Buffers; using System.Buffers.Binary; -using System.Runtime.InteropServices; using System.Security.Cryptography; using ExcelReader.Core.Reader; using ExcelReader.Core.Reader.Internal; @@ -35,7 +34,7 @@ internal static bool IsEncryptedContainer(Stream seekableSource) internal static bool IsEncryptedMemory(ReadOnlyMemory container, ExcelReaderOptions options) { _ = options; - using Stream source = WrapMemory(container); + using var source = new ReadOnlyMemoryStream(container); try { using CfbContainer cfb = CfbContainer.Parse(source, ownsSource: false, ExcelReaderOptions.Default); @@ -49,7 +48,7 @@ internal static bool IsEncryptedMemory(ReadOnlyMemory container, ExcelRead internal static ReadOnlyMemory DecryptToMemory(ReadOnlyMemory container, ExcelReaderOptions options) { - using Stream source = WrapMemory(container); + using var source = new ReadOnlyMemoryStream(container); using CfbContainer cfb = CfbContainer.Parse(source, ownsSource: false, options); if (options.Password is null) { @@ -123,13 +122,6 @@ private static byte[] DecryptWholePackage(byte[] package, PackageCipher cipher, return result; } - private static MemoryStream WrapMemory(ReadOnlyMemory container) - { - return MemoryMarshal.TryGetArray(container, out ArraySegment segment) - ? new MemoryStream(segment.Array!, segment.Offset, segment.Count, writable: false) - : new MemoryStream(container.ToArray(), writable: false); - } - internal static Stream Decrypt(Stream source, bool leaveOpen, ExcelReaderOptions options) { CfbContainer cfb = CfbContainer.Parse(source, ownsSource: !leaveOpen, options); diff --git a/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs b/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs index 055cd8bb..8a5d6ec1 100644 --- a/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs +++ b/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs @@ -1,6 +1,7 @@ using System.Collections.Frozen; using System.Diagnostics.CodeAnalysis; using System.Globalization; +using System.Linq.Expressions; using System.Reflection; using System.Runtime.CompilerServices; using System.Text; @@ -574,6 +575,13 @@ private static RefAction CompileSetter(PropertyInfo prop) { return setter.CreateDelegate>(); } + if (RuntimeFeature.IsDynamicCodeCompiled) + { + // One compiled call per cell instead of this wrapper plus the open setter delegate inside it. + ParameterExpression model = Expression.Parameter(typeof(T).MakeByRefType()); + ParameterExpression value = Expression.Parameter(typeof(TProp)); + return Expression.Lambda>(Expression.Assign(Expression.Property(model, prop), value), model, value).Compile(); + } Action act = setter.CreateDelegate>(); return (ref model, value) => act(model, value); } diff --git a/src/ExcelReader.Core/Parser/Internal/SparseRowProjection.cs b/src/ExcelReader.Core/Parser/Internal/SparseRowProjection.cs index 71be8232..b89f066b 100644 --- a/src/ExcelReader.Core/Parser/Internal/SparseRowProjection.cs +++ b/src/ExcelReader.Core/Parser/Internal/SparseRowProjection.cs @@ -1,4 +1,5 @@ using ExcelReader.Core.Reader; +using ExcelReader.Core.Reader.Internal; namespace ExcelReader.Core.Parser.Internal @@ -84,9 +85,10 @@ internal static void ParseRow( seen[..bindings.Length].Clear(); } int bindingIndex = 0; - foreach (RowCell rowCell in row.Cells) + ReadOnlySpan cells = row.Descriptors; + for (int i = 0; i < cells.Length; i++) { - int column = rowCell.ColumnIndex; + int column = cells[i].Column; while (bindingIndex < bindings.Length && bindings[bindingIndex].Column < column) { bindingIndex++; @@ -100,12 +102,12 @@ internal static void ParseRow( { continue; } - Cell cell = rowCell.Value; - if (cell.Type == CellType.Empty) + if (cells[i].Type == CellType.Empty) { bindingIndex++; continue; } + Cell cell = row.CellAt(i); if (binding.Parser(ref model, in cell, in context)) { if (track && binding.RequireValue) diff --git a/src/ExcelReader.Core/Parser/Internal/TypeMapper.cs b/src/ExcelReader.Core/Parser/Internal/TypeMapper.cs index a83ff156..b47729b0 100644 --- a/src/ExcelReader.Core/Parser/Internal/TypeMapper.cs +++ b/src/ExcelReader.Core/Parser/Internal/TypeMapper.cs @@ -1,5 +1,7 @@ using System.Diagnostics.CodeAnalysis; +using System.Linq.Expressions; using System.Reflection; +using System.Runtime.CompilerServices; namespace ExcelReader.Core.Parser.Internal { @@ -84,8 +86,17 @@ private static TypeMapInfo Build(bool csvTextDates) } bool useDefault = typeof(T).IsValueType && typeof(T).GetConstructor(Type.EmptyTypes) is null; - Func? factory = useDefault ? null : static () => Activator.CreateInstance(); + Func? factory = useDefault ? null : BuildFactory(); return new TypeMapInfo([.. propertyMaps], factory, useDefault); } + + private static Func BuildFactory() + { + if (RuntimeFeature.IsDynamicCodeCompiled) + { + return Expression.Lambda>(Expression.New(typeof(T))).Compile(); + } + return static () => Activator.CreateInstance(); + } } } diff --git a/src/ExcelReader.Core/Parser/ParallelCsv/RangedFileStream.cs b/src/ExcelReader.Core/Parser/ParallelCsv/RangedFileStream.cs index d2e2f681..69878783 100644 --- a/src/ExcelReader.Core/Parser/ParallelCsv/RangedFileStream.cs +++ b/src/ExcelReader.Core/Parser/ParallelCsv/RangedFileStream.cs @@ -1,9 +1,10 @@ using System.Runtime.CompilerServices; +using ExcelReader.Core.Reader.Internal; using Microsoft.Win32.SafeHandles; namespace ExcelReader.Core.Parser.ParallelCsv { - internal sealed class RangedFileStream : Stream + internal sealed class RangedFileStream : ReadOnlyStream { private readonly SafeFileHandle _handle; private long _offset; @@ -16,20 +17,6 @@ internal RangedFileStream(SafeFileHandle handle, long start) _offset = start; } - public override bool CanRead => true; - - public override bool CanSeek => false; - - public override bool CanWrite => false; - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } - public override int Read(Span buffer) { int n = RandomAccess.Read(_handle, buffer, _offset); @@ -57,24 +44,5 @@ public override Task ReadAsync(byte[] buffer, int offset, int count, Cancel ValidateBufferArguments(buffer, offset, count); return ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); } - - public override void Flush() - { - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } } } diff --git a/src/ExcelReader.Core/Reader/Cell.cs b/src/ExcelReader.Core/Reader/Cell.cs index e599c2b6..285e01ef 100644 --- a/src/ExcelReader.Core/Reader/Cell.cs +++ b/src/ExcelReader.Core/Reader/Cell.cs @@ -100,10 +100,11 @@ public bool TryParse(IFormatProvider? provider, [MaybeNullWhen(false)] out T result = Unsafe.As(ref fast); return true; } - if (typeof(T) == typeof(decimal) && UsesDotDecimalSeparator(provider) && FastDecimal.TryParse(Value, out decimal fastDecimal)) + if (typeof(T) == typeof(decimal)) { - result = Unsafe.As(ref fastDecimal); - return true; + bool parsed = TryParseDecimalText(Value, provider, out decimal text); + result = Unsafe.As(ref text); + return parsed; } if (TryParseAsciiDigits(Value, provider, out result)) { @@ -125,15 +126,12 @@ public bool TryParse(IFormatProvider? provider, [MaybeNullWhen(false)] out T } if (typeof(T) == typeof(decimal)) { - if (double.IsNaN(_number) || double.IsInfinity(_number) - || _number < (double)decimal.MinValue || _number > (double)decimal.MaxValue) - { - result = default; - return false; - } - decimal m = (decimal)_number; + ReadOnlySpan text = Value; + bool converted = text.IsEmpty + ? TryNumberToDecimal(_number, out decimal m) + : TryParseDecimalText(text, CultureInfo.InvariantCulture, out m); result = Unsafe.As(ref m); - return true; + return converted; } if (TryParseIntegral(out result)) { @@ -145,6 +143,62 @@ public bool TryParse(IFormatProvider? provider, [MaybeNullWhen(false)] out T : T.TryParse(Value, provider, out result); } + private static bool TryParseDecimalText(ReadOnlySpan utf8, IFormatProvider provider, out decimal value) + { + return (UsesDotDecimalSeparator(provider) && FastDecimal.TryParse(utf8, out value)) + || decimal.TryParse(utf8, NumberStyles.Number | NumberStyles.AllowExponent, provider, out value); + } + + // A binary cell keeps no text, so this goes through the shortest text that round-trips the double, the + // text XLSX stores: the (decimal)double cast rounds to 15 digits on .NET 10 but exactly from .NET 11. + [SkipLocalsInit] + private static bool TryNumberToDecimal(double number, out decimal value) + { + if (double.IsNaN(number) || double.IsInfinity(number) + || number < (double)decimal.MinValue || number > (double)decimal.MaxValue) + { + value = default; + return false; + } + if (Math.Abs(number) < 9007199254740992.0 && number == Math.Truncate(number)) + { + value = (long)number; + return true; + } + if (TryShortDecimal(number, out value)) + { + return true; + } + Span buffer = stackalloc byte[32]; + if (!CellFormatter.TryFormatDouble(number, buffer, out int written)) + { + value = default; + return false; + } + return TryParseDecimalText(buffer[..written], CultureInfo.InvariantCulture, out value); + } + + // The fewest decimal places whose integer mantissa (at most 15 digits) divides back to the same + // double: the decimal its shortest round-trip text would parse to, without formatting that text. + private static bool TryShortDecimal(double number, out decimal value) + { + const double MaxMantissa = 1e15; + double magnitude = Math.Abs(number); + double pow10 = 1; + for (int scale = 0; scale <= 15 && magnitude * pow10 < MaxMantissa; scale++, pow10 *= 10) + { + double scaled = Math.Round(magnitude * pow10); + if (scaled / pow10 == magnitude) + { + ulong mantissa = (ulong)scaled; + value = new decimal((int)mantissa, (int)(mantissa >> 32), 0, number < 0, (byte)scale); + return true; + } + } + value = default; + return false; + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] private static bool TryParseAsciiDigits(ReadOnlySpan utf8, IFormatProvider provider, [MaybeNullWhen(false)] out T result) where T : IUtf8SpanParsable diff --git a/src/ExcelReader.Core/Reader/Csv/CsvReader.cs b/src/ExcelReader.Core/Reader/Csv/CsvReader.cs index 1e0c07bd..e7e91934 100644 --- a/src/ExcelReader.Core/Reader/Csv/CsvReader.cs +++ b/src/ExcelReader.Core/Reader/Csv/CsvReader.cs @@ -116,7 +116,7 @@ private static ReadOnlyMemory Transcode(ReadOnlyMemory data, Encodin { return data; } - using MemoryStream source = XlsCompoundFile.AsStream(data); + using var source = new ReadOnlyMemoryStream(data); using Stream transcoding = Encoding.CreateTranscodingStream(source, encoding!, Encoding.UTF8, leaveOpen: true); using MemoryStream target = new(data.Length); transcoding.CopyTo(target); diff --git a/src/ExcelReader.Core/Reader/Csv/ReplayStream.cs b/src/ExcelReader.Core/Reader/Csv/ReplayStream.cs index ffaee20e..805d5959 100644 --- a/src/ExcelReader.Core/Reader/Csv/ReplayStream.cs +++ b/src/ExcelReader.Core/Reader/Csv/ReplayStream.cs @@ -1,3 +1,5 @@ +using ExcelReader.Core.Reader.Internal; + namespace ExcelReader.Core.Reader.Csv { /// @@ -5,26 +7,12 @@ namespace ExcelReader.Core.Reader.Csv /// as everything read so far still fits in limit bytes. Disposing ends a pass; /// closes the stream underneath. /// - internal sealed class ReplayStream(Stream inner, bool leaveOpen, int limit) : Stream + internal sealed class ReplayStream(Stream inner, bool leaveOpen, int limit) : ReadOnlyStream { private MemoryStream? _recorded = new(); private int _position; private int _busy; - public override bool CanRead => true; - - public override bool CanSeek => false; - - public override bool CanWrite => false; - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } - internal bool TryBeginPass() { if (Interlocked.CompareExchange(ref _busy, 1, 0) != 0) @@ -109,25 +97,6 @@ private void Record(ReadOnlySpan data) _position += data.Length; } - public override void Flush() - { - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - protected override void Dispose(bool disposing) { Volatile.Write(ref _busy, 0); diff --git a/src/ExcelReader.Core/Reader/Internal/ReadOnlyMemoryStream.cs b/src/ExcelReader.Core/Reader/Internal/ReadOnlyMemoryStream.cs new file mode 100644 index 00000000..aff9fa01 --- /dev/null +++ b/src/ExcelReader.Core/Reader/Internal/ReadOnlyMemoryStream.cs @@ -0,0 +1,82 @@ +namespace ExcelReader.Core.Reader.Internal +{ + /// A seekable read-only stream over a , whatever backs it, without copying it. + internal sealed class ReadOnlyMemoryStream(ReadOnlyMemory memory) : ReadOnlyStream + { + private long _position; + + public override bool CanSeek => true; + + public override long Length => memory.Length; + + public override long Position + { + get => _position; + set + { + ArgumentOutOfRangeException.ThrowIfNegative(value); + _position = value; + } + } + + public override int Read(byte[] buffer, int offset, int count) + { + ValidateBufferArguments(buffer, offset, count); + return Read(buffer.AsSpan(offset, count)); + } + + public override int Read(Span buffer) + { + long remaining = memory.Length - _position; + if (remaining <= 0) + { + return 0; + } + int count = (int)Math.Min(buffer.Length, remaining); + memory.Span.Slice((int)_position, count).CopyTo(buffer); + _position += count; + return count; + } + + public override int ReadByte() + { + if (_position >= memory.Length) + { + return -1; + } + return memory.Span[(int)_position++]; + } + + public override ValueTask ReadAsync(Memory buffer, CancellationToken cancellationToken = default) + { + if (cancellationToken.IsCancellationRequested) + { + return ValueTask.FromCanceled(cancellationToken); + } + return new ValueTask(Read(buffer.Span)); + } + + public override Task ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + ValidateBufferArguments(buffer, offset, count); + return ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + long position = origin switch + { + SeekOrigin.Begin => offset, + SeekOrigin.Current => _position + offset, + SeekOrigin.End => memory.Length + offset, + _ => throw new ArgumentOutOfRangeException(nameof(origin)), + }; + if (position < 0) + { + throw new IOException("An attempt was made to move the position before the beginning of the stream."); + } + _position = position; + return position; + } + } +} diff --git a/src/ExcelReader.Core/Reader/Internal/ReadOnlyStream.cs b/src/ExcelReader.Core/Reader/Internal/ReadOnlyStream.cs new file mode 100644 index 00000000..0019b1df --- /dev/null +++ b/src/ExcelReader.Core/Reader/Internal/ReadOnlyStream.cs @@ -0,0 +1,42 @@ +namespace ExcelReader.Core.Reader.Internal +{ + /// + /// A read-only stream, forward-only unless a subclass overrides , + /// , and . Subclasses supply the reads. + /// + internal abstract class ReadOnlyStream : Stream + { + public override bool CanRead => true; + + public override bool CanSeek => false; + + public override bool CanWrite => false; + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() + { + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + } +} diff --git a/src/ExcelReader.Core/Reader/Internal/Utf8StringCache.cs b/src/ExcelReader.Core/Reader/Internal/Utf8StringCache.cs index 89403801..47d26b6a 100644 --- a/src/ExcelReader.Core/Reader/Internal/Utf8StringCache.cs +++ b/src/ExcelReader.Core/Reader/Internal/Utf8StringCache.cs @@ -4,20 +4,27 @@ namespace ExcelReader.Core.Reader.Internal { /// - /// Fixed-size, direct-mapped, content-keyed string cache: dedups repeated short text (categorical + /// Fixed-size, 4-way set-associative, content-keyed string cache: dedups repeated short text (categorical /// CSV/inline-string columns) without needing a stable table index the way the shared-string dedup - /// cache does. A hash collision just evicts the older entry — the cached string is re-verified on + /// cache does. A full set evicts its oldest entry — the cached string is re-verified on /// every hit by re-encoding it back to UTF-8 into a stack buffer, so a collision can only cost cache /// effectiveness, never return the wrong string. No growth, no rehash, and — unlike keeping a /// separate copy of each key's raw bytes — a miss allocates nothing beyond the one /// GetString() would have allocated anyway: high-cardinality (mostly-unique) columns pay no more than /// the no-cache baseline, so the cache is safe to leave on by default. /// + /// + /// is seeded per process, so which values share a set changes from run to run. A + /// direct-mapped cache let two hot values of a 10-value column evict each other in ~3% of processes, and + /// re-materialized a third of that column's strings there; four ways keep any four values of a set cached. + /// internal sealed class Utf8StringCache { - private const int BucketCount = 1024; + private const int SetCount = 256; + private const int Ways = 4; private const int MaxKeyLength = 64; - private readonly string?[] _values = new string[BucketCount]; + private readonly string?[] _values = new string[SetCount * Ways]; + private readonly int[] _hashes = new int[SetCount * Ways]; /// Returns a cached, deduplicated string for when one is /// available; otherwise materializes and caches a new one (or, past , @@ -33,14 +40,25 @@ internal string GetOrAdd(ReadOnlySpan utf8) { return Encoding.UTF8.GetString(utf8); } - int bucket = ComputeHash(utf8) & (BucketCount - 1); - string? cached = _values[bucket]; - if (cached is not null && Matches(cached, utf8)) + int hash = ComputeHash(utf8); + int start = (hash & (SetCount - 1)) * Ways; + for (int i = start; i < start + Ways; i++) { - return cached; + string? cached = _values[i]; + if (cached is null) + { + break; + } + if (_hashes[i] == hash && Matches(cached, utf8)) + { + return cached; + } } string value = Encoding.UTF8.GetString(utf8); - _values[bucket] = value; + Array.Copy(_values, start, _values, start + 1, Ways - 1); + Array.Copy(_hashes, start, _hashes, start + 1, Ways - 1); + _values[start] = value; + _hashes[start] = hash; return value; } diff --git a/src/ExcelReader.Core/Reader/Row.cs b/src/ExcelReader.Core/Reader/Row.cs index 82f285b0..89b7407d 100644 --- a/src/ExcelReader.Core/Reader/Row.cs +++ b/src/ExcelReader.Core/Reader/Row.cs @@ -1,3 +1,4 @@ +using System.Runtime.CompilerServices; using ExcelReader.Core.Reader.Internal; namespace ExcelReader.Core.Reader @@ -34,6 +35,12 @@ internal Row(ReadOnlySpan cells, ReadOnlySpan rowValues, ReadOnl internal ReadOnlySpan Descriptors => _cells; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal Cell CellAt(int index) + { + return _cells[index].ToCell(_rowValues, _shared, _rowBuffer, _sharedStringCache, _contentCache); + } + internal ReadOnlySpan RowValues => _rowValues; internal ReadOnlySpan Shared => _shared; diff --git a/src/ExcelReader.Core/Reader/Sources/ByteSourceStream.cs b/src/ExcelReader.Core/Reader/Sources/ByteSourceStream.cs index 6ae56e11..59db005f 100644 --- a/src/ExcelReader.Core/Reader/Sources/ByteSourceStream.cs +++ b/src/ExcelReader.Core/Reader/Sources/ByteSourceStream.cs @@ -1,9 +1,10 @@ using System.Buffers; +using ExcelReader.Core.Reader.Internal; namespace ExcelReader.Core.Reader.Sources { /// Forward-only view of one byte range of a , with its own offset. - internal sealed class ByteSourceStream : Stream + internal sealed class ByteSourceStream : ReadOnlyStream { private const int BufferSize = 64 * 1024; @@ -24,20 +25,6 @@ internal ByteSourceStream(ByteSource source, long offset, long length, bool buff _end = offset + length; } - public override bool CanRead => true; - - public override bool CanSeek => false; - - public override bool CanWrite => false; - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } - public override int Read(byte[] buffer, int offset, int count) { return Read(buffer.AsSpan(offset, count)); @@ -122,25 +109,6 @@ private int TakeBuffered(Span destination) return count; } - public override void Flush() - { - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - protected override void Dispose(bool disposing) { _disposed = true; diff --git a/src/ExcelReader.Core/Reader/Sources/SourceStream.cs b/src/ExcelReader.Core/Reader/Sources/SourceStream.cs index 99dfb19f..2406b637 100644 --- a/src/ExcelReader.Core/Reader/Sources/SourceStream.cs +++ b/src/ExcelReader.Core/Reader/Sources/SourceStream.cs @@ -1,4 +1,5 @@ using System.Diagnostics.CodeAnalysis; +using ExcelReader.Core.Reader.Internal; namespace ExcelReader.Core.Reader.Sources { @@ -7,18 +8,14 @@ namespace ExcelReader.Core.Reader.Sources /// source back out, so the paths that open a workbook from a stream read the source positionally. /// [SuppressMessage("Usage", "CA2213", Justification = "The source is disposed through Interlocked.Exchange so a taken source is not.")] - internal sealed class SourceStream(ByteSource source) : Stream + internal sealed class SourceStream(ByteSource source) : ReadOnlyStream { private readonly ByteSource _source = source; private ByteSource? _owned = source; private long _position; - public override bool CanRead => true; - public override bool CanSeek => true; - public override bool CanWrite => false; - public override long Length => _source.Length; public override long Position @@ -73,20 +70,6 @@ public override long Seek(long offset, SeekOrigin origin) return _position; } - public override void Flush() - { - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - protected override void Dispose(bool disposing) { if (disposing) diff --git a/src/ExcelReader.Core/Reader/Xls/XlsCompoundFile.cs b/src/ExcelReader.Core/Reader/Xls/XlsCompoundFile.cs index 7e8b65c5..e7c51ffd 100644 --- a/src/ExcelReader.Core/Reader/Xls/XlsCompoundFile.cs +++ b/src/ExcelReader.Core/Reader/Xls/XlsCompoundFile.cs @@ -1,5 +1,4 @@ using System.Diagnostics.CodeAnalysis; -using System.Runtime.InteropServices; using ExcelReader.Core.Crypto; using ExcelReader.Core.Reader.Internal; using ExcelReader.Core.Reader.Sources; @@ -32,7 +31,7 @@ internal static WorkbookStream OpenWorkbook(Stream stream, bool leaveOpen, Excel internal static WorkbookStream OpenWorkbook(ReadOnlyMemory data, ExcelReaderOptions? options = null) { - using MemoryStream metadata = AsStream(data); + using var metadata = new ReadOnlyMemoryStream(data); return BuildWorkbook(metadata, ownsSource: false, options ?? ExcelReaderOptions.Default, memory: data); } @@ -88,15 +87,6 @@ private static (Stream Source, bool OwnsSource) EnsureSeekable(Stream stream, bo return (ms, true); } - internal static MemoryStream AsStream(ReadOnlyMemory data) - { - if (MemoryMarshal.TryGetArray(data, out ArraySegment segment)) - { - return new MemoryStream(segment.Array!, segment.Offset, segment.Count, writable: false); - } - return new MemoryStream(data.ToArray(), writable: false); - } - private static WorkbookStream BuildWorkbook(Stream source, bool ownsSource, ExcelReaderOptions options, ReadOnlyMemory memory = default) { CfbContainer cfb = CfbContainer.Parse(source, ownsSource, options, memory); diff --git a/src/ExcelReader.Core/Reader/Zip/Inflate/InflateStream.cs b/src/ExcelReader.Core/Reader/Zip/Inflate/InflateStream.cs index 2d335554..1feb292e 100644 --- a/src/ExcelReader.Core/Reader/Zip/Inflate/InflateStream.cs +++ b/src/ExcelReader.Core/Reader/Zip/Inflate/InflateStream.cs @@ -1,12 +1,13 @@ using System.Buffers; using System.Diagnostics; +using ExcelReader.Core.Reader.Internal; namespace ExcelReader.Core.Reader.Zip.Inflate { /// /// Read-only stream that inflates a raw DEFLATE stream. Owns inner and disposes it. /// - internal sealed class InflateStream : Stream + internal sealed class InflateStream : ReadOnlyStream { private const int InputSize = 64 * 1024; private const int History = 32 * 1024; @@ -35,15 +36,6 @@ internal InflateStream(Stream inner) } public override bool CanRead => !_disposed; - public override bool CanSeek => false; - public override bool CanWrite => false; - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } public override int Read(byte[] buffer, int offset, int count) { @@ -103,25 +95,6 @@ public override Task ReadAsync(byte[] buffer, int offset, int count, Cancel return ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); } - public override void Flush() - { - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - protected override void Dispose(bool disposing) { if (!_disposed) diff --git a/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs b/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs index fff4ad26..fa843a6b 100644 --- a/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs +++ b/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs @@ -1,3 +1,4 @@ +using ExcelReader.Core.Reader.Internal; using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Reader.Zip @@ -37,7 +38,7 @@ internal void Add(long bytes) internal long Remaining => _limit <= 0 ? long.MaxValue : _limit - Interlocked.Read(ref _total); } - internal sealed class LimitedReadStream : Stream + internal sealed class LimitedReadStream : ReadOnlyStream { internal const string ChecksumMismatch = "The ZIP entry's data does not match its CRC-32."; @@ -68,15 +69,6 @@ internal LimitedReadStream( } public override bool CanRead => _inner.CanRead; - public override bool CanSeek => false; - public override bool CanWrite => false; - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } public override void Flush() { @@ -103,21 +95,6 @@ public override Task ReadAsync(byte[] buffer, int offset, int count, Cancel return ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); } - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - protected override void Dispose(bool disposing) { if (disposing) diff --git a/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs b/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs index bf5a3885..5017b3bc 100644 --- a/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs +++ b/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs @@ -2,11 +2,12 @@ using System.Diagnostics.CodeAnalysis; using System.Runtime.ExceptionServices; using System.Threading.Channels; +using ExcelReader.Core.Reader.Internal; using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Reader.Zip { - internal sealed class PrefetchStream : Stream + internal sealed class PrefetchStream : ReadOnlyStream { private const int ChunkSize = 64 * 1024; private const int ChannelCapacity = 4; @@ -40,23 +41,6 @@ internal PrefetchStream(Stream inner, long declaredLength = 0, uint? expectedCrc _producer = Task.Run(() => ProduceAsync(_cts.Token), _cts.Token); } - public override bool CanRead => true; - - public override bool CanSeek => false; - - public override bool CanWrite => false; - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); set => throw new NotSupportedException(); - } - - public override void Flush() - { - } - public override int Read(byte[] buffer, int offset, int count) { return Read(buffer.AsSpan(offset, count)); @@ -93,21 +77,6 @@ public override Task ReadAsync(byte[] buffer, int offset, int count, Cancel return ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); } - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - private bool EnsureCurrentChunk() { if (_currentBuffer is not null) diff --git a/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs b/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs index 0c52640f..fa576e7c 100644 --- a/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs +++ b/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs @@ -451,7 +451,7 @@ private LimitedReadStream WrapEntryStream( { ThrowIfEntryDataInvalid(dataOffset, entry); Stream raw = HasMemory - ? ToReadableMemoryStream(CompressedSlice(dataOffset, entry.CompressedSize)) + ? new ReadOnlyMemoryStream(CompressedSlice(dataOffset, entry.CompressedSize)) : new ByteSourceStream(_source, dataOffset, entry.CompressedSize); Stream opened = entry.Method == 0 ? raw : new InflateStream(raw); return WorkbookLookups.Wrap( @@ -850,7 +850,7 @@ private static ZipPart InflateToPart(ReadOnlyMemory compressed, long uncom byte[] rented = ArrayPool.Shared.Rent(size); try { - using MemoryStream source = ToReadableMemoryStream(compressed); + using var source = new ReadOnlyMemoryStream(compressed); using var inflate = new InflateStream(source); try { @@ -878,14 +878,5 @@ private static void ThrowChecksumMismatch() { throw new InvalidDataException(LimitedReadStream.ChecksumMismatch); } - - private static MemoryStream ToReadableMemoryStream(ReadOnlyMemory compressed) - { - if (MemoryMarshal.TryGetArray(compressed, out ArraySegment segment)) - { - return new MemoryStream(segment.Array!, segment.Offset, segment.Count, writable: false); - } - return new MemoryStream(compressed.ToArray(), writable: false); - } } } diff --git a/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs b/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs index f3b9c53e..0aa0f16a 100644 --- a/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs +++ b/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs @@ -1,62 +1,98 @@ using System.Globalization; -using System.Runtime.InteropServices; using ExcelReader.Core.Reader.Xlsb; using ExcelReader.Core.Writer.Xlsb; namespace ExcelReader.Core.Writer.Internal { + /// + /// The workbook's shared strings. A single writer's texts live in one dictionary, looked up with one + /// hash; parallel workers read it without a lock and add their new texts to shards split by hash, + /// so workers adding different texts take different locks. The GetOrAdd overloads never run + /// while workers call , which is what keeps those lock-free reads safe. + /// internal sealed class SharedStringTable { - private const int DefaultCapacity = 1024; + private const int ShardCount = 64; - private readonly Dictionary _indexes = new(DefaultCapacity, StringComparer.Ordinal); - private readonly List _values = new(DefaultCapacity); - private readonly Dictionary.AlternateLookup> _spanIndexes; + private readonly Shard[] _shards = new Shard[ShardCount]; + private readonly Dictionary _serial = new(StringComparer.Ordinal); + private readonly Dictionary.AlternateLookup> _serialSpan; + private int _count; + private int _uniqueCount; internal SharedStringTable() { - _spanIndexes = _indexes.GetAlternateLookup>(); + for (int i = 0; i < ShardCount; i++) + { + _shards[i] = new Shard(); + } + _serialSpan = _serial.GetAlternateLookup>(); } - internal int Count { get; private set; } - internal int UniqueCount => _values.Count; + internal int Count => Volatile.Read(ref _count); + + internal int UniqueCount => Volatile.Read(ref _uniqueCount); internal int GetOrAdd(string value) { - Count++; - ref int index = ref CollectionsMarshal.GetValueRefOrAddDefault(_indexes, value, out bool existed); - if (existed) + _count++; + if (_serial.TryGetValue(value, out int index)) { return index; } - index = _values.Count; - _values.Add(value); + index = SerialMiss(value); + _serial.Add(value, index); return index; } internal int GetOrAdd(ReadOnlySpan value) { - Count++; - if (_spanIndexes.TryGetValue(value, out int index)) + _count++; + if (_serialSpan.TryGetValue(value, out int index)) { return index; } - string added = value.ToString(); - index = _values.Count; - _indexes.Add(added, index); - _values.Add(added); + index = SerialMiss(value); + _serialSpan.TryAdd(value, index); return index; } - /// Counts references that were resolved without calling . + // Workers may have added it to a shard; a text new to both takes the next index. + private int SerialMiss(ReadOnlySpan value) + { + return ShardFor(value).SpanIndexes.TryGetValue(value, out int index) ? index : _uniqueCount++; + } + + /// Resolves a text from a parallel worker without counting the reference; see . + internal int GetOrAddConcurrent(ReadOnlySpan value) + { + if (_serialSpan.TryGetValue(value, out int serial)) + { + return serial; + } + Shard shard = ShardFor(value); + lock (shard.Gate) + { + if (shard.SpanIndexes.TryGetValue(value, out int index)) + { + return index; + } + index = Interlocked.Increment(ref _uniqueCount) - 1; + shard.SpanIndexes.TryAdd(value, index); + return index; + } + } + + /// Counts references resolved without the counting GetOrAdd overloads. internal void AddReferences(int count) { - Count += count; + Interlocked.Add(ref _count, count); } internal BiffBuffer ToXlsxBytes() { - BiffBuffer xml = new(Math.Max(256, _values.Count * 32)); + string[] values = Values(); + BiffBuffer xml = new(Math.Max(256, values.Length * 32)); xml.WriteUtf8( "" + $""u8); - foreach (ref readonly string value in CollectionsMarshal.AsSpan(_values)) + foreach (string value in values) { xml.Write(" entry in _serial) + { + values[entry.Value] = entry.Key; + } + foreach (Shard shard in _shards) + { + foreach (KeyValuePair entry in shard.Indexes) + { + values[entry.Value] = entry.Key; + } + } + return values; + } + + private Shard ShardFor(ReadOnlySpan value) + { + return _shards[string.GetHashCode(value, StringComparison.Ordinal) & (ShardCount - 1)]; + } + private static bool NeedsPreserveSpace(string value) { return value.Length > 0 && (char.IsWhiteSpace(value[0]) || char.IsWhiteSpace(value[^1])); } + + private sealed class Shard + { + internal Shard() + { + SpanIndexes = Indexes.GetAlternateLookup>(); + } + + internal Dictionary Indexes { get; } = new(StringComparer.Ordinal); + + internal Dictionary.AlternateLookup> SpanIndexes { get; } + + internal Lock Gate { get; } = new(); + } } } diff --git a/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs b/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs index b8015939..d2462976 100644 --- a/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs +++ b/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs @@ -31,6 +31,20 @@ internal static void SetColumnWidth( columnWidths[columnIndex] = width; } + internal static SortedSet Columns(Dictionary? columnStyles, Dictionary? columnWidths) + { + var columns = new SortedSet(); + if (columnStyles is not null) + { + columns.UnionWith(columnStyles.Keys); + } + if (columnWidths is not null) + { + columns.UnionWith(columnWidths.Keys); + } + return columns; + } + private static void RequireNotStarted(WriterState state, object writer) { WriterStateGuard.ThrowIfEnded(state, writer); diff --git a/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs b/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs index 690f507e..07a7099f 100644 --- a/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs +++ b/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs @@ -5,7 +5,7 @@ namespace ExcelReader.Core.Writer.Internal { - internal sealed class WriteOffloadStream : Stream + internal sealed class WriteOffloadStream : WriteOnlyStream { private const int ChannelCapacity = 4; @@ -32,34 +32,6 @@ internal WriteOffloadStream(Stream inner) _consumer = Task.Run(ConsumeAsync); } - public override bool CanRead => false; - - public override bool CanSeek => false; - - public override bool CanWrite => true; - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); set => throw new NotSupportedException(); - } - - public override int Read(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - public override void Write(byte[] buffer, int offset, int count) { Write(buffer.AsSpan(offset, count)); diff --git a/src/ExcelReader.Core/Writer/Internal/WriteOnlyStream.cs b/src/ExcelReader.Core/Writer/Internal/WriteOnlyStream.cs new file mode 100644 index 00000000..80b9f729 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Internal/WriteOnlyStream.cs @@ -0,0 +1,37 @@ +namespace ExcelReader.Core.Writer.Internal +{ + /// A forward-only, write-only stream. Subclasses supply the writes and . + internal abstract class WriteOnlyStream : Stream + { + public override bool CanRead => false; + + public override bool CanSeek => false; + + public override bool CanWrite => true; + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public abstract override void Flush(); + + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + } +} diff --git a/src/ExcelReader.Core/Writer/Parallel/WorkerStringCache.cs b/src/ExcelReader.Core/Writer/Parallel/WorkerStringCache.cs index 2218aedb..f4e91bbf 100644 --- a/src/ExcelReader.Core/Writer/Parallel/WorkerStringCache.cs +++ b/src/ExcelReader.Core/Writer/Parallel/WorkerStringCache.cs @@ -4,71 +4,52 @@ namespace ExcelReader.Core.Writer.Parallel { /// /// One worker's view of the workbook's shared-string table: texts it has already resolved are - /// answered locally, and only a text new to this worker takes the table's lock. + /// answered locally, and only a text new to this worker takes one of the table's shard locks. /// internal sealed class WorkerStringCache { private const int MaxEntries = 65_536; private readonly SharedStringTable _table; - private readonly Lock _gate; private readonly Dictionary _indexes = new(StringComparer.Ordinal); private readonly Dictionary.AlternateLookup> _spanIndexes; - private int _localHits; + private int _references; - internal WorkerStringCache(SharedStringTable table, Lock gate) + internal WorkerStringCache(SharedStringTable table) { _table = table; - _gate = gate; _spanIndexes = _indexes.GetAlternateLookup>(); } internal int GetOrAdd(string value) { - if (_indexes.TryGetValue(value, out int index)) - { - _localHits++; - return index; - } - return Resolve(value); + return GetOrAdd(value.AsSpan()); } internal int GetOrAdd(ReadOnlySpan value) { + _references++; if (_spanIndexes.TryGetValue(value, out int index)) { - _localHits++; return index; } - return Resolve(value.ToString()); - } - - // The table counts every reference; ones answered locally are reported in bulk. - internal void FlushReferences() - { - if (_localHits == 0) - { - return; - } - lock (_gate) + index = _table.GetOrAddConcurrent(value); + if (_indexes.Count < MaxEntries) { - _table.AddReferences(_localHits); + _spanIndexes.TryAdd(value, index); } - _localHits = 0; + return index; } - private int Resolve(string value) + // The table counts every reference; this worker's are reported in bulk. + internal void FlushReferences() { - int index; - lock (_gate) + if (_references == 0) { - index = _table.GetOrAdd(value); - } - if (_indexes.Count < MaxEntries) - { - _indexes.Add(value, index); + return; } - return index; + _table.AddReferences(_references); + _references = 0; } } } diff --git a/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs b/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs index a4ad1730..52501e87 100644 --- a/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs @@ -12,7 +12,7 @@ namespace ExcelReader.Core.Writer.Xlsb { /// Writes a single worksheet's rows into an .xlsb workbook produced by . - public sealed class XlsbSheetWriter : ISheetWriter, ITableSheetWriter, IBatchSheet + public sealed class XlsbSheetWriter : ISheetWriter, ITableSheetWriter, IBatchSheet, IZipSheet { private const int SpillThreshold = 64 * 1024; @@ -86,7 +86,7 @@ private XlsbSheetWriter(XlsbSheetWriter source, int batchRows) _state = WriterState.Started; if (_owner.UseSharedStrings) { - _workerStrings = new WorkerStringCache(_owner.SharedStrings!, _owner.SharedStringsGate); + _workerStrings = new WorkerStringCache(_owner.SharedStrings!); } } @@ -165,16 +165,7 @@ private void WriteColInfos() { return; } - var columns = new SortedSet(); - if (_columnStyles is not null) - { - columns.UnionWith(_columnStyles.Keys); - } - if (_columnWidths is not null) - { - columns.UnionWith(_columnWidths.Keys); - } - foreach (int columnIndex in columns) + foreach (int columnIndex in SheetColumnValidation.Columns(_columnStyles, _columnWidths)) { int styleId = _columnStyles is not null && _columnStyles.TryGetValue(columnIndex, out int s) ? s : 0; bool hasWidth = false; @@ -387,12 +378,12 @@ private async ValueTask FaultAsync() Release(faulted: true); } - internal void Abandon() + void IZipSheet.Abandon() { Fault(); } - internal ValueTask AbandonAsync() + ValueTask IZipSheet.AbandonAsync() { return FaultAsync(); } diff --git a/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs b/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs index 436c7c96..1d4c7a7d 100644 --- a/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs @@ -9,29 +9,20 @@ namespace ExcelReader.Core.Writer.Xlsb { /// Writes an .xlsb workbook (BIFF12 binary format) to a stream, one sheet at a time. - public sealed class XlsbWorkbookWriter : IWorkbookWriter + public sealed class XlsbWorkbookWriter : IWorkbookWriter, IZipWorkbookParts { - private readonly ZipPackageWriter _zip; - private readonly Stream _stream; - private readonly bool _leaveOpen; + private readonly ZipWorkbookPackage _package; private readonly bool _date1904; private readonly CompressionLevel _compression; private readonly bool _prefetchWrite; private readonly SharedStringTable? _sharedStrings; private readonly StyleTable _styles = new(); private readonly List _sheets = []; - private bool _ended; - private XlsbSheetWriter? _activeSheet; private readonly HashSet _sheetNames = new(StringComparer.OrdinalIgnoreCase); - private bool _faulted; - private bool _zipClosed; - private bool _disposed; - private XlsbWorkbookWriter(ZipPackageWriter zip, Stream stream, bool leaveOpen, XlsbWriterOptions options) + private XlsbWorkbookWriter(Stream stream, bool leaveOpen, XlsbWriterOptions options) { - _zip = zip; - _stream = stream; - _leaveOpen = leaveOpen; + _package = new ZipWorkbookPackage(this, "XLSB", stream, leaveOpen, options.Compression); _date1904 = options.Date1904; UseSharedStrings = options.UseSharedStrings; _compression = options.Compression; @@ -47,8 +38,7 @@ private XlsbWorkbookWriter(ZipPackageWriter zip, Stream stream, bool leaveOpen, public static XlsbWorkbookWriter Create(Stream stream, bool leaveOpen = false, XlsbWriterOptions? options = null) { ArgumentNullException.ThrowIfNull(stream); - ZipPackageWriter zip = new(stream); - return new XlsbWorkbookWriter(zip, stream, leaveOpen, options ?? XlsbWriterOptions.Default); + return new XlsbWorkbookWriter(stream, leaveOpen, options ?? XlsbWriterOptions.Default); } /// @@ -61,11 +51,12 @@ public XlsbSheetWriter AddSheet(string name) public XlsbSheetWriter AddSheet(string name, ExcelSheetVisibility visibility) { WriterStateGuard.RequireCanAddSheet( - _ended, this, name, _activeSheet is not null, nameof(XlsbSheetWriter), visibility); + _package.Ended, this, name, _package.ActiveSheet is not null, nameof(XlsbSheetWriter), visibility); WriterStateGuard.ClaimSheetName(_sheetNames, name); int sheetId = _sheets.Count + 1; - _activeSheet = new XlsbSheetWriter(this, _zip, name, sheetId, visibility, _date1904, _compression, _prefetchWrite); - return _activeSheet; + var sheet = new XlsbSheetWriter(this, _package.Zip, name, sheetId, visibility, _date1904, _compression, _prefetchWrite); + _package.ActiveSheet = sheet; + return sheet; } internal void RegisterSheet(XlsbSheetWriter sheet) @@ -75,12 +66,7 @@ internal void RegisterSheet(XlsbSheetWriter sheet) internal void NotifySheetEnded(bool faulted) { - _activeSheet = null; - if (faulted) - { - _ended = true; - _faulted = true; - } + _package.SheetEnded(faulted); } internal bool UseSharedStrings { get; } @@ -105,8 +91,6 @@ internal SharedStringTable? SharedStrings } } - internal Lock SharedStringsGate { get; } = new(); - /// public int AddStyle(CellStyle style) { @@ -121,125 +105,45 @@ public int AddStyle(CellStyle style) /// public void End() { - ObjectDisposedException.ThrowIf(_ended, this); - _ended = true; - try - { - _activeSheet?.Dispose(); - if (_sheets.Count == 0) - { - throw new InvalidOperationException("A workbook must contain at least one sheet."); - } - - WriteRootRels(); - WriteWorkbook(); - WriteWorkbookRels(); - WriteStyles(); - WriteSharedStrings(); - WriteAppProperties(); - WriteTables(); - WriteContentTypes(); - CloseZip(); - } - catch - { - AbandonZip(); - throw; - } + _package.End(); } /// - public async ValueTask EndAsync(CancellationToken ct = default) - { - ObjectDisposedException.ThrowIf(_ended, this); - ct.ThrowIfCancellationRequested(); - _ended = true; - try - { - if (_activeSheet is not null) - { - await _activeSheet.DisposeAsync().ConfigureAwait(false); - } - if (_sheets.Count == 0) - { - throw new InvalidOperationException("A workbook must contain at least one sheet."); - } - - await WriteRootRelsAsync(ct).ConfigureAwait(false); - await WriteWorkbookAsync(ct).ConfigureAwait(false); - await WriteWorkbookRelsAsync(ct).ConfigureAwait(false); - await WriteStylesAsync(ct).ConfigureAwait(false); - await WriteSharedStringsAsync(ct).ConfigureAwait(false); - await WriteAppPropertiesAsync(ct).ConfigureAwait(false); - await WriteTablesAsync(ct).ConfigureAwait(false); - await WriteContentTypesAsync(ct).ConfigureAwait(false); - await CloseZipAsync(ct).ConfigureAwait(false); - } - catch - { - await AbandonZipAsync().ConfigureAwait(false); - throw; - } - } - - private void CloseZip() + public ValueTask EndAsync(CancellationToken ct = default) { - _zipClosed = true; - _zip.Finish(); + return _package.EndAsync(ct); } - private ValueTask CloseZipAsync(CancellationToken ct = default) + int IZipWorkbookParts.SheetCount { - _zipClosed = true; - return _zip.FinishAsync(ct); - } - - private void AbandonZip() - { - _faulted = true; - _activeSheet?.Abandon(); - if (!_zipClosed) + get { - _zipClosed = true; - _zip.Abandon(); + return _sheets.Count; } } - private async ValueTask AbandonZipAsync() + void IZipWorkbookParts.WriteParts() { - _faulted = true; - if (_activeSheet is not null) - { - await _activeSheet.AbandonAsync().ConfigureAwait(false); - } - if (!_zipClosed) - { - _zipClosed = true; - await _zip.AbandonAsync().ConfigureAwait(false); - } + _package.WriteEntry("_rels/.rels", BuildRootRelsXml()); + WriteWorkbook(); + _package.WriteEntry("xl/_rels/workbook.bin.rels", BuildWorkbookRelsXml()); + WriteStyles(); + WriteSharedStrings(); + _package.WriteEntry("docProps/app.xml", AppPropertiesXml); + WriteTables(); + _package.WriteEntry("[Content_Types].xml", BuildContentTypesXml()); } - private void ReleaseStream() + async ValueTask IZipWorkbookParts.WritePartsAsync(CancellationToken ct) { - if (_leaveOpen) - { - return; - } - if (_faulted) - { - FailureCleanup.Dispose(_stream); - return; - } - _stream.Dispose(); - } - - private ValueTask ReleaseStreamAsync() - { - if (_leaveOpen) - { - return ValueTask.CompletedTask; - } - return _faulted ? FailureCleanup.DisposeAsync(_stream) : _stream.DisposeAsync(); + await _package.WriteEntryAsync("_rels/.rels", BuildRootRelsXml(), ct).ConfigureAwait(false); + await WriteWorkbookAsync(ct).ConfigureAwait(false); + await _package.WriteEntryAsync("xl/_rels/workbook.bin.rels", BuildWorkbookRelsXml(), ct).ConfigureAwait(false); + await WriteStylesAsync(ct).ConfigureAwait(false); + await WriteSharedStringsAsync(ct).ConfigureAwait(false); + await _package.WriteEntryAsync("docProps/app.xml", AppPropertiesXml, ct).ConfigureAwait(false); + await WriteTablesAsync(ct).ConfigureAwait(false); + await _package.WriteEntryAsync("[Content_Types].xml", BuildContentTypesXml(), ct).ConfigureAwait(false); } /// @@ -248,13 +152,13 @@ private ValueTask ReleaseStreamAsync() /// public void Flush() { - _stream.Flush(); + _package.Flush(); } /// public ValueTask FlushAsync(CancellationToken ct = default) { - return ZipEntryWriter.FlushAsync(_stream, ct); + return _package.FlushAsync(ct); } /// @@ -263,73 +167,13 @@ public ValueTask FlushAsync(CancellationToken ct = default) /// public void Dispose() { - if (_disposed) - { - return; - } - _disposed = true; - try - { - if (!_ended) - { - if (_sheets.Count == 0 && _activeSheet is null) - { - _ended = true; - CloseZip(); - } - else - { - End(); - } - } - else if (!_zipClosed) - { - AbandonZip(); - } - } - catch - { - _faulted = true; - ReleaseStream(); - throw; - } - ReleaseStream(); + _package.Dispose(); } /// - public async ValueTask DisposeAsync() + public ValueTask DisposeAsync() { - if (_disposed) - { - return; - } - _disposed = true; - try - { - if (!_ended) - { - if (_sheets.Count == 0 && _activeSheet is null) - { - _ended = true; - await CloseZipAsync().ConfigureAwait(false); - } - else - { - await EndAsync().ConfigureAwait(false); - } - } - else if (!_zipClosed) - { - await AbandonZipAsync().ConfigureAwait(false); - } - } - catch - { - _faulted = true; - await ReleaseStreamAsync().ConfigureAwait(false); - throw; - } - await ReleaseStreamAsync().ConfigureAwait(false); + return _package.DisposeAsync(); } private static string BuildRootRelsXml() @@ -341,16 +185,6 @@ private static string BuildRootRelsXml() ""; } - private void WriteRootRels() - { - WriteEntry("_rels/.rels", BuildRootRelsXml()); - } - - private ValueTask WriteRootRelsAsync(CancellationToken ct) - { - return WriteEntryAsync("_rels/.rels", BuildRootRelsXml(), ct); - } - private void BuildWorkbookBin(BiffBuffer payload, BiffBuffer data) { Biff12RecordWriter.WriteRecord(data, Brt.BeginBook); @@ -381,7 +215,7 @@ private void WriteWorkbook() using BiffBuffer payload = new(256); using BiffBuffer data = new(1024); BuildWorkbookBin(payload, data); - WriteEntry("xl/workbook.bin", data.Span); + _package.WriteEntry("xl/workbook.bin", data.Span); } private async ValueTask WriteWorkbookAsync(CancellationToken ct) @@ -389,7 +223,7 @@ private async ValueTask WriteWorkbookAsync(CancellationToken ct) using BiffBuffer payload = new(256); using BiffBuffer data = new(1024); BuildWorkbookBin(payload, data); - await WriteEntryAsync("xl/workbook.bin", data.Memory, ct).ConfigureAwait(false); + await _package.WriteEntryAsync("xl/workbook.bin", data.Memory, ct).ConfigureAwait(false); } private string BuildWorkbookRelsXml() @@ -407,16 +241,6 @@ private string BuildWorkbookRelsXml() return sb.ToString(); } - private void WriteWorkbookRels() - { - WriteEntry("xl/_rels/workbook.bin.rels", BuildWorkbookRelsXml()); - } - - private ValueTask WriteWorkbookRelsAsync(CancellationToken ct) - { - return WriteEntryAsync("xl/_rels/workbook.bin.rels", BuildWorkbookRelsXml(), ct); - } - private void BuildStylesBin(BiffBuffer payload, BiffBuffer data) { Dictionary numFmtIds = _styles.AssignCustomNumberFormatIds(); @@ -459,7 +283,7 @@ private void WriteStyles() using BiffBuffer payload = new(96); using BiffBuffer data = new(512); BuildStylesBin(payload, data); - WriteEntry("xl/styles.bin", data.Span); + _package.WriteEntry("xl/styles.bin", data.Span); } private async ValueTask WriteStylesAsync(CancellationToken ct) @@ -467,7 +291,7 @@ private async ValueTask WriteStylesAsync(CancellationToken ct) using BiffBuffer payload = new(96); using BiffBuffer data = new(512); BuildStylesBin(payload, data); - await WriteEntryAsync("xl/styles.bin", data.Memory, ct).ConfigureAwait(false); + await _package.WriteEntryAsync("xl/styles.bin", data.Memory, ct).ConfigureAwait(false); } private static void WriteStyles(BiffBuffer data, BiffBuffer payload) @@ -496,13 +320,13 @@ private static void WriteFmt(BiffBuffer data, BiffBuffer payload, int numFmtId, private void WriteSharedStrings() { using var data = _sharedStrings is null ? EmptySharedStrings() : _sharedStrings.ToXlsbBytes(); - WriteEntry("xl/sharedStrings.bin", data.Span); + _package.WriteEntry("xl/sharedStrings.bin", data.Span); } private async ValueTask WriteSharedStringsAsync(CancellationToken ct) { using var data = _sharedStrings is null ? EmptySharedStrings() : _sharedStrings.ToXlsbBytes(); - await WriteEntryAsync("xl/sharedStrings.bin", data.Memory, ct).ConfigureAwait(false); + await _package.WriteEntryAsync("xl/sharedStrings.bin", data.Memory, ct).ConfigureAwait(false); } private static BiffBuffer EmptySharedStrings() @@ -572,16 +396,6 @@ private static void WriteFill(BiffBuffer data, BiffBuffer payload, uint pattern) "" + "ExcelReader1.0000"; - private void WriteAppProperties() - { - WriteEntry("docProps/app.xml", AppPropertiesXml); - } - - private ValueTask WriteAppPropertiesAsync(CancellationToken ct) - { - return WriteEntryAsync("docProps/app.xml", AppPropertiesXml, ct); - } - private void WriteTables() { foreach (XlsbSheetWriter sheet in _sheets) @@ -590,13 +404,13 @@ private void WriteTables() { continue; } - WriteEntry($"xl/worksheets/_rels/sheet{sheet.SheetId}.bin.rels", TablePackage.SheetRelsXml(sheet.Tables, "bin")); + _package.WriteEntry($"xl/worksheets/_rels/sheet{sheet.SheetId}.bin.rels", TablePackage.SheetRelsXml(sheet.Tables, "bin")); foreach (WrittenTable table in sheet.Tables) { using BiffBuffer data = new(256); using BiffBuffer payload = new(128); XlsbTableRecords.WriteTable(data, payload, table); - WriteEntry($"xl/tables/table{table.Id}.bin", data.Span); + _package.WriteEntry($"xl/tables/table{table.Id}.bin", data.Span); } } } @@ -609,7 +423,7 @@ private async ValueTask WriteTablesAsync(CancellationToken ct) { continue; } - await WriteEntryAsync($"xl/worksheets/_rels/sheet{sheet.SheetId}.bin.rels", TablePackage.SheetRelsXml(sheet.Tables, "bin"), ct).ConfigureAwait(false); + await _package.WriteEntryAsync($"xl/worksheets/_rels/sheet{sheet.SheetId}.bin.rels", TablePackage.SheetRelsXml(sheet.Tables, "bin"), ct).ConfigureAwait(false); foreach (WrittenTable table in sheet.Tables) { await WriteTablePartAsync(table, ct).ConfigureAwait(false); @@ -622,7 +436,7 @@ private async ValueTask WriteTablePartAsync(WrittenTable table, CancellationToke using BiffBuffer data = new(256); using BiffBuffer payload = new(128); XlsbTableRecords.WriteTable(data, payload, table); - await WriteEntryAsync($"xl/tables/table{table.Id}.bin", data.Memory, ct).ConfigureAwait(false); + await _package.WriteEntryAsync($"xl/tables/table{table.Id}.bin", data.Memory, ct).ConfigureAwait(false); } private static void WriteXf(BiffBuffer data, BiffBuffer payload, int numFmtId, bool isStyleXf = false) @@ -670,34 +484,5 @@ private string BuildContentTypesXml() return sb.ToString(); } - private void WriteContentTypes() - { - WriteEntry("[Content_Types].xml", BuildContentTypesXml()); - } - - private ValueTask WriteContentTypesAsync(CancellationToken ct) - { - return WriteEntryAsync("[Content_Types].xml", BuildContentTypesXml(), ct); - } - - private ValueTask WriteEntryAsync(string entryName, string content, CancellationToken ct) - { - return ZipEntryWriter.WriteTextAsync(_zip, entryName, content, _compression, ct); - } - - private ValueTask WriteEntryAsync(string entryName, ReadOnlyMemory content, CancellationToken ct) - { - return ZipEntryWriter.WriteBytesAsync(_zip, entryName, content, _compression, ct); - } - - private void WriteEntry(string entryName, string content) - { - ZipEntryWriter.WriteText(_zip, entryName, content, _compression); - } - - private void WriteEntry(string entryName, ReadOnlySpan content) - { - ZipEntryWriter.WriteBytes(_zip, entryName, content, _compression); - } } } diff --git a/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs b/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs index 20340061..ccf508fd 100644 --- a/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs @@ -14,7 +14,7 @@ namespace ExcelReader.Core.Writer.Xlsx /// /// Writes a single worksheet's XML into an XLSX ZIP archive, buffering rows and flushing them to the entry stream once they cross a size threshold. /// - public sealed class XlsxSheetWriter : ISheetWriter, ITableSheetWriter, IBatchSheet + public sealed class XlsxSheetWriter : ISheetWriter, ITableSheetWriter, IBatchSheet, IZipSheet { private readonly TableTracker _tables = new(); private readonly XlsxWorkbookWriter _owner; @@ -65,7 +65,7 @@ private XlsxSheetWriter(XlsxSheetWriter source, int batchRows) _state = WriterState.Started; if (_owner.UseSharedStrings) { - _workerStrings = new WorkerStringCache(_owner.SharedStrings!, _owner.SharedStringsGate); + _workerStrings = new WorkerStringCache(_owner.SharedStrings!); } } @@ -129,23 +129,14 @@ private void EnsureStarted() } try { - ZipEntryStream entry = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.xml", _compression); - _entry = entry; - _stream = _offloadWrite ? new WriteOffloadStream(entry) : entry; - _rowBuffer.Reset(); - _rowBuffer.WriteUtf8( - "" + - $"{BuildColsXml()}"); - _stream.Write(_rowBuffer.Span); + OpenEntry().Write(_rowBuffer.Span); } catch { Fault(); throw; } - _rowBuffer.Reset(); - _state = WriterState.Started; - _owner.RegisterSheet(Name, SheetId, Visibility); + MarkStarted(); } internal ValueTask EnsureStartedAsync(CancellationToken ct) @@ -158,20 +149,31 @@ private async ValueTask StartCoreAsync(CancellationToken ct) { try { - ZipEntryStream entry = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.xml", _compression); - _entry = entry; - _stream = _offloadWrite ? new WriteOffloadStream(entry) : entry; - _rowBuffer.Reset(); - _rowBuffer.WriteUtf8( - "" + - $"{BuildColsXml()}"); - await _stream.WriteAsync(_rowBuffer.Memory, ct).ConfigureAwait(false); + await OpenEntry().WriteAsync(_rowBuffer.Memory, ct).ConfigureAwait(false); } catch { await FaultAsync().ConfigureAwait(false); throw; } + MarkStarted(); + } + + // Leaves the worksheet header in the row buffer for the caller to write. + private Stream OpenEntry() + { + ZipEntryStream entry = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.xml", _compression); + _entry = entry; + _stream = _offloadWrite ? new WriteOffloadStream(entry) : entry; + _rowBuffer.Reset(); + _rowBuffer.WriteUtf8( + "" + + $"{BuildColsXml()}"); + return _stream; + } + + private void MarkStarted() + { _rowBuffer.Reset(); _state = WriterState.Started; _owner.RegisterSheet(Name, SheetId, Visibility); @@ -183,17 +185,8 @@ private string BuildColsXml() { return string.Empty; } - var columns = new SortedSet(); - if (_columnStyles is not null) - { - columns.UnionWith(_columnStyles.Keys); - } - if (_columnWidths is not null) - { - columns.UnionWith(_columnWidths.Keys); - } var sb = new StringBuilder(""); - foreach (int columnIndex in columns) + foreach (int columnIndex in SheetColumnValidation.Columns(_columnStyles, _columnWidths)) { int oneBased = columnIndex + 1; sb.Append(CultureInfo.InvariantCulture, $" /// Writes a workbook to the XLSX (Office Open XML) format, streaming each sheet's rows into a ZIP archive as they're written. /// - public sealed class XlsxWorkbookWriter : IWorkbookWriter + public sealed class XlsxWorkbookWriter : IWorkbookWriter, IZipWorkbookParts { - private readonly ZipPackageWriter _zip; - private readonly Stream _stream; - private readonly bool _leaveOpen; + private readonly ZipWorkbookPackage _package; private readonly CompressionLevel _compression; private readonly bool _prefetchWrite; private readonly SharedStringTable? _sharedStrings; private readonly StyleTable _styles = new(); private readonly List<(string Name, int SheetId, ExcelSheetVisibility Visibility)> _sheets = []; private readonly List<(int SheetId, IReadOnlyList Tables)> _sheetTables = []; - private bool _ended; - private bool _sheetActive; - private XlsxSheetWriter? _activeSheet; private readonly HashSet _sheetNames = new(StringComparer.OrdinalIgnoreCase); - private bool _faulted; - private bool _zipClosed; - private bool _disposed; - private XlsxWorkbookWriter(ZipPackageWriter zip, Stream stream, bool leaveOpen, XlsxWriterOptions options) + private XlsxWorkbookWriter(Stream stream, bool leaveOpen, XlsxWriterOptions options) { - _zip = zip; - _stream = stream; - _leaveOpen = leaveOpen; + _package = new ZipWorkbookPackage(this, "XLSX", stream, leaveOpen, options.Compression); UseSharedStrings = options.UseSharedStrings; Date1904 = options.Date1904; _compression = options.Compression; @@ -51,8 +41,7 @@ private XlsxWorkbookWriter(ZipPackageWriter zip, Stream stream, bool leaveOpen, public static XlsxWorkbookWriter Create(Stream stream, bool leaveOpen = false, XlsxWriterOptions? options = null) { ArgumentNullException.ThrowIfNull(stream); - ZipPackageWriter zip = new(stream); - return new XlsxWorkbookWriter(zip, stream, leaveOpen, options ?? XlsxWriterOptions.Default); + return new XlsxWorkbookWriter(stream, leaveOpen, options ?? XlsxWriterOptions.Default); } /// @@ -74,12 +63,12 @@ public XlsxSheetWriter AddSheet(string name) public XlsxSheetWriter AddSheet(string name, ExcelSheetVisibility visibility) { WriterStateGuard.RequireCanAddSheet( - _ended, this, name, _sheetActive, nameof(XlsxSheetWriter), visibility); + _package.Ended, this, name, _package.ActiveSheet is not null, nameof(XlsxSheetWriter), visibility); WriterStateGuard.ClaimSheetName(_sheetNames, name); - _sheetActive = true; int sheetId = _sheets.Count + 1; - _activeSheet = new XlsxSheetWriter(this, _zip, name, sheetId, visibility, _compression, _prefetchWrite); - return _activeSheet; + var sheet = new XlsxSheetWriter(this, _package.Zip, name, sheetId, visibility, _compression, _prefetchWrite); + _package.ActiveSheet = sheet; + return sheet; } internal void RegisterSheet(string name, int sheetId, ExcelSheetVisibility visibility) @@ -97,39 +86,9 @@ internal void RegisterTables(int sheetId, IReadOnlyList tables) } } - private void WriteTables() - { - foreach ((int sheetId, IReadOnlyList tables) in _sheetTables) - { - WriteEntry($"xl/worksheets/_rels/sheet{sheetId}.xml.rels", TablePackage.SheetRelsXml(tables, "xml")); - foreach (WrittenTable table in tables) - { - WriteEntry($"xl/tables/table{table.Id}.xml", XlsxTableXml.Table(table)); - } - } - } - - private async ValueTask WriteTablesAsync(CancellationToken ct) - { - foreach ((int sheetId, IReadOnlyList tables) in _sheetTables) - { - await WriteEntryAsync($"xl/worksheets/_rels/sheet{sheetId}.xml.rels", TablePackage.SheetRelsXml(tables, "xml"), ct).ConfigureAwait(false); - foreach (WrittenTable table in tables) - { - await WriteEntryAsync($"xl/tables/table{table.Id}.xml", XlsxTableXml.Table(table), ct).ConfigureAwait(false); - } - } - } - internal void NotifySheetEnded(bool faulted) { - _sheetActive = false; - _activeSheet = null; - if (faulted) - { - _ended = true; - _faulted = true; - } + _package.SheetEnded(faulted); } internal bool UseSharedStrings { get; } @@ -154,8 +113,6 @@ internal SharedStringTable? SharedStrings } } - internal Lock SharedStringsGate { get; } = new(); - /// public int AddStyle(CellStyle style) { @@ -172,130 +129,67 @@ public int AddStyle(CellStyle style) /// No sheet has been added yet. public void End() { - ObjectDisposedException.ThrowIf(_ended, this); - _ended = true; - try - { - _activeSheet?.Dispose(); - if (_sheets.Count == 0) - { - throw new InvalidOperationException("An XLSX workbook must contain at least one sheet."); - } - WriteTables(); - WriteEntry("_rels/.rels", BuildRootRelsXml()); - WriteEntry("xl/styles.xml", BuildStylesXml()); - if (_sharedStrings is not null) - { - using var bytes = _sharedStrings.ToXlsxBytes(); - WriteEntry("xl/sharedStrings.xml", bytes.Memory.Span); - } - WriteEntry("xl/workbook.xml", BuildWorkbookXml()); - WriteEntry("xl/_rels/workbook.xml.rels", BuildWorkbookRelsXml()); - WriteEntry("[Content_Types].xml", BuildContentTypesXml()); - CloseZip(); - } - catch - { - AbandonZip(); - throw; - } + _package.End(); } /// /// The workbook has already been ended. /// No sheet has been added yet. - public async ValueTask EndAsync(CancellationToken ct = default) - { - ObjectDisposedException.ThrowIf(_ended, this); - ct.ThrowIfCancellationRequested(); - _ended = true; - try - { - if (_activeSheet is not null) - { - await _activeSheet.DisposeAsync().ConfigureAwait(false); - } - if (_sheets.Count == 0) - { - throw new InvalidOperationException("An XLSX workbook must contain at least one sheet."); - } - await WriteTablesAsync(ct).ConfigureAwait(false); - await WriteEntryAsync("_rels/.rels", BuildRootRelsXml(), ct).ConfigureAwait(false); - await WriteStylesAsync(ct).ConfigureAwait(false); - if (_sharedStrings is not null) - { - await WriteSharedStringsAsync(ct).ConfigureAwait(false); - } - await WriteWorkbookAsync(ct).ConfigureAwait(false); - await WriteWorkbookRelsAsync(ct).ConfigureAwait(false); - await WriteContentTypesAsync(ct).ConfigureAwait(false); - await CloseZipAsync(ct).ConfigureAwait(false); - } - catch - { - await AbandonZipAsync().ConfigureAwait(false); - throw; - } - } - - private void CloseZip() + public ValueTask EndAsync(CancellationToken ct = default) { - _zipClosed = true; - _zip.Finish(); + return _package.EndAsync(ct); } - private ValueTask CloseZipAsync(CancellationToken ct = default) + int IZipWorkbookParts.SheetCount { - _zipClosed = true; - return _zip.FinishAsync(ct); - } - - private void AbandonZip() - { - _faulted = true; - _activeSheet?.Abandon(); - if (!_zipClosed) + get { - _zipClosed = true; - _zip.Abandon(); + return _sheets.Count; } } - private async ValueTask AbandonZipAsync() + void IZipWorkbookParts.WriteParts() { - _faulted = true; - if (_activeSheet is not null) + foreach ((int sheetId, IReadOnlyList tables) in _sheetTables) { - await _activeSheet.AbandonAsync().ConfigureAwait(false); + _package.WriteEntry($"xl/worksheets/_rels/sheet{sheetId}.xml.rels", TablePackage.SheetRelsXml(tables, "xml")); + foreach (WrittenTable table in tables) + { + _package.WriteEntry($"xl/tables/table{table.Id}.xml", XlsxTableXml.Table(table)); + } } - if (!_zipClosed) + _package.WriteEntry("_rels/.rels", BuildRootRelsXml()); + _package.WriteEntry("xl/styles.xml", BuildStylesXml()); + if (_sharedStrings is not null) { - _zipClosed = true; - await _zip.AbandonAsync().ConfigureAwait(false); + using var bytes = _sharedStrings.ToXlsxBytes(); + _package.WriteEntry("xl/sharedStrings.xml", bytes.Memory.Span); } + _package.WriteEntry("xl/workbook.xml", BuildWorkbookXml()); + _package.WriteEntry("xl/_rels/workbook.xml.rels", BuildWorkbookRelsXml()); + _package.WriteEntry("[Content_Types].xml", BuildContentTypesXml()); } - private void ReleaseStream() + async ValueTask IZipWorkbookParts.WritePartsAsync(CancellationToken ct) { - if (_leaveOpen) - { - return; - } - if (_faulted) + foreach ((int sheetId, IReadOnlyList tables) in _sheetTables) { - FailureCleanup.Dispose(_stream); - return; + await _package.WriteEntryAsync($"xl/worksheets/_rels/sheet{sheetId}.xml.rels", TablePackage.SheetRelsXml(tables, "xml"), ct).ConfigureAwait(false); + foreach (WrittenTable table in tables) + { + await _package.WriteEntryAsync($"xl/tables/table{table.Id}.xml", XlsxTableXml.Table(table), ct).ConfigureAwait(false); + } } - _stream.Dispose(); - } - - private ValueTask ReleaseStreamAsync() - { - if (_leaveOpen) + await _package.WriteEntryAsync("_rels/.rels", BuildRootRelsXml(), ct).ConfigureAwait(false); + await _package.WriteEntryAsync("xl/styles.xml", BuildStylesXml(), ct).ConfigureAwait(false); + if (_sharedStrings is not null) { - return ValueTask.CompletedTask; + using var bytes = _sharedStrings.ToXlsxBytes(); + await _package.WriteEntryAsync("xl/sharedStrings.xml", bytes.Memory, ct).ConfigureAwait(false); } - return _faulted ? FailureCleanup.DisposeAsync(_stream) : _stream.DisposeAsync(); + await _package.WriteEntryAsync("xl/workbook.xml", BuildWorkbookXml(), ct).ConfigureAwait(false); + await _package.WriteEntryAsync("xl/_rels/workbook.xml.rels", BuildWorkbookRelsXml(), ct).ConfigureAwait(false); + await _package.WriteEntryAsync("[Content_Types].xml", BuildContentTypesXml(), ct).ConfigureAwait(false); } /// @@ -304,13 +198,13 @@ private ValueTask ReleaseStreamAsync() /// public void Flush() { - _stream.Flush(); + _package.Flush(); } /// public ValueTask FlushAsync(CancellationToken ct = default) { - return ZipEntryWriter.FlushAsync(_stream, ct); + return _package.FlushAsync(ct); } /// @@ -319,73 +213,13 @@ public ValueTask FlushAsync(CancellationToken ct = default) /// public void Dispose() { - if (_disposed) - { - return; - } - _disposed = true; - try - { - if (!_ended) - { - if (_sheets.Count == 0 && _activeSheet is null) - { - _ended = true; - CloseZip(); - } - else - { - End(); - } - } - else if (!_zipClosed) - { - AbandonZip(); - } - } - catch - { - _faulted = true; - ReleaseStream(); - throw; - } - ReleaseStream(); + _package.Dispose(); } /// - public async ValueTask DisposeAsync() + public ValueTask DisposeAsync() { - if (_disposed) - { - return; - } - _disposed = true; - try - { - if (!_ended) - { - if (_sheets.Count == 0 && _activeSheet is null) - { - _ended = true; - await CloseZipAsync().ConfigureAwait(false); - } - else - { - await EndAsync().ConfigureAwait(false); - } - } - else if (!_zipClosed) - { - await AbandonZipAsync().ConfigureAwait(false); - } - } - catch - { - _faulted = true; - await ReleaseStreamAsync().ConfigureAwait(false); - throw; - } - await ReleaseStreamAsync().ConfigureAwait(false); + return _package.DisposeAsync(); } private static string BuildRootRelsXml() @@ -396,11 +230,6 @@ private static string BuildRootRelsXml() ""; } - private ValueTask WriteStylesAsync(CancellationToken ct) - { - return WriteEntryAsync("xl/styles.xml", BuildStylesXml(), ct); - } - private string BuildStylesXml() { IReadOnlyList styles = _styles.Styles; @@ -485,12 +314,6 @@ private static void AppendCellXfs(StringBuilder sb, IReadOnlyList sty sb.Append(""); } - private async ValueTask WriteSharedStringsAsync(CancellationToken ct) - { - using var bytes = _sharedStrings!.ToXlsxBytes(); - await WriteEntryAsync("xl/sharedStrings.xml", bytes.Memory, ct).ConfigureAwait(false); - } - private static string StateAttribute(ExcelSheetVisibility visibility) { return visibility switch @@ -523,11 +346,6 @@ private string BuildWorkbookXml() return sb.ToString(); } - private ValueTask WriteWorkbookAsync(CancellationToken ct) - { - return WriteEntryAsync("xl/workbook.xml", BuildWorkbookXml(), ct); - } - private string BuildWorkbookRelsXml() { StringBuilder sb = new(); @@ -546,11 +364,6 @@ private string BuildWorkbookRelsXml() return sb.ToString(); } - private ValueTask WriteWorkbookRelsAsync(CancellationToken ct) - { - return WriteEntryAsync("xl/_rels/workbook.xml.rels", BuildWorkbookRelsXml(), ct); - } - private string BuildContentTypesXml() { StringBuilder sb = new(); @@ -579,31 +392,6 @@ private string BuildContentTypesXml() return sb.ToString(); } - private ValueTask WriteContentTypesAsync(CancellationToken ct) - { - return WriteEntryAsync("[Content_Types].xml", BuildContentTypesXml(), ct); - } - - private ValueTask WriteEntryAsync(string entryName, string content, CancellationToken ct) - { - return ZipEntryWriter.WriteTextAsync(_zip, entryName, content, _compression, ct); - } - - private ValueTask WriteEntryAsync(string entryName, ReadOnlyMemory content, CancellationToken ct) - { - return ZipEntryWriter.WriteBytesAsync(_zip, entryName, content, _compression, ct); - } - - private void WriteEntry(string entryName, string content) - { - ZipEntryWriter.WriteText(_zip, entryName, content, _compression); - } - - private void WriteEntry(string entryName, ReadOnlySpan content) - { - ZipEntryWriter.WriteBytes(_zip, entryName, content, _compression); - } - private static string EscapeAttribute(string value) { return SecurityElement.Escape(value); diff --git a/src/ExcelReader.Core/Writer/Zip/BufferSink.cs b/src/ExcelReader.Core/Writer/Zip/BufferSink.cs index 2a910804..fcd682a6 100644 --- a/src/ExcelReader.Core/Writer/Zip/BufferSink.cs +++ b/src/ExcelReader.Core/Writer/Zip/BufferSink.cs @@ -7,7 +7,7 @@ namespace ExcelReader.Core.Writer.Zip /// is set it drops what it is given, which is how the final block a /// DeflateStream writes on dispose is kept out of a stream of segments. /// - internal sealed class BufferSink : Stream + internal sealed class BufferSink : WriteOnlyStream { private readonly BiffBuffer _target; @@ -18,69 +18,10 @@ internal BufferSink(BiffBuffer target) internal bool Discard { get; set; } - public override bool CanRead - { - get - { - return false; - } - } - - public override bool CanSeek - { - get - { - return false; - } - } - - public override bool CanWrite - { - get - { - return true; - } - } - - public override long Length - { - get - { - throw new NotSupportedException(); - } - } - - public override long Position - { - get - { - throw new NotSupportedException(); - } - set - { - throw new NotSupportedException(); - } - } - public override void Flush() { } - public override int Read(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - public override void Write(byte[] buffer, int offset, int count) { Write(buffer.AsSpan(offset, count)); diff --git a/src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs b/src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs index 41d01fbf..4c908845 100644 --- a/src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs +++ b/src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs @@ -9,7 +9,7 @@ namespace ExcelReader.Core.Writer.Zip /// what it is given; closing it completes the local header in place, by seeking back, or with a /// data descriptor. /// - internal sealed class ZipEntryStream : Stream + internal sealed class ZipEntryStream : WriteOnlyStream { private const int FlushThreshold = 64 * 1024; private const int MaxWriteSlice = 1 << 20; @@ -63,22 +63,6 @@ private enum Completion PatchZip64, } - public override bool CanRead - { - get - { - return false; - } - } - - public override bool CanSeek - { - get - { - return false; - } - } - public override bool CanWrite { get @@ -87,26 +71,6 @@ public override bool CanWrite } } - public override long Length - { - get - { - throw new NotSupportedException(); - } - } - - public override long Position - { - get - { - throw new NotSupportedException(); - } - set - { - throw new NotSupportedException(); - } - } - public override void Flush() { if (_closed) @@ -141,21 +105,6 @@ public override async Task FlushAsync(CancellationToken cancellationToken) } } - public override int Read(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - public override void Write(byte[] buffer, int offset, int count) { Write(buffer.AsSpan(offset, count)); diff --git a/src/ExcelReader.Core/Writer/Zip/ZipWorkbookPackage.cs b/src/ExcelReader.Core/Writer/Zip/ZipWorkbookPackage.cs new file mode 100644 index 00000000..8709241d --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/ZipWorkbookPackage.cs @@ -0,0 +1,273 @@ +using System.IO.Compression; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Zip +{ + /// The workbook-level parts a ZIP-packaged workbook writes once its last sheet has ended. + internal interface IZipWorkbookParts + { + int SheetCount { get; } + + void WriteParts(); + + ValueTask WritePartsAsync(CancellationToken ct); + } + + /// A sheet that a ends or abandons along with its workbook. + internal interface IZipSheet : IDisposable, IAsyncDisposable + { + void Abandon(); + + ValueTask AbandonAsync(); + } + + /// + /// The end, abandon and dispose lifecycle shared by the XLSX and XLSB workbook writers: one open + /// sheet at a time, the workbook parts written at end, and the destination released exactly once. + /// + internal sealed class ZipWorkbookPackage + { + private readonly IZipWorkbookParts _parts; + private readonly string _format; + private readonly Stream _stream; + private readonly bool _leaveOpen; + private readonly CompressionLevel _compression; + private bool _faulted; + private bool _zipClosed; + private bool _disposed; + + internal ZipWorkbookPackage(IZipWorkbookParts parts, string format, Stream stream, bool leaveOpen, CompressionLevel compression) + { + _parts = parts; + _format = format; + _stream = stream; + _leaveOpen = leaveOpen; + _compression = compression; + Zip = new ZipPackageWriter(stream); + } + + internal ZipPackageWriter Zip { get; } + + internal bool Ended { get; private set; } + + internal IZipSheet? ActiveSheet { get; set; } + + internal void SheetEnded(bool faulted) + { + ActiveSheet = null; + if (faulted) + { + Ended = true; + _faulted = true; + } + } + + internal void End() + { + ObjectDisposedException.ThrowIf(Ended, _parts); + Ended = true; + try + { + ActiveSheet?.Dispose(); + RequireSheet(); + _parts.WriteParts(); + _zipClosed = true; + Zip.Finish(); + } + catch + { + Abandon(); + throw; + } + } + + internal async ValueTask EndAsync(CancellationToken ct) + { + ObjectDisposedException.ThrowIf(Ended, _parts); + ct.ThrowIfCancellationRequested(); + Ended = true; + try + { + if (ActiveSheet is not null) + { + await ActiveSheet.DisposeAsync().ConfigureAwait(false); + } + RequireSheet(); + await _parts.WritePartsAsync(ct).ConfigureAwait(false); + _zipClosed = true; + await Zip.FinishAsync(ct).ConfigureAwait(false); + } + catch + { + await AbandonAsync().ConfigureAwait(false); + throw; + } + } + + private void RequireSheet() + { + if (_parts.SheetCount == 0) + { + throw new InvalidOperationException($"An {_format} workbook must contain at least one sheet."); + } + } + + private void Abandon() + { + _faulted = true; + ActiveSheet?.Abandon(); + if (!_zipClosed) + { + _zipClosed = true; + Zip.Abandon(); + } + } + + private async ValueTask AbandonAsync() + { + _faulted = true; + if (ActiveSheet is not null) + { + await ActiveSheet.AbandonAsync().ConfigureAwait(false); + } + if (!_zipClosed) + { + _zipClosed = true; + await Zip.AbandonAsync().ConfigureAwait(false); + } + } + + internal void Flush() + { + _stream.Flush(); + } + + internal ValueTask FlushAsync(CancellationToken ct) + { + return ZipEntryWriter.FlushAsync(_stream, ct); + } + + internal void Dispose() + { + if (_disposed) + { + return; + } + _disposed = true; + try + { + Close(); + } + catch + { + _faulted = true; + ReleaseStream(); + throw; + } + ReleaseStream(); + } + + internal async ValueTask DisposeAsync() + { + if (_disposed) + { + return; + } + _disposed = true; + try + { + await CloseAsync().ConfigureAwait(false); + } + catch + { + _faulted = true; + await ReleaseStreamAsync().ConfigureAwait(false); + throw; + } + await ReleaseStreamAsync().ConfigureAwait(false); + } + + private void Close() + { + if (Ended) + { + if (!_zipClosed) + { + Abandon(); + } + return; + } + if (_parts.SheetCount == 0 && ActiveSheet is null) + { + Ended = true; + _zipClosed = true; + Zip.Finish(); + return; + } + End(); + } + + private async ValueTask CloseAsync() + { + if (Ended) + { + if (!_zipClosed) + { + await AbandonAsync().ConfigureAwait(false); + } + return; + } + if (_parts.SheetCount == 0 && ActiveSheet is null) + { + Ended = true; + _zipClosed = true; + await Zip.FinishAsync().ConfigureAwait(false); + return; + } + await EndAsync(CancellationToken.None).ConfigureAwait(false); + } + + private void ReleaseStream() + { + if (_leaveOpen) + { + return; + } + if (_faulted) + { + FailureCleanup.Dispose(_stream); + return; + } + _stream.Dispose(); + } + + private ValueTask ReleaseStreamAsync() + { + if (_leaveOpen) + { + return ValueTask.CompletedTask; + } + return _faulted ? FailureCleanup.DisposeAsync(_stream) : _stream.DisposeAsync(); + } + + internal void WriteEntry(string entryName, string content) + { + ZipEntryWriter.WriteText(Zip, entryName, content, _compression); + } + + internal void WriteEntry(string entryName, ReadOnlySpan content) + { + ZipEntryWriter.WriteBytes(Zip, entryName, content, _compression); + } + + internal ValueTask WriteEntryAsync(string entryName, string content, CancellationToken ct) + { + return ZipEntryWriter.WriteTextAsync(Zip, entryName, content, _compression, ct); + } + + internal ValueTask WriteEntryAsync(string entryName, ReadOnlyMemory content, CancellationToken ct) + { + return ZipEntryWriter.WriteBytesAsync(Zip, entryName, content, _compression, ct); + } + } +} diff --git a/src/ExcelReader.Native/Exports.Reading.cs b/src/ExcelReader.Native/Exports.Reading.cs index 389c5ef5..38e0bb84 100644 --- a/src/ExcelReader.Native/Exports.Reading.cs +++ b/src/ExcelReader.Native/Exports.Reading.cs @@ -198,6 +198,71 @@ public static int SheetIndex(nint handle, byte* name, int nameLength, int* outIn return status; } + [UnmanagedCallersOnly(EntryPoint = "xl_table_count")] + public static int TableCount(nint handle, int* outCount) + { + NativeApi.ClearLastError(); + + if (outCount is null) + { + return NativeStatus.InvalidArgument; + } + + int status = ReadApi.TableCount(Resolve(handle), out int count); + *outCount = count; + return status; + } + + [UnmanagedCallersOnly(EntryPoint = "xl_table_info_at")] + public static int TableInfoAt(nint handle, int index, NativeTableInfo* outInfo) + { + NativeApi.ClearLastError(); + + if (outInfo is null) + { + return NativeStatus.InvalidArgument; + } + + int status = ReadApi.TableInfoAt(Resolve(handle), index, out NativeTableInfo info); + *outInfo = info; + return status; + } + + [UnmanagedCallersOnly(EntryPoint = "xl_table_name_at")] + public static int TableNameAt(nint handle, int index, byte* buffer, int capacity, int* outLength) + { + NativeApi.ClearLastError(); + + if (!IsValidOutBuffer(buffer, capacity, outLength)) + { + return NativeStatus.InvalidArgument; + } + + int status = ReadApi.TableNameAt(Resolve(handle), index, new Span(buffer, capacity), out int length); + *outLength = length; + return status; + } + + [UnmanagedCallersOnly(EntryPoint = "xl_table_index")] + public static int TableIndex(nint handle, byte* name, int nameLength, int* outIndex) + { + NativeApi.ClearLastError(); + + if (outIndex is null) + { + return NativeStatus.InvalidArgument; + } + *outIndex = -1; + if (nameLength < 0 || (name is null && nameLength != 0)) + { + return NativeStatus.InvalidArgument; + } + + int status = ReadApi.TableIndex(Resolve(handle), new ReadOnlySpan(name, nameLength), out int index); + *outIndex = index; + return status; + } + [UnmanagedCallersOnly(EntryPoint = "xl_rows_open")] public static int RowsOpen(nint handle, int sheet, nint* outRows) { diff --git a/src/ExcelReader.Native/Exports.Writer.cs b/src/ExcelReader.Native/Exports.Writer.cs index 8d692833..d625adeb 100644 --- a/src/ExcelReader.Native/Exports.Writer.cs +++ b/src/ExcelReader.Native/Exports.Writer.cs @@ -256,6 +256,52 @@ public static int EndSheet(nint handle) return RunWriterOp(handle, static writerHandle => writerHandle.EndSheet()); } + [UnmanagedCallersOnly(EntryPoint = "xl_write_sheet")] + public static int WriteSheet(nint handle, byte* name, int nameLength, NativeColumnSpecRaw* specs, NativeTable* table, NativeSheetOptionsRaw* options) + { + if (name is null || !TypedApi.IsValidNameLength(nameLength) || specs is null || table is null + || !TypedApi.IsValidSpecCount(table->ColumnCount)) + { + NativeApi.SetLastError("xl_write_sheet needs a name, specs and a table with 1..max columns."); + return NativeStatus.InvalidArgument; + } + if (!TryResolveWriter(handle, out NativeWriterHandle? writerHandle)) + { + return NativeStatus.InvalidHandle; + } + try + { + if (!NativeExcelTable.TryRead(options, out NativeExcelTable? excelTable, out string? optionsError)) + { + NativeApi.SetLastError(optionsError); + return NativeStatus.InvalidArgument; + } + if (!TryDecodeColumnSpecs(specs, table->ColumnCount, out NativeColumnSpec[] decoded)) + { + NativeApi.SetLastError("xl_write_sheet got a column spec with an invalid name."); + return NativeStatus.InvalidArgument; + } + if (!WriteApi.TryValidateWriteTable(decoded, *table, out bool hasHeader, out string? validationError)) + { + NativeApi.SetLastError(validationError); + return NativeStatus.InvalidArgument; + } + if (excelTable is not null && !hasHeader) + { + NativeApi.SetLastError("an Excel table needs every column named: its header row holds the column names."); + return NativeStatus.InvalidArgument; + } + writerHandle.WriteSheet(Encoding.UTF8.GetString(name, nameLength), decoded, *table, hasHeader, excelTable); + NativeApi.ClearLastError(); + return NativeStatus.Ok; + } + catch (Exception exception) + { + NativeApi.SetLastError(exception.Message); + return NativeStatus.Error; + } + } + [UnmanagedCallersOnly(EntryPoint = "xl_close_write_handle")] public static int CloseWriteHandle(nint handle) { diff --git a/src/ExcelReader.Native/README.md b/src/ExcelReader.Native/README.md index 79b37b7a..3cefa1b9 100644 --- a/src/ExcelReader.Native/README.md +++ b/src/ExcelReader.Native/README.md @@ -54,7 +54,8 @@ may get worse, not only better. Reading exports: `xl_open_file`, `xl_open_memory`, `xl_open_source`, `xl_open_stream`, `xl_set_source_error`, `xl_close`, `xl_sheet_count`, `xl_sheet_name_at`, -`xl_sheet_visibility_at`, `xl_sheet_index`, `xl_is_date1904`, `xl_rows_open`, `xl_rows_next`, +`xl_sheet_visibility_at`, `xl_sheet_index`, `xl_table_count`, `xl_table_info_at`, `xl_table_name_at`, +`xl_table_index`, `xl_is_date1904`, `xl_rows_open`, `xl_rows_next`, `xl_rows_next_view`, `xl_rows_foreach`, `xl_aggregate`, `xl_rows_read_all_blob`, `xl_rows_read_all_decoded`, `xl_rows_close`, `xl_free_rows`, `xl_parse_typed`, `xl_free_table`, `xl_typed_reader_open`, `xl_typed_reader_next`, `xl_typed_reader_close`, `xl_infer_schema`, `xl_free_schema`, `xl_last_error`, `xl_last_error_ptr`, @@ -65,6 +66,13 @@ and opens its own enumerator, so calls on one handle may run on several threads typed reader or Arrow stream is used by one thread at a time and keeps working after `xl_close` on its workbook. +Excel tables are read through the same calls. `XL_TABLE_SHEET(table)` passed as the `sheet` of any +read (rows, callbacks, typed, Arrow, schema inference, aggregation) reads only that table's range, +header row first, with the column indexes of the sheet holding it; `xl_sheet_name_at` and +`xl_sheet_visibility_at` report that sheet. `xl_table_count`, `xl_table_info_at` (sheet, range, +header and totals row counts), `xl_table_name_at` and `xl_table_index` (ignoring case, `-1` when +absent) describe the workbook's tables. A table read is never partitioned. + Writing has two layers. The one-shot export, `xl_write_typed` (plus its in-memory twin `xl_write_typed_to_memory`), takes a whole `xl_table` and writes it in a single call; it takes an `xl_write_options*` that follows the exact same `struct_size` contract as `xl_open_options`: the @@ -79,14 +87,19 @@ several threads: `degree_of_parallelism` `0` uses every processor and `1` writes is therefore parallel, while the 7.0 layout (`struct_size` 32) and a NULL options pointer stay serial. The `xl_table` buffers are read from several threads, so they must not change during the call. With `use_shared_strings` on, parallel runs give the same content in files that differ byte for byte. -XLS, CSV and the handle API below ignore both fields. +XLS, CSV and the handle API's per-cell calls below ignore both fields. The streaming alternative is `xl_writer_handle`: one sheet and one row open at a time, written directly as each call arrives instead of building an `xl_table` up front. Call order is `xl_open_write_handle`/`xl_open_write_handle_to_memory`, then per sheet `xl_start_sheet`..`xl_end_sheet`, each containing `xl_start_row`..`xl_end_row` with one `xl_write_string`/`xl_write_int64`/ `xl_write_float64`/`xl_write_bool`/`xl_write_date`/`xl_write_time`/`xl_write_timestamp`/`xl_write_null` -call per cell, then `xl_close_write_handle`. `xl_write_handle_bytes` reads back a memory-backed +call per cell, then `xl_close_write_handle`. Between sheets, `xl_write_sheet` writes a whole +`xl_table` as one sheet, as `xl_write_typed` does, using the handle's `degree_of_parallelism` and +`batch_rows`; it fails while a sheet is open. Its `xl_sheet_options*` (NULL for a plain sheet, same +`struct_size` contract) can write the sheet as an Excel table: `table_name`, `table_style` (NULL keeps +`TableStyleMedium2`) and `XL_TABLE_*` flags for no style, row and column stripes and first- and +last-column emphasis. A table needs every column named and an XLSX or XLSB handle. `xl_write_handle_bytes` reads back a memory-backed handle's bytes so far without releasing it. Both write paths release their buffers with `xl_free_buffer` (`xl_table`/`xl_inferred_schema` keep their own `xl_free_table`/`xl_free_schema`). diff --git a/src/ExcelReader.Native/Reading/CallbackReadStream.cs b/src/ExcelReader.Native/Reading/CallbackReadStream.cs index 34cc01d6..9a0565dd 100644 --- a/src/ExcelReader.Native/Reading/CallbackReadStream.cs +++ b/src/ExcelReader.Native/Reading/CallbackReadStream.cs @@ -1,8 +1,9 @@ using System.Diagnostics.CodeAnalysis; +using ExcelReader.Core.Reader.Internal; namespace ExcelReader.Native.Reading { - internal sealed unsafe class CallbackReadStream : Stream + internal sealed unsafe class CallbackReadStream : ReadOnlyStream { private readonly delegate* unmanaged _read; private readonly delegate* unmanaged _release; @@ -31,20 +32,6 @@ internal static bool TryCreate( return true; } - public override bool CanRead => true; - - public override bool CanSeek => false; - - public override bool CanWrite => false; - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } - public override int Read(Span buffer) { ObjectDisposedException.ThrowIf(Volatile.Read(ref _released) != 0, this); @@ -90,25 +77,6 @@ public override int Read(byte[] buffer, int offset, int count) return Read(buffer.AsSpan(offset, count)); } - public override void Flush() - { - } - - public override long Seek(long offset, SeekOrigin origin) - { - throw new NotSupportedException(); - } - - public override void SetLength(long value) - { - throw new NotSupportedException(); - } - - public override void Write(byte[] buffer, int offset, int count) - { - throw new NotSupportedException(); - } - protected override void Dispose(bool disposing) { if (Interlocked.Exchange(ref _released, 1) == 0 && _release is not null) diff --git a/src/ExcelReader.Native/Reading/ReadApi.Sheets.cs b/src/ExcelReader.Native/Reading/ReadApi.Sheets.cs index c60af9cb..63f5e0f5 100644 --- a/src/ExcelReader.Native/Reading/ReadApi.Sheets.cs +++ b/src/ExcelReader.Native/Reading/ReadApi.Sheets.cs @@ -8,15 +8,15 @@ internal static partial class ReadApi internal static int ResolveSheet(NativeHandle handle, int sheet, out IExcelSheet? resolved) { resolved = null; - if (sheet < 0) + if (sheet == -1) { - NativeApi.SetLastError($"sheet must be a zero-based index; got {sheet}."); + NativeApi.SetLastError("sheet must be a zero-based index, or XL_TABLE_SHEET(table) for a table; got -1."); return NativeStatus.InvalidArgument; } try { - resolved = handle.Workbook.SheetAt(sheet); + resolved = sheet >= 0 ? handle.Workbook.SheetAt(sheet) : TableAt(handle, FirstTableSheet - sheet).AsSheet(); return NativeStatus.Ok; } catch (Exception exception) diff --git a/src/ExcelReader.Native/Reading/ReadApi.Tables.cs b/src/ExcelReader.Native/Reading/ReadApi.Tables.cs new file mode 100644 index 00000000..0630c895 --- /dev/null +++ b/src/ExcelReader.Native/Reading/ReadApi.Tables.cs @@ -0,0 +1,146 @@ +using System.Runtime.InteropServices; +using System.Text; +using ExcelReader.Core.Reader; + +namespace ExcelReader.Native.Reading +{ + [StructLayout(LayoutKind.Sequential)] + internal struct NativeTableInfo + { + public int Sheet; + public int FirstRow; + public int LastRow; + public int FirstColumn; + public int LastColumn; + public int HeaderRowCount; + public int TotalsRowCount; + } + + internal static partial class ReadApi + { + // XL_TABLE_SHEET(table) in excelreader.h: tables take the sheet indexes below -1. + internal const int FirstTableSheet = -2; + + private static ExcelTable TableAt(NativeHandle handle, int table) + { + IReadOnlyList tables = handle.Workbook.Tables; + ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual(table, tables.Count, nameof(table)); + return tables[table]; + } + + internal static int TableCount(NativeHandle? handle, out int count) + { + count = 0; + if (handle is null) + { + return NativeStatus.InvalidHandle; + } + NativeApi.ClearLastError(); + try + { + count = handle.Workbook.Tables.Count; + return NativeStatus.Ok; + } + catch (Exception exception) + { + NativeApi.SetLastError(exception.Message); + return NativeStatus.Error; + } + } + + internal static int TableInfoAt(NativeHandle? handle, int index, out NativeTableInfo info) + { + info = default; + if (handle is null) + { + return NativeStatus.InvalidHandle; + } + NativeApi.ClearLastError(); + if (index < 0) + { + NativeApi.SetLastError($"table must be a zero-based index; got {index}."); + return NativeStatus.InvalidArgument; + } + try + { + ExcelTable table = TableAt(handle, index); + info = new NativeTableInfo + { + Sheet = table.Sheet.Index, + FirstRow = table.FirstRow, + LastRow = table.LastRow, + FirstColumn = table.FirstColumn, + LastColumn = table.LastColumn, + HeaderRowCount = table.HeaderRowCount, + TotalsRowCount = table.TotalsRowCount, + }; + return NativeStatus.Ok; + } + catch (Exception exception) + { + NativeApi.SetLastError(exception.Message); + return NativeStatus.Error; + } + } + + internal static int TableNameAt(NativeHandle? handle, int index, Span buffer, out int length) + { + length = 0; + if (handle is null) + { + return NativeStatus.InvalidHandle; + } + NativeApi.ClearLastError(); + if (index < 0) + { + NativeApi.SetLastError($"table must be a zero-based index; got {index}."); + return NativeStatus.InvalidArgument; + } + try + { + return CopyUtf8(TableAt(handle, index).Name, buffer, out length); + } + catch (Exception exception) + { + NativeApi.SetLastError(exception.Message); + return NativeStatus.Error; + } + } + + internal static int TableIndex(NativeHandle? handle, ReadOnlySpan utf8Name, out int index) + { + index = -1; + if (handle is null) + { + return NativeStatus.InvalidHandle; + } + NativeApi.ClearLastError(); + try + { + IExcelWorkbook workbook = handle.Workbook; + if (workbook.TryGetTable(Encoding.UTF8.GetString(utf8Name), out ExcelTable? table)) + { + index = IndexOf(workbook.Tables, table); + } + return NativeStatus.Ok; + } + catch (Exception exception) + { + NativeApi.SetLastError(exception.Message); + return NativeStatus.Error; + } + } + + private static int IndexOf(IReadOnlyList tables, ExcelTable table) + { + for (int i = 0; i < tables.Count; i++) + { + if (ReferenceEquals(tables[i], table)) + { + return i; + } + } + return -1; + } + } +} diff --git a/src/ExcelReader.Native/Writer/NativeSheetOptions.cs b/src/ExcelReader.Native/Writer/NativeSheetOptions.cs new file mode 100644 index 00000000..33579ae8 --- /dev/null +++ b/src/ExcelReader.Native/Writer/NativeSheetOptions.cs @@ -0,0 +1,85 @@ +using System.Diagnostics.CodeAnalysis; +using System.Runtime.InteropServices; +using System.Text; +using ExcelReader.Core.Writer; +using ExcelReader.Native.Typed; + +namespace ExcelReader.Native.Writer +{ + // See xl_sheet_options in excelreader.h for the authoritative field list. + [StructLayout(LayoutKind.Sequential)] + internal unsafe struct NativeSheetOptionsRaw + { + public int StructSize; + public int TableNameLen; + public byte* TableName; + public int TableStyleLen; + public byte* TableStyle; + public int TableFlags; + } + + /// An Excel table to write a sheet as, decoded from . + internal sealed record NativeExcelTable(string Name, ExcelTableOptions Options) + { + internal const int NoStyle = 1; + internal const int NoRowStripes = 2; + internal const int ColumnStripes = 4; + internal const int FirstColumn = 8; + internal const int LastColumn = 16; + private const int KnownFlags = NoStyle | NoRowStripes | ColumnStripes | FirstColumn | LastColumn; + + internal static unsafe bool TryRead(NativeSheetOptionsRaw* options, out NativeExcelTable? table, [NotNullWhen(false)] out string? error) + { + table = null; + error = null; + if (options is null) + { + return true; + } + if (options->StructSize != sizeof(NativeSheetOptionsRaw)) + { + error = $"xl_sheet_options.struct_size is {options->StructSize}, but this library expects {sizeof(NativeSheetOptionsRaw)}."; + return false; + } + int flags = options->TableFlags; + if (options->TableName is null) + { + if (options->TableStyle is not null || flags != 0) + { + error = "xl_sheet_options sets a table style or flags without a table_name."; + return false; + } + return true; + } + if ((flags & ~KnownFlags) != 0) + { + error = $"xl_sheet_options.table_flags has unknown bits; got {flags}."; + return false; + } + if (!TypedApi.IsValidNameLength(options->TableNameLen)) + { + error = $"xl_sheet_options.table_name_len is out of range; got {options->TableNameLen}."; + return false; + } + string? style = ExcelTableOptions.Default.StyleName; + if (options->TableStyle is not null) + { + if ((flags & NoStyle) != 0 || !TypedApi.IsValidNameLength(options->TableStyleLen)) + { + error = "xl_sheet_options.table_style needs a valid table_style_len and cannot be combined with XL_TABLE_NO_STYLE."; + return false; + } + style = Encoding.UTF8.GetString(options->TableStyle, options->TableStyleLen); + } + table = new NativeExcelTable(Encoding.UTF8.GetString(options->TableName, options->TableNameLen), new ExcelTableOptions + { + StyleName = (flags & NoStyle) != 0 ? null : style, + ShowRowStripes = (flags & NoRowStripes) == 0, + ShowColumnStripes = (flags & ColumnStripes) != 0, + ShowFirstColumn = (flags & FirstColumn) != 0, + ShowLastColumn = (flags & LastColumn) != 0, + }); + return true; + } + } +} diff --git a/src/ExcelReader.Native/Writer/NativeWriterHandle.cs b/src/ExcelReader.Native/Writer/NativeWriterHandle.cs index f16c627a..4056e0f6 100644 --- a/src/ExcelReader.Native/Writer/NativeWriterHandle.cs +++ b/src/ExcelReader.Native/Writer/NativeWriterHandle.cs @@ -44,24 +44,29 @@ public void Dispose() protected abstract void Dispose(bool disposing); + internal abstract void WriteSheet(string name, NativeColumnSpec[] specs, NativeTable table, bool hasHeader, NativeExcelTable? excelTable); + internal static NativeWriterHandle Create(Stream stream, int format, NativeWriteOptions options) { bool date1904 = options.Date1904 ?? false; bool sharedStrings = options.UseSharedStrings ?? false; + ParallelWriteOptions? parallel = options.ToParallelWriteOptions(format); switch (format) { case NativeFormat.Xlsx: return new NativeWriterHandle( - XlsxWorkbookWriter.Create(stream, options: new XlsxWriterOptions { Date1904 = date1904, UseSharedStrings = sharedStrings })); + XlsxWorkbookWriter.Create(stream, options: new XlsxWriterOptions { Date1904 = date1904, UseSharedStrings = sharedStrings }), + parallel, static (sheet, rows, writeRow, opts) => sheet.WriteRecordsParallelAsync(rows, writeRow, opts)); case NativeFormat.Xlsb: return new NativeWriterHandle( - XlsbWorkbookWriter.Create(stream, options: new XlsbWriterOptions { Date1904 = date1904, UseSharedStrings = sharedStrings })); + XlsbWorkbookWriter.Create(stream, options: new XlsbWriterOptions { Date1904 = date1904, UseSharedStrings = sharedStrings }), + parallel, static (sheet, rows, writeRow, opts) => sheet.WriteRecordsParallelAsync(rows, writeRow, opts)); case NativeFormat.Xls: return new NativeWriterHandle( - XlsWorkbookWriter.Create(stream, options: new XlsWriterOptions { Date1904 = date1904 })); + XlsWorkbookWriter.Create(stream, options: new XlsWriterOptions { Date1904 = date1904 }), null, null); case NativeFormat.Csv: return new NativeWriterHandle( - CsvWorkbookWriter.Create(stream, options: options.ToCsvWriterOptions())); + CsvWorkbookWriter.Create(stream, options: options.ToCsvWriterOptions()), null, null); default: throw new ArgumentOutOfRangeException(nameof(format), format, "Unsupported write format."); } @@ -73,13 +78,18 @@ internal sealed class NativeWriterHandle : NativeWriterHandle where TRow : class, IRowWriter { private readonly IWorkbookWriter _workbook; + private readonly ParallelWriteOptions? _parallel; + private readonly Func, Action, ParallelWriteOptions, ValueTask>? _writeParallel; private TSheet? _sheet; private TRow? _row; private bool _closed; - internal NativeWriterHandle(IWorkbookWriter workbook) + internal NativeWriterHandle(IWorkbookWriter workbook, ParallelWriteOptions? parallel, + Func, Action, ParallelWriteOptions, ValueTask>? writeParallel) { _workbook = workbook; + _parallel = parallel; + _writeParallel = writeParallel; } internal override void StartSheet(string name) @@ -174,6 +184,21 @@ internal override void EndSheet() _sheet = null; } + internal override void WriteSheet(string name, NativeColumnSpec[] specs, NativeTable table, bool hasHeader, NativeExcelTable? excelTable) + { + if (_sheet is not null) + { + throw new InvalidOperationException("A sheet is already open; call xl_end_sheet before writing a whole sheet."); + } + if (excelTable is not null && !typeof(TSheet).IsAssignableTo(typeof(ITableSheetWriter))) + { + throw new NotSupportedException("Excel tables can only be written to XLSX or XLSB."); + } + _sheet = _workbook.AddSheet(name); + WriteApi.WriteSheetBody(_workbook, _sheet, specs, table, hasHeader, excelTable, _parallel, _writeParallel); + EndSheet(); + } + internal override void Close() { if (_closed) diff --git a/src/ExcelReader.Native/Writer/WriteApi.cs b/src/ExcelReader.Native/Writer/WriteApi.cs index 27143a3e..31c68663 100644 --- a/src/ExcelReader.Native/Writer/WriteApi.cs +++ b/src/ExcelReader.Native/Writer/WriteApi.cs @@ -223,25 +223,7 @@ private static void WriteWorkbook(IWorkbookWriter workbook try { TSheet sheet = workbook.AddSheet(sheetName); - ApplyTemporalStyles(workbook, sheet, table); - - if (hasHeader) - { - WriteHeaderRow(sheet, specs); - } - if (parallel is not null && writeParallel is not null) - { - writeParallel(sheet, RowIndices(table.RowCount), (row, index) => WriteDataCells(row, table, index), parallel) - .AsTask().GetAwaiter().GetResult(); - } - else - { - for (long row = 0; row < table.RowCount; row++) - { - WriteDataRow(sheet, table, row); - } - } - + WriteSheetBody(workbook, sheet, specs, table, hasHeader, null, parallel, writeParallel); sheet.End(); sheet.Dispose(); workbook.End(); @@ -252,6 +234,45 @@ private static void WriteWorkbook(IWorkbookWriter workbook } } + internal static void WriteSheetBody(IWorkbookWriter workbook, TSheet sheet, NativeColumnSpec[] specs, NativeTable table, bool hasHeader, + NativeExcelTable? excelTable, ParallelWriteOptions? parallel, Func, Action, ParallelWriteOptions, ValueTask>? writeParallel) + where TSheet : ISheetWriter + where TRow : IRowWriter + { + ApplyTemporalStyles(workbook, sheet, table); + + if (excelTable is not null) + { + var tables = (ITableSheetWriter)sheet; + tables.BeginTable(excelTable.Name, [.. specs.Select(spec => spec.Names[0])], excelTable.Options); + WriteDataRows(sheet, table, parallel, writeParallel); + tables.EndTable(); + return; + } + if (hasHeader) + { + WriteHeaderRow(sheet, specs); + } + WriteDataRows(sheet, table, parallel, writeParallel); + } + + private static void WriteDataRows(TSheet sheet, NativeTable table, + ParallelWriteOptions? parallel, Func, Action, ParallelWriteOptions, ValueTask>? writeParallel) + where TSheet : ISheetWriter + where TRow : IRowWriter + { + if (parallel is not null && writeParallel is not null) + { + writeParallel(sheet, RowIndices(table.RowCount), (row, index) => WriteDataCells(row, table, index), parallel) + .AsTask().GetAwaiter().GetResult(); + return; + } + for (long row = 0; row < table.RowCount; row++) + { + WriteDataRow(sheet, table, row); + } + } + private static void ApplyTemporalStyles(IWorkbookWriter workbook, TSheet sheet, NativeTable table) where TSheet : ISheetWriter where TRow : IRowWriter diff --git a/src/ExcelReader.Native/include/excelreader.def b/src/ExcelReader.Native/include/excelreader.def index 454ceb9e..18e44e32 100644 --- a/src/ExcelReader.Native/include/excelreader.def +++ b/src/ExcelReader.Native/include/excelreader.def @@ -11,6 +11,10 @@ EXPORTS xl_is_date1904 xl_sheet_visibility_at xl_sheet_index + xl_table_count + xl_table_info_at + xl_table_name_at + xl_table_index xl_rows_open xl_rows_close xl_rows_next @@ -47,6 +51,7 @@ EXPORTS xl_write_null xl_end_row xl_end_sheet + xl_write_sheet xl_close_write_handle xl_write_handle_bytes xl_csv_aggregate_file diff --git a/src/ExcelReader.Native/include/excelreader.h b/src/ExcelReader.Native/include/excelreader.h index 4b14563b..384ec470 100644 --- a/src/ExcelReader.Native/include/excelreader.h +++ b/src/ExcelReader.Native/include/excelreader.h @@ -132,6 +132,30 @@ int32_t xl_sheet_index(xl_workbook* handle, const uint8_t* name, int32_t name_le int32_t xl_is_date1904(xl_workbook* handle, int32_t* out_flag); +/* Excel tables. Passed as the `sheet` of any read call, XL_TABLE_SHEET(table) reads only the table's + range, header row first, with the column indexes of the sheet holding it. Rows and columns are + zero-based and include the header and totals rows. */ +#define XL_TABLE_SHEET(table) (-2 - (table)) + +typedef struct xl_table_info { + int32_t sheet; + int32_t first_row; + int32_t last_row; + int32_t first_column; + int32_t last_column; + int32_t header_row_count; + int32_t totals_row_count; +} xl_table_info; + +int32_t xl_table_count(xl_workbook* handle, int32_t* out_count); + +int32_t xl_table_info_at(xl_workbook* handle, int32_t index, xl_table_info* out_info); + +int32_t xl_table_name_at(xl_workbook* handle, int32_t index, uint8_t* buffer, int32_t capacity, int32_t* out_len); + +/* Matches names ignoring case; *out_index is -1 when no table has the name. */ +int32_t xl_table_index(xl_workbook* handle, const uint8_t* name, int32_t name_len, int32_t* out_index); + int32_t xl_rows_open(xl_workbook* handle, int32_t sheet, xl_rows** out_rows); int32_t xl_rows_close(xl_rows* rows); @@ -288,6 +312,27 @@ int32_t xl_write_null(xl_writer_handle* handle, int32_t type); int32_t xl_end_row(xl_writer_handle* handle); int32_t xl_end_sheet(xl_writer_handle* handle); + +#define XL_TABLE_NO_STYLE 1 +#define XL_TABLE_NO_ROW_STRIPES 2 +#define XL_TABLE_COLUMN_STRIPES 4 +#define XL_TABLE_FIRST_COLUMN 8 +#define XL_TABLE_LAST_COLUMN 16 + +typedef struct xl_sheet_options { + int32_t struct_size; + int32_t table_name_len; + const uint8_t* table_name; /* NULL: a plain sheet; otherwise an Excel table (XLSX/XLSB only) */ + int32_t table_style_len; + const uint8_t* table_style; /* NULL: TableStyleMedium2 */ + int32_t table_flags; /* XL_TABLE_* */ +} xl_sheet_options; + +/* One whole sheet, as xl_write_typed writes it; fails while a sheet is open. A NULL options writes a + plain sheet; a table needs every column named. */ +int32_t xl_write_sheet(xl_writer_handle* handle, const uint8_t* name, int32_t name_len, + const xl_column_spec* specs, const xl_table* table, const xl_sheet_options* options); + int32_t xl_close_write_handle(xl_writer_handle* handle); int32_t xl_write_handle_bytes(xl_writer_handle* handle, xl_buffer* out_buffer); diff --git a/src/ExcelReader.Native/include/excelreader.hpp b/src/ExcelReader.Native/include/excelreader.hpp index ad5b2b0e..daaa2753 100644 --- a/src/ExcelReader.Native/include/excelreader.hpp +++ b/src/ExcelReader.Native/include/excelreader.hpp @@ -202,6 +202,32 @@ namespace xl int32_t batch_rows = 0; ///< XLSX/XLSB only: rows per parallel batch. 0 uses the library default (8192). }; + /// @brief Writes a sheet as an Excel table ("Format as Table"), through WriterHandle. XLSX/XLSB only. + struct TableOptions + { + std::string_view name; ///< Unique in the workbook, ignoring case, and valid as an Excel name. Borrowed: must outlive the write. + std::optional style = "TableStyleMedium2"; ///< A built-in table style, or empty for no style. + bool row_stripes = true; ///< Band alternate rows. + bool column_stripes = false; ///< Band alternate columns. + bool first_column = false; ///< Emphasize the first column. + bool last_column = false; ///< Emphasize the last column. + }; + + /// @brief One Excel table's place in a workbook. Rows and columns are 0-based and include the + /// header and totals rows. Read its cells through Workbook::table. + struct TableInfo + { + std::string name; + int32_t index = 0; ///< Position among the workbook's tables, as Workbook::table takes it. + int32_t sheet = 0; ///< Index of the sheet holding the table. + int32_t first_row = 0; + int32_t last_row = 0; + int32_t first_column = 0; + int32_t last_column = 0; + int32_t header_row_count = 0; ///< 1, or 0 when the header row is turned off. + int32_t totals_row_count = 0; ///< 1 when the table shows a totals row. + }; + namespace detail { @@ -230,6 +256,20 @@ namespace xl return opts; } + inline xl_sheet_options to_c(const TableOptions &table) noexcept + { + xl_sheet_options opts{}; + opts.struct_size = sizeof(xl_sheet_options); + opts.table_name = reinterpret_cast(table.name.data()); + opts.table_name_len = static_cast(table.name.size()); + opts.table_style = table.style ? reinterpret_cast(table.style->data()) : nullptr; + opts.table_style_len = table.style ? static_cast(table.style->size()) : 0; + opts.table_flags = (table.style ? 0 : XL_TABLE_NO_STYLE) | (table.row_stripes ? 0 : XL_TABLE_NO_ROW_STRIPES) | + (table.column_stripes ? XL_TABLE_COLUMN_STRIPES : 0) | + (table.first_column ? XL_TABLE_FIRST_COLUMN : 0) | (table.last_column ? XL_TABLE_LAST_COLUMN : 0); + return opts; + } + inline xl_write_options to_c(const WriteOptions &options) noexcept { xl_write_options opts{}; @@ -1282,6 +1322,62 @@ namespace xl return std::optional{Sheet(handle_, index)}; } + /// @brief Every Excel table, in workbook order. + std::expected, Error> tables() const + { + int32_t count = 0; + int32_t status = xl_table_count(handle_, &count); + if (status != XL_OK) + { + return std::unexpected(detail::make_error(status)); + } + std::vector result; + for (int32_t i = 0; i < count; ++i) + { + xl_table_info raw{}; + status = xl_table_info_at(handle_, i, &raw); + if (status != XL_OK) + { + return std::unexpected(detail::make_error(status)); + } + auto name = detail::fill_string([this, i](uint8_t *buffer, int32_t capacity, int32_t *out_len) + { return xl_table_name_at(handle_, i, buffer, capacity, out_len); }); + if (!name) + { + return std::unexpected(std::move(name.error())); + } + result.push_back(TableInfo{std::move(*name), i, raw.sheet, raw.first_row, raw.last_row, raw.first_column, + raw.last_column, raw.header_row_count, raw.totals_row_count}); + } + return result; + } + + /// @brief Table `index` read as a sheet. + /// + /// Every Sheet read covers only the table's range, header row first, with the column indexes of + /// the sheet holding it; its name and visibility are that sheet's. Makes no native call: an index + /// outside the workbook is reported by the first read. + /// @param index 0-based table position. + Sheet table(int32_t index) const noexcept { return Sheet(handle_, XL_TABLE_SHEET(index)); } + + /// @brief The table called `name`, compared without regard to case, read as table() reads it. + /// @return The table's sheet, or empty when no table matches. + std::expected, Error> table_by_name(std::string_view name) const + { + int32_t index = -1; + int32_t status = xl_table_index(handle_, reinterpret_cast(name.data()), + static_cast(name.size()), &index); + if (status != XL_OK) + { + return std::unexpected(detail::make_error(status)); + } + if (index < 0) + { + return std::optional{}; + } + return std::optional{table(index)}; + } + /// @brief Whether dates count from 1904 instead of 1900. /// /// A date cell's serial number depends on it. @@ -2554,8 +2650,8 @@ namespace xl namespace detail { - template - std::expected write_rows(Destination &destination, R &&rows, const WriteOptions &options) + template + std::expected write_rows(R &&rows, Write &&write) { using T = std::remove_cvref_t>; static constexpr auto bindings = ExcelMapper::get_bindings(); @@ -2584,7 +2680,7 @@ namespace xl // refs borrows pointers into builders, so both must stay in the scope of the write call. const std::array refs = to_refs(builders, bindings, indices); - return write_columns(destination, refs, options); + return write(std::span(refs)); } } @@ -2597,7 +2693,8 @@ namespace xl template std::expected write_sheet(std::string_view path, R &&rows, const WriteOptions &options = {}) { - return detail::write_rows(path, std::forward(rows), options); + return detail::write_rows(std::forward(rows), [&](std::span columns) + { return write_columns(path, columns, options); }); } /// @brief write_sheet into memory. @@ -2607,14 +2704,15 @@ namespace xl template std::expected write_sheet(std::vector &out, R &&rows, const WriteOptions &options = {}) { - return detail::write_rows(out, std::forward(rows), options); + return detail::write_rows(std::forward(rows), [&](std::span columns) + { return write_columns(out, columns, options); }); } /// @brief Writes a workbook as calls arrive instead of building it in memory first. /// /// Per sheet: start_sheet, then for each row start_row, one write per cell from left to right, and - /// end_row; then end_sheet. A call out of order returns an error rather than corrupting the + /// end_row; then end_sheet. Or one write_columns or write_sheet per whole sheet. A call out of order returns an error rather than corrupting the /// output. Finish with finish(); destruction finishes too, but cannot report a failure. class WriterHandle { @@ -2674,6 +2772,36 @@ namespace xl /// @brief Ends the current sheet. std::expected end_sheet() { return status_result(xl_end_sheet(handle_)); } + /// @brief Writes columns as a whole sheet, as write_columns does, in one call instead of one per cell. + /// + /// Uses the parallelism of the options the handle was opened with; their sheet name does not + /// apply. Fails while a sheet started with start_sheet is open. + /// @param name The sheet's name. + /// @param columns As for write_columns. + /// @param table Writes the sheet as an Excel table (XLSX/XLSB only, every column named). + std::expected write_columns(std::string_view name, std::span columns, + const std::optional &table = std::nullopt) + { + const std::optional options = + table ? std::optional(detail::to_c(*table)) : std::nullopt; + return detail::write_table(columns, [&](const xl_column_spec *specs, const xl_table *raw) + { return xl_write_sheet(handle_, reinterpret_cast(name.data()), + static_cast(name.size()), specs, raw, + options ? &*options : nullptr); }); + } + + /// @brief Writes rows as a whole sheet, as write_sheet does, on top of write_columns. + /// @param name The sheet's name. + /// @param rows As for write_sheet. + /// @param table As for write_columns. + template + std::expected write_sheet(std::string_view name, R &&rows, + const std::optional &table = std::nullopt) + { + return detail::write_rows(std::forward(rows), [&](std::span columns) + { return write_columns(name, columns, table); }); + } + /// @brief Writes the next cell. /// @param value A string, integer, floating-point, bool or std::chrono date or time value, as a /// field of an ExcelMapper row may hold; an empty std::optional writes a blank cell. diff --git a/tests/ExcelReader.Benchmarks/ExcelReader.Benchmarks.csproj b/tests/ExcelReader.Benchmarks/ExcelReader.Benchmarks.csproj index c45e5851..a149f676 100644 --- a/tests/ExcelReader.Benchmarks/ExcelReader.Benchmarks.csproj +++ b/tests/ExcelReader.Benchmarks/ExcelReader.Benchmarks.csproj @@ -24,7 +24,7 @@ - + diff --git a/tests/ExcelReader.Tests/Native/NativeApiTests.Tables.cs b/tests/ExcelReader.Tests/Native/NativeApiTests.Tables.cs new file mode 100644 index 00000000..9ff875da --- /dev/null +++ b/tests/ExcelReader.Tests/Native/NativeApiTests.Tables.cs @@ -0,0 +1,195 @@ +using System.Text; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Writer; +using ExcelReader.Native; +using ExcelReader.Native.Reading; +using ExcelReader.Native.Writer; + +namespace ExcelReader.Tests.Native +{ + public sealed partial class NativeApiTests + { + private static byte[] WriteSalesTable(int format, int degreeOfParallelism, NativeExcelTable excelTable) + { + NativeTable sales = BuildQtyNameTable([3L, 7L], ["widget", "gadget"]); + try + { + NativeWriteOptionsRaw raw = DefaultWriteOptionsRaw(); + raw.DegreeOfParallelism = degreeOfParallelism; + raw.BatchRows = 1; + Assert.True(NativeWriteOptions.TryDecode(raw, null, out NativeWriteOptions options, out _)); + Assert.Equal(NativeStatus.Ok, WriteApi.OpenWriteHandleToMemory(format, options, out NativeWriterHandle? handle)); + Assert.NotNull(handle); + using (handle) + { + handle.StartSheet("Notas"); + handle.StartRow(); + handle.WriteString("antes"); + handle.EndRow(); + handle.EndSheet(); + handle.WriteSheet("Vendas", QtyNameSpecs(), sales, hasHeader: true, excelTable); + Assert.Equal(NativeStatus.Ok, WriteApi.GetWriteHandleBytes(handle, out byte[]? bytes)); + return bytes!; + } + } + finally + { + FreeBuiltTable(ref sales); + } + } + + private static List ReadViewRows(NativeHandle handle, int sheet) + { + List rows = []; + using NativeRowCursor cursor = OpenRows(handle, sheet); + while (ReadApi.NextRowView(cursor, out NativeRow row) == NativeStatus.Ok) + { + rows.Add(string.Join('|', ReadView(row).Select(cell => cell.Value))); + } + return rows; + } + + [Theory] + [InlineData(NativeFormat.Xlsx, 1)] + [InlineData(NativeFormat.Xlsb, 1)] + [InlineData(NativeFormat.Xlsx, 4)] + [InlineData(NativeFormat.Xlsb, 4)] + public void WriteSheet_Should_Write_An_Excel_Table_The_Table_Exports_Read_Back(int format, int degreeOfParallelism) + { + var excelTable = new NativeExcelTable("Sales", ExcelTableOptions.Default with { StyleName = "TableStyleLight9", ShowColumnStripes = true }); + byte[] bytes = WriteSalesTable(format, degreeOfParallelism, excelTable); + + using (IExcelWorkbook workbook = Excel.Open(bytes.AsMemory())) + { + ExcelTable written = Assert.Single(workbook.Tables); + Assert.Equal("TableStyleLight9", written.StyleName); + Assert.Equal(["qty", "name"], written.ColumnNames, StringComparer.Ordinal); + } + + Assert.Equal(NativeStatus.Ok, ReadApi.OpenMemory(bytes, format, out NativeHandle? handle)); + try + { + Assert.Equal(NativeStatus.Ok, ReadApi.TableCount(handle, out int count)); + Assert.Equal(1, count); + Assert.Equal(NativeStatus.Ok, ReadApi.TableInfoAt(handle, 0, out NativeTableInfo info)); + Assert.Equal((1, 0, 2, 0, 1, 1, 0), + (info.Sheet, info.FirstRow, info.LastRow, info.FirstColumn, info.LastColumn, info.HeaderRowCount, info.TotalsRowCount)); + + Span name = stackalloc byte[16]; + Assert.Equal(NativeStatus.Ok, ReadApi.TableNameAt(handle, 0, name, out int length)); + Assert.Equal("Sales", Encoding.UTF8.GetString(name[..length])); + Assert.Equal(NativeStatus.BufferTooSmall, ReadApi.TableNameAt(handle, 0, name[..2], out int needed)); + Assert.Equal(5, needed); + + Assert.Equal(NativeStatus.Ok, ReadApi.TableIndex(handle, "sales"u8, out int index)); + Assert.Equal(0, index); + Assert.Equal(NativeStatus.Ok, ReadApi.TableIndex(handle, "missing"u8, out int missing)); + Assert.Equal(-1, missing); + + Assert.Equal(["qty|name", "3|widget", "7|gadget"], ReadViewRows(handle!, ReadApi.FirstTableSheet), StringComparer.Ordinal); + Assert.Equal(NativeStatus.Ok, ReadApi.SheetNameAt(handle, ReadApi.FirstTableSheet, name, out length)); + Assert.Equal("Vendas", Encoding.UTF8.GetString(name[..length])); + } + finally + { + ReadApi.Close(handle); + } + } + + [Fact] + public void Table_Exports_Should_Reject_A_Table_That_Does_Not_Exist() + { + byte[] bytes = WriteSalesTable(NativeFormat.Xlsx, 1, new NativeExcelTable("Sales", ExcelTableOptions.Default)); + Assert.Equal(NativeStatus.Ok, ReadApi.OpenMemory(bytes, NativeFormat.Xlsx, out NativeHandle? handle)); + try + { + Assert.Equal(NativeStatus.Error, ReadApi.TableInfoAt(handle, 1, out _)); + Assert.Equal(NativeStatus.InvalidArgument, ReadApi.TableInfoAt(handle, -1, out _)); + Assert.Equal(NativeStatus.Error, ReadApi.OpenRows(handle, ReadApi.FirstTableSheet - 1, out _)); + Assert.Equal(NativeStatus.InvalidArgument, ReadApi.OpenRows(handle, -1, out _)); + } + finally + { + ReadApi.Close(handle); + } + } + + [Theory] + [InlineData(NativeFormat.Xls)] + [InlineData(NativeFormat.Csv)] + public void WriteSheet_Should_Refuse_An_Excel_Table_Outside_Xlsx_And_Xlsb(int format) + { + NativeTable sales = BuildQtyNameTable([3L], ["widget"]); + try + { + Assert.Equal(NativeStatus.Ok, WriteApi.OpenWriteHandleToMemory(format, DefaultWriteOptions(), out NativeWriterHandle? handle)); + Assert.NotNull(handle); + using (handle) + { + Assert.Throws(() => + handle.WriteSheet("Vendas", QtyNameSpecs(), sales, hasHeader: true, new NativeExcelTable("Sales", ExcelTableOptions.Default))); + handle.WriteSheet("Vendas", QtyNameSpecs(), sales, hasHeader: true, excelTable: null); + } + } + finally + { + FreeBuiltTable(ref sales); + } + } + + [Fact] + public unsafe void SheetOptions_Should_Decode_A_Table_And_Its_Flags() + { + fixed (byte* name = "Sales"u8, style = "TableStyleDark1"u8) + { + var raw = new NativeSheetOptionsRaw + { + StructSize = sizeof(NativeSheetOptionsRaw), + TableName = name, + TableNameLen = 5, + TableStyle = style, + TableStyleLen = 15, + TableFlags = NativeExcelTable.NoRowStripes | NativeExcelTable.FirstColumn, + }; + Assert.True(NativeExcelTable.TryRead(&raw, out NativeExcelTable? table, out _)); + Assert.Equal(new NativeExcelTable("Sales", new ExcelTableOptions + { + StyleName = "TableStyleDark1", + ShowRowStripes = false, + ShowFirstColumn = true, + }), table); + + raw.TableStyle = null; + raw.TableFlags = NativeExcelTable.NoStyle; + Assert.True(NativeExcelTable.TryRead(&raw, out table, out _)); + Assert.Null(table!.Options.StyleName); + + Assert.True(NativeExcelTable.TryRead(null, out table, out _)); + Assert.Null(table); + } + } + + [Fact] + public unsafe void SheetOptions_Should_Reject_Contradictory_Or_Unknown_Settings() + { + fixed (byte* name = "Sales"u8, style = "TableStyleDark1"u8) + { + NativeSheetOptionsRaw valid = new() { StructSize = sizeof(NativeSheetOptionsRaw), TableName = name, TableNameLen = 5 }; + NativeSheetOptionsRaw[] invalid = + [ + valid with { StructSize = 8 }, + valid with { TableFlags = 64 }, + valid with { TableStyle = style, TableStyleLen = 15, TableFlags = NativeExcelTable.NoStyle }, + valid with { TableName = null, TableFlags = NativeExcelTable.ColumnStripes }, + valid with { TableNameLen = -1 }, + ]; + foreach (NativeSheetOptionsRaw raw in invalid) + { + NativeSheetOptionsRaw copy = raw; + Assert.False(NativeExcelTable.TryRead(©, out _, out string? error)); + Assert.NotEmpty(error); + } + } + } + } +} diff --git a/tests/ExcelReader.Tests/Native/NativeApiTests.Writer.cs b/tests/ExcelReader.Tests/Native/NativeApiTests.Writer.cs index 6e074684..70926917 100644 --- a/tests/ExcelReader.Tests/Native/NativeApiTests.Writer.cs +++ b/tests/ExcelReader.Tests/Native/NativeApiTests.Writer.cs @@ -1087,6 +1087,87 @@ public void WriteTyped_Should_Write_No_Header_Row_When_Specs_Are_Unnamed() } } + [Theory] + [InlineData(NativeFormat.Xlsx, 1)] + [InlineData(NativeFormat.Xlsb, 1)] + [InlineData(NativeFormat.Xlsx, 4)] + [InlineData(NativeFormat.Xlsb, 4)] + public void WriteTable_Should_Write_Each_Table_As_A_Sheet_Beside_Streamed_Ones(int format, int degreeOfParallelism) + { + NativeTable sales = BuildQtyNameTable([3L, 7L], ["widget", "gadget"]); + NativeTable stock = BuildQtyNameTable([10L, 20L, 30L, 40L, 50L], ["a", "b", "c", "d", "e"]); + try + { + NativeWriteOptionsRaw raw = DefaultWriteOptionsRaw(); + raw.DegreeOfParallelism = degreeOfParallelism; + raw.BatchRows = 2; + Assert.True(NativeWriteOptions.TryDecode(raw, null, out NativeWriteOptions options, out _)); + Assert.Equal(NativeStatus.Ok, WriteApi.OpenWriteHandleToMemory(format, options, out NativeWriterHandle? handle)); + Assert.NotNull(handle); + + handle.WriteSheet("Vendas", QtyNameSpecs(), sales, hasHeader: true, excelTable: null); + handle.StartSheet("Resumo"); + handle.StartRow(); + handle.WriteString("total"); + handle.EndRow(); + handle.EndSheet(); + handle.WriteSheet("Estoque", QtyNameSpecs(), stock, hasHeader: false, excelTable: null); + + Assert.Equal(NativeStatus.Ok, WriteApi.GetWriteHandleBytes(handle, out byte[]? bytes)); + handle.Dispose(); + + using IExcelWorkbook workbook = Excel.Open(bytes.AsMemory()); + Assert.Equal(["Vendas", "Resumo", "Estoque"], Enumerable.Range(0, workbook.SheetCount).Select(i => workbook.SheetAt(i).Name), StringComparer.Ordinal); + Assert.Equal(["qty|name", "3|widget", "7|gadget"], ReadCells(workbook.SheetAt(0)), StringComparer.Ordinal); + Assert.Equal(["total"], ReadCells(workbook.SheetAt(1)), StringComparer.Ordinal); + Assert.Equal(["10|a", "20|b", "30|c", "40|d", "50|e"], ReadCells(workbook.SheetAt(2)), StringComparer.Ordinal); + } + finally + { + FreeBuiltTable(ref sales); + FreeBuiltTable(ref stock); + } + } + + [Fact] + public void WriteTable_Should_Throw_While_A_Streamed_Sheet_Is_Open() + { + NativeTable table = BuildQtyNameTable([3L], ["widget"]); + try + { + Assert.Equal(NativeStatus.Ok, WriteApi.OpenWriteHandleToMemory(NativeFormat.Xlsx, DefaultWriteOptions(), out NativeWriterHandle? handle)); + Assert.NotNull(handle); + using (handle) + { + handle.StartSheet("Open"); + Assert.Throws(() => handle.WriteSheet("Table", QtyNameSpecs(), table, hasHeader: true, excelTable: null)); + handle.EndSheet(); + handle.WriteSheet("Table", QtyNameSpecs(), table, hasHeader: true, excelTable: null); + } + } + finally + { + FreeBuiltTable(ref table); + } + } + + private static List ReadCells(IExcelSheet sheet) + { + List rows = []; + using IExcelRowEnumerator enumerator = sheet.GetEnumerator(); + while (enumerator.MoveNext()) + { + Row row = enumerator.Current; + string[] cells = new string[row.ColumnCount]; + for (int column = 0; column < cells.Length; column++) + { + cells[column] = row[column].GetString(); + } + rows.Add(string.Join('|', cells)); + } + return rows; + } + [Fact] public void WriteTyped_Should_Apply_The_Csv_Delimiter_Override() { diff --git a/tests/ExcelReader.Tests/Reader/CellStoredDecimalTests.cs b/tests/ExcelReader.Tests/Reader/CellStoredDecimalTests.cs new file mode 100644 index 00000000..6f55df5c --- /dev/null +++ b/tests/ExcelReader.Tests/Reader/CellStoredDecimalTests.cs @@ -0,0 +1,67 @@ +using System.Diagnostics.CodeAnalysis; +using System.Globalization; +using System.Text; +using ExcelReader.Core.Reader; + +namespace ExcelReader.Tests.Reader +{ + public class CellStoredDecimalTests + { + [Theory] + [InlineData(1.23, "1.23")] + [InlineData(0.1 + 0.2, "0.30000000000000004")] + [InlineData(1e-7, "0.0000001")] + [InlineData(-45.5, "-45.5")] + [InlineData(123456789.0, "123456789")] + [InlineData(1e20, "100000000000000000000")] + [InlineData(-0.07, "-0.07")] + [InlineData(123456789012.345, "123456789012.345")] + [InlineData(1.5e-20, "0.000000000000000000015")] + public void StoredNumberParsesAsTheDecimalOfItsXlsxText(double number, string expected) + { + string text = number.ToString("R", CultureInfo.InvariantCulture); + var stored = new Cell(CellType.Number, Encoding.ASCII.GetBytes(text), number, hasNumber: true, styleIndex: 0); + var textBacked = new Cell(CellType.Number, Encoding.ASCII.GetBytes(text)); + + Assert.True(stored.TryParse(CultureInfo.InvariantCulture, out decimal fromStored)); + Assert.True(textBacked.TryParse(CultureInfo.InvariantCulture, out decimal fromText)); + + Assert.Equal(decimal.Parse(expected, CultureInfo.InvariantCulture), fromStored); + Assert.Equal(fromText, fromStored); + } + + [Fact] + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "Needs a reproducible seeded PRNG for deterministic test content, not cryptographic randomness.")] + public void RandomStoredNumbersMatchTheirShortestText() + { + var random = new Random(20261022); + for (int i = 0; i < 20_000; i++) + { + double number = i % 2 == 0 + ? Math.Round((random.NextDouble() - 0.5) * Math.Pow(10, random.Next(-3, 12)), random.Next(0, 8)) + : BitConverter.Int64BitsToDouble(random.NextInt64()) % 1e18; + if (!double.IsFinite(number)) + { + continue; + } + string text = number.ToString("R", CultureInfo.InvariantCulture); + var stored = new Cell(CellType.Number, ReadOnlySpan.Empty, number, hasNumber: true, styleIndex: 0); + + Assert.True(stored.TryParse(CultureInfo.InvariantCulture, out decimal fromStored), text); + Assert.Equal(decimal.Parse(text, NumberStyles.Float, CultureInfo.InvariantCulture), fromStored); + } + } + + [Theory] + [InlineData(double.NaN)] + [InlineData(double.PositiveInfinity)] + [InlineData(1e30)] + public void StoredNumberOutsideDecimalRangeDoesNotParse(double number) + { + var stored = new Cell(CellType.Number, ReadOnlySpan.Empty, number, hasNumber: true, styleIndex: 0); + + Assert.False(stored.TryParse(CultureInfo.InvariantCulture, out decimal _)); + } + } +} diff --git a/tests/ExcelReader.Tests/Reader/Internal/ReadOnlyMemoryStreamTests.cs b/tests/ExcelReader.Tests/Reader/Internal/ReadOnlyMemoryStreamTests.cs new file mode 100644 index 00000000..c2e3c1a4 --- /dev/null +++ b/tests/ExcelReader.Tests/Reader/Internal/ReadOnlyMemoryStreamTests.cs @@ -0,0 +1,32 @@ +using ExcelReader.Core.Reader.Internal; + +namespace ExcelReader.Tests.Reader.Internal +{ + public sealed class ReadOnlyMemoryStreamTests + { + [Fact] + public async Task Reads_Seeks_And_Stops_At_The_End_Of_Memory_Without_An_Array() + { + using var manager = new NonArrayMemoryManager([1, 2, 3, 4, 5]); + using var stream = new ReadOnlyMemoryStream(manager.Memory); + byte[] buffer = new byte[3]; + + Assert.Equal(5, stream.Length); + Assert.Equal(3, stream.Read(buffer, 0, 3)); + Assert.Equal([1, 2, 3], buffer); + Assert.Equal(4, stream.ReadByte()); + Assert.Equal(1, await stream.ReadAsync(buffer.AsMemory(), TestContext.Current.CancellationToken)); + Assert.Equal(5, buffer[0]); + Assert.Equal(-1, stream.ReadByte()); + Assert.Equal(0, stream.Read(buffer)); + + Assert.Equal(3, stream.Seek(-2, SeekOrigin.End)); + Assert.Equal(4, stream.ReadByte()); + Assert.Equal(1, stream.Seek(-3, SeekOrigin.Current)); + stream.Position = 10; + Assert.Equal(0, stream.Read(buffer)); + Assert.Throws(() => stream.Seek(-1, SeekOrigin.Begin)); + Assert.False(stream.CanWrite); + } + } +} diff --git a/tests/ExcelReader.Tests/Reader/Utf8StringCacheTests.cs b/tests/ExcelReader.Tests/Reader/Utf8StringCacheTests.cs index 6ed9cf32..a369316f 100644 --- a/tests/ExcelReader.Tests/Reader/Utf8StringCacheTests.cs +++ b/tests/ExcelReader.Tests/Reader/Utf8StringCacheTests.cs @@ -44,36 +44,55 @@ public void ValueLongerThanCapBypassesCacheButStillCorrect() } [Fact] - public void BucketCollisionEvictsOlderEntryWithoutCorruptingEither() + public void ValuesSharingABucketStayCachedTogether() { - const int bucketCount = 1024; - byte[] keyA = Encoding.UTF8.GetBytes("v0"); - int bucketOfA = BucketOf(keyA, bucketCount); - byte[] keyB = keyA; - int i = 1; - while (true) + byte[][] keys = KeysInOneBucket(2); + var cache = new Utf8StringCache(); + string a = cache.GetOrAdd(keys[0]); + string b = cache.GetOrAdd(keys[1]); + Assert.Same(a, cache.GetOrAdd(keys[0])); + Assert.Same(b, cache.GetOrAdd(keys[1])); + } + + [Fact] + public void OverfullBucketEvictsOldestEntryWithoutCorruptingAny() + { + byte[][] keys = KeysInOneBucket(5); + var cache = new Utf8StringCache(); + string first = cache.GetOrAdd(keys[0]); + foreach (byte[] key in keys[1..]) + { + Assert.Equal(Encoding.UTF8.GetString(key), cache.GetOrAdd(key)); + } + string again = cache.GetOrAdd(keys[0]); + Assert.Equal(first, again); + Assert.NotSame(first, again); + foreach (byte[] key in keys) + { + Assert.Equal(Encoding.UTF8.GetString(key), cache.GetOrAdd(key)); + } + } + + private static byte[][] KeysInOneBucket(int count) + { + var keys = new List { Encoding.UTF8.GetBytes("v0") }; + int bucket = BucketOf(keys[0]); + for (int i = 1; keys.Count < count; i++) { byte[] candidate = Encoding.UTF8.GetBytes("v" + i.ToString(System.Globalization.CultureInfo.InvariantCulture)); - if (BucketOf(candidate, bucketCount) == bucketOfA && !candidate.AsSpan().SequenceEqual(keyA)) + if (BucketOf(candidate) == bucket) { - keyB = candidate; - break; + keys.Add(candidate); } - i++; } - var cache = new Utf8StringCache(); - string a = Encoding.UTF8.GetString(keyA); - string b = Encoding.UTF8.GetString(keyB); - Assert.Equal(a, cache.GetOrAdd(keyA)); - Assert.Equal(b, cache.GetOrAdd(keyB)); - Assert.Equal(a, cache.GetOrAdd(keyA)); + return [.. keys]; } - private static int BucketOf(ReadOnlySpan utf8, int bucketCount) + private static int BucketOf(ReadOnlySpan utf8) { HashCode hash = default; hash.AddBytes(utf8); - return hash.ToHashCode() & (bucketCount - 1); + return hash.ToHashCode() & 1023; } [Fact] diff --git a/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs b/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs index de083346..cfac93d5 100644 --- a/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs +++ b/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs @@ -851,24 +851,6 @@ private static uint Crc32(ReadOnlySpan data) return ~crc; } - [Fact] - public void ZipArchiveEntryOpenSilentlyTruncatesAtDeclaredLengthUnderOverDelivery() - { - using MemoryStream built = WorkbookBuilder.Build("""1"""); - byte[] zipBytes = built.ToArray(); - uint realLength = ReadDeclaredUncompressedSize(zipBytes, "xl/workbook.xml"); - Assert.True(realLength > 0); - PatchCentralDirectoryUInt32(zipBytes, "xl/workbook.xml", fieldOffset: 24, realLength - 1); - - using var ms = new MemoryStream(zipBytes); - using var zip = new ZipArchive(ms, ZipArchiveMode.Read); - ZipArchiveEntry entry = zip.GetEntry("xl/workbook.xml")!; - using Stream stream = entry.Open(); - byte[] buffer = new byte[(int)entry.Length]; - stream.ReadExactly(buffer); - Assert.Equal(-1, stream.ReadByte()); - } - [Fact] public void DeflatedEntryCutShortIsRejectedWhenOpenedAsAPart() { @@ -952,12 +934,6 @@ private static byte[] BuildZipWithCutDeflatedEntry(string entryName) return zipBytes; } - private static uint ReadDeclaredUncompressedSize(byte[] zipBytes, string entryName) - { - int cdOffset = FindCentralDirectoryOffset(zipBytes, entryName); - return BinaryPrimitives.ReadUInt32LittleEndian(zipBytes.AsSpan(cdOffset + 24, 4)); - } - private static byte[] ReadViaZipArchive(byte[] zipBytes, string entryName) { using var ms = new MemoryStream(zipBytes); diff --git a/tests/ExcelReader.Tests/Writer/Parallel/WorkerStringCacheTests.cs b/tests/ExcelReader.Tests/Writer/Parallel/WorkerStringCacheTests.cs new file mode 100644 index 00000000..d5502fc6 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Parallel/WorkerStringCacheTests.cs @@ -0,0 +1,89 @@ +using System.Globalization; +using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Parallel; + +namespace ExcelReader.Tests.Writer.Parallel +{ + public class WorkerStringCacheTests + { + [Fact] + public void GetOrAdd_Should_Not_Allocate_For_A_Text_The_Table_Already_Holds_When_The_Local_Cache_Is_Full() + { + var table = new SharedStringTable(); + var cache = new WorkerStringCache(table); + for (int i = 0; i < 65_536; i++) + { + cache.GetOrAdd(i.ToString(CultureInfo.InvariantCulture).AsSpan()); + } + int expected = table.GetOrAdd("already-shared"); + ReadOnlySpan text = "already-shared".AsSpan(); + + cache.GetOrAdd(text); + long before = GC.GetAllocatedBytesForCurrentThread(); + int index = cache.GetOrAdd(text); + long allocated = GC.GetAllocatedBytesForCurrentThread() - before; + + Assert.Equal(expected, index); + Assert.Equal(0, allocated); + Assert.Equal(65_537, table.UniqueCount); + } + + [Fact] + public void GetOrAdd_Should_Answer_A_Span_From_The_Local_Cache_And_Count_References_On_Flush() + { + var table = new SharedStringTable(); + var cache = new WorkerStringCache(table); + + int first = cache.GetOrAdd("widget".AsSpan()); + int second = cache.GetOrAdd("widget"); + + Assert.Equal(first, second); + Assert.Equal(1, table.UniqueCount); + Assert.Equal(0, table.Count); + cache.FlushReferences(); + Assert.Equal(2, table.Count); + } + + [Fact] + public void A_Text_Added_By_One_Path_Should_Keep_Its_Index_On_The_Other() + { + var table = new SharedStringTable(); + + int serialA = table.GetOrAdd("a"); + int concurrentA = table.GetOrAddConcurrent("a"); + int concurrentB = table.GetOrAddConcurrent("b"); + int serialB = table.GetOrAdd("b".AsSpan()); + int serialC = table.GetOrAdd("c"); + + Assert.Equal([0, 0, 1, 1, 2], [serialA, concurrentA, concurrentB, serialB, serialC]); + Assert.Equal(3, table.UniqueCount); + using BiffBuffer xml = table.ToXlsxBytes(); + Assert.Contains("abc", System.Text.Encoding.UTF8.GetString(xml.Span), StringComparison.Ordinal); + } + + [Fact] + public void GetOrAddConcurrent_Should_Give_Every_Thread_The_Same_Dense_Indexes() + { + const int texts = 20_000; + const int threads = 8; + var table = new SharedStringTable(); + string[] values = [.. Enumerable.Range(0, texts).Select(i => "text-" + i.ToString(CultureInfo.InvariantCulture))]; + int[][] seen = new int[threads][]; + + System.Threading.Tasks.Parallel.For(0, threads, new ParallelOptions { MaxDegreeOfParallelism = threads }, thread => + { + int[] indexes = new int[texts]; + for (int step = 0; step < texts; step++) + { + int i = (step * 7919 + thread * 104_729) % texts; + indexes[i] = table.GetOrAddConcurrent(values[i]); + } + seen[thread] = indexes; + }); + + Assert.Equal(texts, table.UniqueCount); + Assert.All(seen, indexes => Assert.Equal(seen[0], indexes)); + Assert.Equal(Enumerable.Range(0, texts), seen[0].Order()); + } + } +}