Add reusable OOXML core extraction session

CHANGELOG.md

@@ -18,6 +18,10 @@ All notable changes to this project are documented in this file. This changelog
 
 ### Added
 
+- Added an internal streaming OOXML core extraction session for cells, shared
+  strings, formulas, hyperlinks, merged cells, defined names, print areas and
+  explicit tables, with shared ZIP access for optional shapes and charts.
+
 - Added regression coverage for previous-reader parity, workbook resource
   lifetime and loader counts, subprocess import boundaries, and clustering
   equivalence with SciPy-free execution.
@@ -26,6 +30,10 @@ All notable changes to this project are documented in this file. This changelog
 
 ### Fixed
 
+- Preserved accepted plain/rich string segment order and retained drawings when
+  an OOXML session resolves a non-default workbook part; avoided repeated full
+  merge/link scans when constructing sparse cell rows.
+
 - Preserved live pipeline backend overrides through lazy backend resolution.
 - Avoided materializing blank cells when reading a shared sparse worksheet.
 

dev-docs/README.md

@@ -9,6 +9,9 @@
 
 ## Reading order
 
+Implementation milestones and explicitly deferred extraction work are recorded
+in the [extraction roadmap](roadmap.md).
+
 When you change the codebase, read these materials in the following order:
 
 1. Check the public contract in `docs/`.

dev-docs/agents/contributing.md

@@ -10,7 +10,7 @@ This file contains **special guidelines** for AI coding agents such as ChatGPT,
 4. The `models` layer must remain completely side-effect-free.
 5. Do not mix I/O processing with core logic.
 6. Keep exception handling fail-safe.
-7. Update the roadmap whenever you add a new feature.
+7. Update the [developer roadmap](../roadmap.md) whenever you add a new feature.
 
 ## Reference priority
 

dev-docs/roadmap.md

@@ -0,0 +1,13 @@
+# Extraction roadmap
+
+This records the scope already established by the extraction issues. It does
+not introduce a new public mode or approve a default-backend migration.
+
+| Milestone | Status | Contract and evidence |
+| --- | --- | --- |
+| Core OOXML session foundation (Issue #149, PR #154) | Implemented; pending merge | Direct cells, strings, formulas, links, merges, names, print areas and explicit tables; shared ZIP/relationships for optional drawings. See [core OOXML specification](specs/ooxml-core-extraction.md) and `tests/core/test_ooxml_extraction_session.py`. |
+| Default-path OOXML migration | Deferred to a separate issue | Current openpyxl pipeline remains selected. Evaluate parity and capability gaps, then assess the migration's ADR and public-contract impact before changing selection. |
+| OOXML table heuristics | Outside Issue #149 | Explicit tables are available now; border/style-based heuristics remain on the existing backend. Any extension needs its own scope and parity evidence. |
+
+Current behavior is defined by [Excel extraction](specs/excel-extraction.md),
+not by the roadmap status labels.

dev-docs/specs/excel-extraction.md

@@ -30,6 +30,14 @@ This document summarizes the current specification for Excel extraction processi
   Existing helper overrides remain observable and may intentionally replace
   the optimized session path.
 
+## Internal OOXML core alternative
+
+Issue #149 adds a streaming `OoxmlExtractionSession` for core worksheet data
+and optional rich extraction through one ZIP. It is available for parity and
+future migration work; the pipeline above continues to use openpyxl.
+See [Core OOXML extraction session](ooxml-core-extraction.md) for supported
+values, formulas, relationships, tables, lifecycle and limitations.
+
 ## Coordinate System
 
 - Rows are 1-based

dev-docs/specs/ooxml-core-extraction.md

@@ -0,0 +1,157 @@
+# Core OOXML extraction session
+
+Issue: [#149](https://github.com/harumiWeb/exstruct/issues/149)
+
+## Scope and selection
+
+`exstruct.core.ooxml_session.OoxmlExtractionSession` is an internal, read-only
+alternative for `.xlsx` / `.xlsm` core extraction. It does not replace the
+openpyxl pipeline or change public models. Default-path migration is separate.
+
+```python
+from pathlib import Path
+from exstruct.core.ooxml_session import OoxmlExtractionSession
+from exstruct.core.backends.ooxml_backend import OoxmlRichBackend
+
+path = Path("book.xlsx")
+with OoxmlExtractionSession(path) as session:
+    sheets = session.sheets()
+    cells = session.extract_cells(include_links=True)
+    formulas = session.extract_formulas_map()
+    merged = session.extract_merged_cells()
+    areas = session.extract_print_areas()
+    tables = session.extract_explicit_tables()
+    names = session.defined_names()
+    charts = OoxmlRichBackend(path, session=session).extract_charts(mode="light")
+```
+
+## Resource lifetime and parsing
+
+- One lazy ZIP is owned by the context and shared by core and optional rich
+  extraction. Independent sessions have independent caches.
+- Worksheet XML and shared strings use `defusedxml.ElementTree.iterparse`.
+  Completed cell/row and shared-string elements are removed from their parents.
+- Accepted plain/rich text segments retain XML document order; phonetic
+  annotations do not contribute cell text.
+- Sparse values, formulas, merges, hyperlinks and table references are cached
+  only after complete worksheet parsing. Subsequent core methods do not reparse
+  worksheets. Memory scales with extracted data and the shared-string dictionary;
+  this is not a constant-memory iterator API.
+- Small workbook, relationship, style and table parts use safe XML trees.
+  Existing drawing parsing retains its separate streaming geometry scan.
+- Closing releases the ZIP and caches, including after context-body errors.
+  Accessing a closed session raises RuntimeError.
+- No openpyxl workbook/worksheet model is created. Existing value/formula
+  normalization and openpyxl scalar format, date and formula-translation helpers
+  are reused; openpyxl remains a dependency.
+- Core malformed XML, missing required parts and invalid scalar data raise
+  errors. Missing optional relationship parts return an empty mapping. No
+  partially parsed worksheet is cached. Rich extraction retains its existing
+  per-sheet best-effort fallback contract.
+
+## Data contract
+
+- `sheets()` returns workbook order, sheet ID, visibility, resolved part path
+  and relationship kind, including chart sheets. Core extraction excludes
+  non-worksheet sheets. Empty worksheets remain present in mappings.
+- `defined_names()` preserves raw text and optional `localSheetId`; named
+  formulas are not evaluated.
+- `extract_cells()` returns `CellRow`: rows 1-based, columns 0-based numeric-string
+  keys, existing missing-token filtering and numeric-string normalization. Sparse
+  gaps do not cause rectangular blank cell materialization.
+- Merged follower membership is indexed once per parsed sheet over stored
+  coordinates. Row/column bisect limits checks to each merge extent without
+  expanding empty rectangles. Hyperlinks use bisect over emitted row keys,
+  preserving last-link-wins behavior without scanning unrelated rows.
+- Shared strings (`s`), plain/rich inline strings (`inlineStr`), formula string
+  caches (`str`), numeric (`n`), boolean (`b`) and ISO date (`d`) values are decoded.
+  Formatting/phonetic annotations are omitted. Excel errors (`e`) are omitted,
+  matching current extraction.
+- Numeric date/time/duration styles and 1900/1904 workbook epochs are honored.
+  Styles serve scalar conversion, not border heuristics or colors.
+- Formula values use stored caches without calculation. `extract_formulas_map()`
+  groups normalized `=`-prefixed formulas at `(row, zero-based column)` coordinates.
+  Shared followers translate relative references from the shared anchor.
+  Array/dynamic formulas retain explicit anchor text without inventing follower
+  formulas. Unresolved shared followers are omitted; unsupported translation
+  is logged and skipped.
+- External hyperlink URLs are preserved exactly. Internal `location` links are
+  omitted, matching current output. Links on filtered values survive only when
+  their row has another emitted value. Range links use zero-based column keys.
+- Merged ranges contain cached raw anchor text or a space for an empty anchor;
+  covered follower values are excluded from rows.
+- Print areas come from `_xlnm.Print_Area`, respecting local scope and quoted
+  names including commas/apostrophes. Unsupported non-finite ranges are logged
+  and skipped.
+- `extract_explicit_tables()` returns referenced table names, display names,
+  ranges and column names, without the border heuristics of `table_candidates`.
+
+## Relationships
+
+`ooxml_package.read_relationships(archive, source_path)` resolves IDs to type,
+target and external status. Internal relative paths resolve against the source
+directory; absolute targets resolve from the ZIP root. External targets are
+preserved and never fetched or read as ZIP parts. These primitives serve
+hyperlinks, tables, worksheets, drawings and charts. Legacy drawing helpers
+remain available as wrappers.
+When drawing extraction shares a core session, workbook enumeration uses the
+same resolved officeDocument part as core extraction, including a relocated
+workbook. Standalone path-based drawing reads retain their conventional default.
+
+## Review follow-up validation
+
+PR #154 adds regressions for accepted interleaved text segments in shared and
+inline strings, a relocated `custom/book.xml` read before any cell extraction,
+and sheet-wide sparse merged ranges with overlapping hyperlink overrides.
+The relocated fixture includes both a chart and a shape and verifies one ZIP
+open per session. Merge coverage indexes only stored coordinates and is reused
+across linked/unlinked row extraction.
+
+A local Windows synthetic comparison of the original `807e206` row builder
+and the revised builder used 40,000 stored cells (20,000 rows, two columns),
+200 single-row merges and 200 links. Exact output equality was asserted:
+original 0.684129 s, revised first call 0.124096 s, revised cached call 0.099105 s.
+These are row-construction timings from one run, not workbook extraction or
+memory measurements.
+
+Follow-up validation: 27 focused tests passed; 1,040 full tests passed with
+12 external-runtime tests deselected and 83.57% coverage. Existing eight-workbook
+parity, Ruff, strict mypy and diff checks passed. These fixes restore extraction
+semantics and improve internal indexing; they introduce no backend-selection,
+public mode or fallback policy change and require no new ADR.
+
+## ADR assessment
+
+- Verdict: `not-needed`; next action: `no-adr`.
+- Rationale: internal parsing capabilities are added without changing backend
+  selection, priority, mode meanings, public output or fallback policy. A future
+  pipeline migration requires a fresh ADR assessment.
+- Domains: extraction implementation and resource reuse.
+- Existing decisions: ADR-0010 (light OOXML rich baseline), ADR-0002 (rich fallback).
+- Evidence triad: this spec and `dev-docs/specs/excel-extraction.md`;
+  `src/exstruct/core/ooxml_session.py`, `ooxml_package.py`, `ooxml_drawing.py`,
+  `backends/ooxml_backend.py`; `tests/core/test_ooxml_extraction_session.py`,
+  `test_ooxml_package.py`, `test_ooxml_drawing.py` and existing pipeline tests.
+
+## Validation
+
+Parity fixtures compare with `OpenpyxlBackend` for both suffixes, link flags,
+cells, formulas, merges and print areas. Additional tests cover shared/rich
+strings, cached and shared formulas, relationships, tables, archive/read reuse,
+sparse sheets, exception cleanup and XML entity rejection. Existing drawing
+and pipeline tests retain rich-artifact guarantees.
+
+```powershell
+rtk uv run --extra all pytest tests/core/test_ooxml_extraction_session.py tests/core/test_ooxml_package.py tests/core/test_ooxml_drawing.py -q
+rtk uv run --extra all pytest -m "not com and not render and not libreoffice" -q --cov=exstruct --cov-report=term --cov-fail-under=80
+rtk uv run ruff check . --no-fix
+rtk uv run --extra all mypy src/exstruct --strict
+```
+
+Local Windows/Python 3.11 verification: 23 focused tests passed; 1,036 tests
+passed with 12 external-runtime tests deselected, total coverage 83.51% (80%
+gate passed). Ruff and strict mypy passed. Excel COM/render and real LibreOffice
+runtime verification are not part of this internal-parser validation.
+An additional comparison of all seven tracked `sample/**/*.xlsx` workbooks and
+`tests/assets/multiple_print_ranges_4sheets.xlsx` matched cells with links,
+formula maps, print areas and merged-range sets against `OpenpyxlBackend`.

src/exstruct/core/backends/ooxml_backend.py

@@ -4,6 +4,7 @@
 
 import logging
 from pathlib import Path
+from typing import TYPE_CHECKING
 from zipfile import BadZipFile
 
 from defusedxml import ElementTree
@@ -13,6 +14,9 @@
 from .base import ChartData, RichBackend, ShapeData
 from .ooxml_shapes import _build_shapes_from_ooxml
 
+if TYPE_CHECKING:
+    from ..ooxml_session import OoxmlExtractionSession
+
 logger = logging.getLogger(__name__)
 
 _OOXML_SUFFIXES = {".xlsx", ".xlsm"}
@@ -21,10 +25,15 @@
 class OoxmlRichBackend(RichBackend):
     """Best-effort rich extraction backed only by OOXML workbook parts."""
 
-    def __init__(self, file_path: Path) -> None:
+    def __init__(
+        self, file_path: Path, *, session: OoxmlExtractionSession | None = None
+    ) -> None:
         """Store the workbook path for lazy OOXML parsing."""
 
         self.file_path = file_path
+        if session is not None and session.file_path.resolve() != file_path.resolve():
+            raise ValueError("OOXML session belongs to a different workbook")
+        self.session = session
         self._drawings: dict[str, SheetDrawingData] | None = None
 
     def extract_shapes(self, *, mode: str) -> ShapeData:
@@ -77,7 +86,11 @@ def _read_drawings(self) -> dict[str, SheetDrawingData]:
             self._drawings = {}
             return self._drawings
         try:
-            self._drawings = read_sheet_drawings(self.file_path)
+            self._drawings = (
+                self.session.read_drawings()
+                if self.session is not None
+                else read_sheet_drawings(self.file_path)
+            )
         except (
             BadZipFile,
             ElementTree.ParseError,