Skip to content

Default light extraction to direct OOXML #156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
harumiWeb merged 4 commits into from
Oct 3, 2026
Merged

Default light extraction to direct OOXML #156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d27ca6d
feat: default light extraction to direct OOXML (#150)
Oct 3, 2026
998f922
docs: sync issue 150 performance table with final baseline
Oct 3, 2026
f86efde
perf: index merged ranges for OOXML table scanning
Oct 3, 2026
db6560c
refactor: make OOXML validation and override references explicit
Oct 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@ All notable changes to this project are documented in this file. This changelog

### Changed

- Switched `.xlsx` / `.xlsm` `light` extraction to the direct OOXML core and
rich pipeline using one shared ZIP. Unsupported OOXML and `colors_map` opt-in
restart the complete openpyxl compatibility pipeline with an
`ooxml_compatibility` warning; per-sheet drawing resilience is preserved.
- Kept ADR-0012's configured SciPy selection for other modes and compatibility
extraction while the direct light table heuristic selects Python BFS per
call without changing the environment variable.
- Replaced production pandas cell extraction with direct openpyxl worksheet
iteration and lazy xlrd reading for legacy `.xls` files, preserving existing
cell normalization and serialized output.
Expand All @@ -18,6 +25,11 @@ All notable changes to this project are documented in this file. This changelog

### Added

- Added light-pipeline parity, archive lifecycle and compatibility-restart
coverage, plus a comparison runner for the previous openpyxl path. The
recorded seven-input Windows benchmark and raw baseline are available in
`benchmark/issue150-results.md` and
`benchmark/baselines/issue150-2026-10-03-windows.json`.
- Added an internal streaming OOXML core extraction session for cells, shared
strings, formulas, hyperlinks, merged cells, defined names, print areas and
explicit tables, with shared ZIP access for optional shapes and charts.
Expand Down
12 changes: 6 additions & 6 deletions README.ja.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,12 @@ LLM/RAG 向けに検出ヒューリスティックや出力モードを調整で
- **Excel → 構造化 JSON**
- セル、図形、チャート、SmartArt、テーブル候補、セル結合範囲、印刷範囲/自動改ページ範囲をシート単位・範囲単位で出力。
- **出力モード**
- `light`: セル＋テーブル候補＋印刷範囲+図形・グラフ（OOXML直接解析によるbest-effort）
- `light`: `.xlsx` / `.xlsm` のセル・テーブル候補・印刷範囲・図形/グラフをOOXML直接解析で抽出（rich artifactはbest-effort）
- `libreoffice`: `.xlsx/.xlsm` 向けの best-effort 非 COM モード。LibreOffice runtime があれば結合セル・図形・コネクタ・チャートを追加
- `standard`: Excel COM でテキスト付き図形＋矢印、チャート、SmartArt、セル結合範囲
- `verbose`: 全図形を幅高さ付きで出力、セルのハイパーリンクも出力。
- **数式取得**
- `formulas_map`（数式文字列 → セル座標）を openpyxl/COM で取得。`verbose` 既定、`include_formulas_map` で制御。
- `formulas_map`（数式文字列 → セル座標）を OOXML または選択された openpyxl/COM 経路で取得。`verbose` 既定、`include_formulas_map` で制御。
- **フォーマット**
- JSON（デフォルトはコンパクト、`--pretty` で整形）、YAML、TOON（任意依存）。
- **ワークブック編集インターフェース**
Expand All @@ -49,7 +49,7 @@ LLM/RAG 向けに検出ヒューリスティックや出力モードを調整で
- **ハイパーリンク抽出**
- `verbose` モード（または `include_cell_links=True` 指定）でセルのリンクを `links` に出力。
- **安全なフォールバック**
- Excel COM または LibreOffice runtime が不在でもプロセスは落ちず、OOXML直接解析に切り替えます。
- `.xlsx` / `.xlsm` の `light` はOOXMLを直接解析します。未対応OOXMLまたは `colors_map` 指定時はpartial結果を破棄し、openpyxl互換パイプライン全体を最初から実行して `ooxml_compatibility` warningを記録します。1シートのdrawing失敗は従来どおりそのシートだけbest-effortで処理します。

## インストール

Expand All @@ -60,7 +60,7 @@ pip install exstruct
オプション依存:

- YAML: `pip install pyyaml`
- 罫線クラスタリングの高速化: `pip install exstruct[fast]`（SciPy）。未導入時は既存の Python 実装へフォールバックします。NumPy は引き続き必須依存です。
- 罫線クラスタリングの高速化: `pip install exstruct[fast]`（SciPy）。通常の直接OOXML light抽出ではPythonの罫線検出を使います。SciPyによる高速化と `EXSTRUCT_BORDER_CLUSTER_BACKEND` の選択は、他モードと互換抽出経路で利用できます。NumPy は引き続き必須依存です。
- TOON: `pip install python-toon`
- レンダリング（PDF/PNG）: Excel + `pip install pypdfium2 pillow`（`mode=libreoffice` では非対応）
- まとめて導入: `pip install exstruct[yaml,toon,render,fast]`
Expand Down Expand Up @@ -276,14 +276,14 @@ set_table_detection_params(

## 出力モード

- **light**: セル＋テーブル候補＋`.xlsx` / `.xlsm` の best-effort OOXML 図形/コネクタ/チャート（COM 不要）。
- **light**: `.xlsx` / `.xlsm` のセル・テーブル候補・印刷範囲・図形/コネクタ/チャートを直接OOXMLで抽出します。通常経路はopenpyxl、pandas、SciPy、xlwingsをimportしません。
- **standard**: テキスト付き図形＋矢印、チャート（COM ありで取得）、テーブル候補。セルのハイパーリンクは `include_cell_links=True` を指定したときのみ出力。
- **verbose**: すべての図形、チャート、`table_candidates`、ハイパーリンク、`colors_map`。

## エラーハンドリング / フォールバック

- Excel COM 不在時はセル＋テーブル候補に自動フォールバックし、`.xlsx` / `.xlsm` では利用可能な OOXML 図形/チャートも best-effort で保持します。
- rich extraction の一部が失敗しても、ExStruct はセル＋テーブル候補を返しつつ、安全に保持できる既存の best-effort artifact は残します。
- 1シートのdrawing失敗はそのシートの対象artifactだけを省き、健全なシートは維持します。未対応OOXMLまたは他のdirect pipeline stageの失敗時はpartial結果を破棄し、openpyxl互換pipeline全体を最初から実行して `ooxml_compatibility` を記録します。
- CLI はエラーを stdout/stderr に出し、失敗時は非ゼロ終了コード。

## 任意レンダリング
Expand Down
12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,16 +44,16 @@ LLM/RAG pipelines and local automation.

- **Excel -> structured JSON**: outputs cells, shapes, charts, SmartArt, table candidates, merged-cell ranges, print areas, and auto page-break areas by sheet or by area.
- **Output modes**:
- `light`: cells + table candidates + print areas + shapes/charts (best-effort via direct OOXML parsing)
- `light`: cells + table candidates + print areas + best-effort shapes/charts via direct OOXML for `.xlsx` / `.xlsm`
- `libreoffice`: best-effort non-COM mode for `.xlsx/.xlsm`. When the LibreOffice runtime is available, it adds merged cells, shapes, connectors, and charts
- `standard`: Excel COM mode with texted shapes + arrows, charts, SmartArt, and merged-cell ranges
- `verbose`: outputs all shapes with width/height and also emits cell hyperlinks
- **Formula extraction**: emits `formulas_map` (formula string -> cell coordinates) via openpyxl/COM. It is enabled by default in `verbose` and can be controlled with `include_formulas_map`.
- **Formula extraction**: emits `formulas_map` (formula string -> cell coordinates) via OOXML or the selected openpyxl/COM path. It is enabled by default in `verbose` and can be controlled with `include_formulas_map`.
- **Formats**: JSON (compact by default, `--pretty` for formatting), YAML, and TOON (optional dependencies).
- **Workbook editing interfaces**: use the editing CLI for primary ExStruct edit flows, keep MCP for host-owned safety controls, and use `exstruct.edit` only when you need the same patch contract from Python.
- **Table detection tuning**: heuristics can be adjusted dynamically through the API.
- **Hyperlink extraction**: in `verbose` mode, or with `include_cell_links=True`, cell links are emitted in `links`.
- **Safe fallback**: if Excel COM or the LibreOffice runtime is unavailable, the process does not crash and falls back to direct OOXML parsing.
- **Safe fallback**: `light` uses direct OOXML for `.xlsx` / `.xlsm` without Excel COM or LibreOffice. Unsupported OOXML or `colors_map` requests restart the complete openpyxl compatibility pipeline with an `ooxml_compatibility` warning; partial results are discarded. A drawing failure isolated to one sheet remains best-effort.

## Installation

Expand All @@ -63,7 +63,7 @@ pip install exstruct

Optional extras:

- Border-clustering acceleration: `pip install exstruct[fast]` (SciPy). Base extraction uses the existing Python fallback when SciPy is unavailable; NumPy remains a core dependency.
- Border-clustering acceleration: `pip install exstruct[fast]` (SciPy). Normal direct light extraction uses Python clustering. SciPy acceleration and `EXSTRUCT_BORDER_CLUSTER_BACKEND` selection apply to other modes and compatibility extraction; NumPy remains a core dependency.
- YAML: `pip install pyyaml`
- TOON: `pip install python-toon`
- Rendering (PDF/PNG): Excel + `pip install pypdfium2 pillow` (`mode=libreoffice` is not supported)
Expand Down Expand Up @@ -280,14 +280,14 @@ Higher values reduce false positives. Lower values reduce missed detections.

## Output Modes

- **light**: cells + table candidates + best-effort OOXML shapes/connectors/charts for `.xlsx` / `.xlsm` (no COM required).
- **light**: direct OOXML cells + table candidates + print areas and best-effort shapes/connectors/charts for `.xlsx` / `.xlsm`. The normal path does not import openpyxl, pandas, SciPy, or xlwings.
- **standard**: texted shapes + arrows, charts (when COM is available), and table candidates. Cell hyperlinks are emitted only when `include_cell_links=True`.
- **verbose**: all shapes, charts, `table_candidates`, hyperlinks, and `colors_map`.

## Error Handling / Fallback

- If Excel COM is unavailable, extraction falls back to cells + table candidates automatically; `.xlsx` / `.xlsm` still preserve best-effort OOXML shapes/charts when available.
- If a rich-extraction step fails, ExStruct still returns cells + table candidates and keeps any already recovered best-effort artifacts where safe.
- A per-sheet drawing failure omits that sheet's affected drawing artifacts while preserving healthy sheets. An unsupported OOXML construct or failure in any other direct pipeline stage discards all partial OOXML data, restarts the complete openpyxl compatibility pipeline, and logs `ooxml_compatibility`.
- The CLI writes errors to stdout/stderr and exits with a non-zero status on failure.

## Optional Rendering
Expand Down
Loading
Loading