Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 178 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ version = "0.2.4"
edition = "2024"
# Edition 2024 needs 1.85; zip raises it to 1.88.
rust-version = "1.88"
description = "Convert documents (doc, docx, odt, rtf, epub, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown"
description = "Convert documents (doc, docx, odt, rtf, epub, chm, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown"
license = "MIT"
repository = "https://github.com/firecrawl/anydoc"
readme = "README.md"
Expand All @@ -26,6 +26,7 @@ cfb = "0.14.0"
csv = "1.4.0"
flate2 = "1"
encoding_rs = "0.8.35"
html5ever = "0.40"
log = "0.4"
pdf-inspector = "1.14.2"
quick-xml = "0.41.0"
Expand Down
24 changes: 21 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![skills.sh](https://skills.sh/b/firecrawl/anydoc)](https://skills.sh/firecrawl/anydoc)

Fast Rust library that converts documents (Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF) into clean GitHub-Flavored Markdown. Includes bindings for [Node.js](node/README.md), [Python](python/README.md), and the [browser](wasm/README.md) (WebAssembly).
Fast Rust library that converts documents (Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CHM, CSV, and PDF) into clean GitHub-Flavored Markdown. Includes bindings for [Node.js](node/README.md), [Python](python/README.md), and the [browser](wasm/README.md) (WebAssembly).

Built by [Firecrawl](https://firecrawl.dev) to turn any office document into LLM-ready Markdown in single-digit milliseconds, with one consistent output no matter which format goes in. It powers [Firecrawl Parse](https://firecrawl.dev/parse), so if you'd rather not run it yourself, the hosted API gives you the same conversion plus our OCR models for the scanned pages anydoc can't read on its own.

Expand Down Expand Up @@ -166,9 +166,20 @@ Only documents that need OCR leave the machine, and the whole document goes, sin
| OpenDocument | `.odt`, `.ods`, `.odp` |
| Rich Text Format | `.rtf` |
| EPUB | `.epub` |
| Compiled HTML Help | `.chm` |
| CSV | `.csv` |
| PDF | `.pdf` |

CHM topics are read in `.hhc` table-of-contents order, with the hierarchy
preserved as a nested list. Unlisted HTML topics are appended in stable path
order; the same fallback applies when the TOC is missing, unreadable, or binary.
Legacy HTML uses BOM/charset declarations, then the book's Windows language
encoding (including GBK/Big5). Topic links, HTML bases, and stylesheet encodings
are resolved locally. LZX supports 4/8-byte reset tables and sequential SpanInfo
streams; the private decoder adaptation and its MIT notice are under
`src/formats/chm/lzx/`. Embedded images remain available on `Document.assets`,
as with the other formats.

## Benchmark

anydoc is measured against six other converters on 100 real-world documents spanning fourteen formats. Scores run from 0 to 100, higher is better; speed is the median time to convert one document.
Expand Down Expand Up @@ -209,7 +220,7 @@ Speed is one warm conversion per document on a Ryzen 9 9950X3D (Windows 11, 64 G

## Format detection

The format is read from the file content, using the marker its specification designates: the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types. CSV has no such marker, so the extension or an explicit format names it instead.
The format is read from the file content, using the marker its specification designates: the CHM `ITSF` signature, the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types. CSV has no such marker, so the extension or an explicit format names it instead.

```rust
Format::from_bytes(&bytes); // Some(Format::Docx), or None when nothing matches
Expand Down Expand Up @@ -261,7 +272,7 @@ document bytes
├─► format detection → content markers, not the extension
│
├─► format parser → one per format (doc, docx, ppt, pptx, xls,
│ xlsx, odt/ods/odp, rtf, epub, csv)
│ xlsx, odt/ods/odp, rtf, epub, chm, csv)
│ │
│ └─► Document → shared model: blocks, inlines, tables,
│ footnotes, assets
Expand All @@ -284,6 +295,13 @@ wasm-pack build wasm --release --target web --scope firecrawl && node --test was

A committed fixture corpus under `tests/fixtures/` is snapshot-tested, `tests/robustness.rs` mutation-tests every fixture, and `fuzz/` carries cargo-fuzz targets per format. The speed and quality benchmark lives in [`bench/`](bench/README.md).

The small CHM fixtures can be rebuilt with `python3 tests/fixture-src/chm.py`.
To run the local ZXR10 manual regression without adding the manual to the repo:

```bash
ANYDOC_CHM_SAMPLE=/path/to/manual.chm cargo test --test chm local_chm_manual -- --ignored --nocapture
```

Releases are tagged `v<version>`, which publishes the crate, the npm package, and the PyPI wheels from [`.github/workflows/release.yml`](.github/workflows/release.yml). The version lives in three places, bumped together for a release:

- [`Cargo.toml`](Cargo.toml): the crate
Expand Down
30 changes: 29 additions & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

Benchmarks anydoc against well-known document-to-markdown converters on the
`samples/` corpus, for both speed and quality (deterministic metrics + LLM judge).
Office, text, and presentation documents; PDFs are out of scope.
Office, text, presentation, and CHM documents; PDFs are out of scope. CHM runs
only against anydoc; the competitor format lists are unchanged.

The corpus is not redistributable and is not in the repo. The harness reads
whatever documents are in `samples/`.
Expand Down Expand Up @@ -48,6 +49,33 @@ python report.py # aggregates everything into out/report.m
All artifacts land in `out/` (gitignored). `convert.py` appends to
`out/timings.jsonl`; `report.py` keeps only the latest row per (tool, file).

## CHM time and memory

The standard harness accepts `--formats chm`. For per-process peak memory and
repeatable comparisons with an external CHM, use the dependency-free macOS/Linux
runner from the repository root:

```
cargo build --release --example convert
python3 -m unittest discover -s bench -p test_chm.py
python3 bench/chm.py /path/to/book.chm --warmup 1 --iters 3
python3 bench/chm.py /path/to/book.chm --binary /path/to/baseline/convert
```

`chm.py` prints one JSONL record with every run and their medians. Redirect stdout
to retain a local result. It never copies the input or saves converted contents.
Each warmup and measured run starts a fresh converter process; warmups warm OS
caches without retaining converter state. Run comparisons serially on the same
machine/build settings.
`conversion_ms` is the CLI's rounded API time (CHM to Markdown, including the
document model); `wall_ms` includes process startup, input I/O, conversion, writing
Markdown to the null device, and process teardown. Assets are not extracted.
`peak_rss_bytes` is the converter process's peak resident set from `wait4`, not
total allocated memory or the Python runner's memory: macOS reports bytes, Linux
reports KiB, normalized here to bytes. Results include binary path, platform, and
input size; record the tested commit/build separately. No wall-clock thresholds
are enforced in CI. Use a local private CHM path without adding it to the corpus.

## Methodology notes

- **Speed**: min of `--iters` warm runs, and the published numbers use `--iters 1`,
Expand Down
Loading