Rich table cells: [[LISTTABLE]] with lists, header rows, merged cells and diagrams

.github/workflows/tools.yml

@@ -1,8 +1,9 @@
 name: Tool image
 
 # Builds the tool container and draws with it, which the main CI cannot: its tests run the tools on the
-# host. Runs when the image definition or the diagram tools change, once a week (a tool can break without
-# a change here, through a Debian package), and by hand. The image takes 5 to 30 minutes to build.
+# host. Runs when the image definition, the diagram tools or the pandoc finishing change, once a week (a
+# tool can break without a change here, through a Debian package), and by hand. The image takes 5 to 30
+# minutes to build.
 on:
   pull_request:
     paths:
@@ -12,6 +13,8 @@ on:
       - "src/py_doc/deck.py"
       - "src/py_doc/diagram_api.py"
       - "src/py_doc/icons.py"
+      - "src/py_doc/listtable.py"
+      - "src/py_doc/pandoc/**"
       - ".github/workflows/tools.yml"
   schedule:
     - cron: "17 5 * * 1"  # Mondays
@@ -42,6 +45,22 @@ jobs:
       - name: Every diagram type, through the container (the guide has one of each)
         working-directory: examples
         run: py-doc build guide -f html -f docx --keep-going
+      - name: The guide's [[LISTTABLE]] in Word, with the pandoc of the image
+        working-directory: examples
+        run: |
+          python - <<'PY'
+          from docx import Document
+          from docx.oxml.ns import qn
+          document = Document("build/guide-v1.0.docx")
+          table = next(t for t in document.tables if t.cell(0, 0).text == "Step")._tbl
+          rows = table.findall(qn("w:tr"))
+          assert table.tblPr.find(qn("w:tblLayout")).get(qn("w:type")) == "fixed", "a fixed layout"
+          assert all(r.find(f"{qn('w:trPr')}/{qn('w:cantSplit')}") is None for r in rows[1:]), "body rows may split"
+          assert table.xpath(".//w:vMerge") and table.xpath(".//w:gridSpan[@w:val='2']"), "merged cells"
+          assert table.xpath(".//wp:inline"), "the diagram in its cell"
+          assert table.xpath(".//w:rStyle[@w:val='HighlightYellow']"), "the highlight style"
+          print(f"{len(rows)} rows, fixed layout, merges, a diagram and the highlight style")
+          PY
       - name: The posters and their PDFs, through the container
         working-directory: examples
         run: py-doc poster --pdf

README.md

@@ -71,7 +71,7 @@ While writing, `py-doc docx guide.md --watch` (or `slides`, or `poster`) builds
 
 py-doc documents itself with py-doc. The sources are in [`docs/`](docs), and the [`py-doc.yml`](py-doc.yml) at the root of this repository builds them: `py-doc build` for Word, HTML and PDF, `py-doc site build` for the web site.
 
-- **User guide**: [introduction](docs/guide/00_introduction.md), [installation](docs/guide/01_install.md), [projects and `py-doc.yml`](docs/guide/02_projects.md), [writing](docs/guide/03_writing.md), [outputs and templates](docs/guide/04_outputs.md), [importing](docs/guide/05_import.md), [C4 architecture](docs/guide/06_c4.md), [linting](docs/guide/07_lint.md), [web sites](docs/guide/08_sites.md), [reference](docs/guide/09_reference.md), [legacy build files and document numbers](docs/guide/10_legacy.md), [generators, the build pipeline and analysis](docs/guide/11_pipeline.md), [posters](docs/guide/12_posters.md), [writing a poster](docs/guide/13_writing_posters.md), [writing a document](docs/guide/14_writing_documents.md), [writing a slide deck](docs/guide/15_writing_slides.md).
+- **User guide**: [introduction](docs/guide/00_introduction.md), [installation](docs/guide/01_install.md), [projects and `py-doc.yml`](docs/guide/02_projects.md), [writing](docs/guide/03_writing.md) (with [rich table cells](docs/guide/03_writing.md#rich-table-cells) of paragraphs, lists, merged cells and diagrams), [outputs and templates](docs/guide/04_outputs.md), [importing](docs/guide/05_import.md), [C4 architecture](docs/guide/06_c4.md), [linting](docs/guide/07_lint.md), [web sites](docs/guide/08_sites.md), [reference](docs/guide/09_reference.md), [legacy build files and document numbers](docs/guide/10_legacy.md), [generators, the build pipeline and analysis](docs/guide/11_pipeline.md), [posters](docs/guide/12_posters.md), [writing a poster](docs/guide/13_writing_posters.md), [writing a document](docs/guide/14_writing_documents.md), [writing a slide deck](docs/guide/15_writing_slides.md).
 - **[Requirements](docs/requirements/00_introduction.md)**: what py-doc must do and how well, with the status of every requirement and the open issues.
 - **[Architecture](docs/architecture/00_introduction.md)**: how py-doc is built, with C4 chapters generated from [its Structurizr model](docs/architecture/workspace.dsl), and the decisions behind it.
 - **[Examples](examples/README.md)**: a small project that uses every feature, with a command for each output.

docs/architecture/02_pipeline.md

@@ -26,7 +26,7 @@ A source with `structurizr:` is replaced by Markdown files before anything else
 
 ## 3. Joining
 
-`pandoc/source.py` reads each included source and expands its directives (`directives.py`: includes, conditions, variables). It splits off the front matter, applies the title and heading offset, and puts a page break block before sources that ask for one. The metadata of the document is merged from the configuration, the first source's front matter and its first heading (`build.py`). Headings are found line by line outside code fences, so a `#` in a code block is never mistaken for one.
+`pandoc/source.py` reads each included source and expands its directives (`directives.py`: includes, conditions, variables). The same pass turns each `[[LISTTABLE]]` into a pandoc grid table in a Div, and each `[[BLOCK:]]` into a Div of its own (`listtable.py`). Done here, their warnings name the file and line, even inside an included file. It splits off the front matter, applies the title and heading offset, and puts a page break block before sources that ask for one. The metadata of the document is merged from the configuration, the first source's front matter and its first heading (`build.py`). Headings are found line by line outside code fences, so a `#` in a code block is never mistaken for one.
 
 ## 4. Diagrams
 
@@ -38,13 +38,14 @@ A source with `structurizr:` is replaced by Markdown files before anything else
 
 | Filter | Job | Formats |
 |---|---|---|
+| `listtable.lua` | The blocks of `[[LISTTABLE]]` cells move into their cells, without captions. In Word a hidden bookmark marks the table for the finishing | all, and site pages |
 | `admonitions.lua` | Callouts in both spellings become a box of their kind | all |
 | `pagebreak.lua` | Page break blocks become a page break, or a new slide | all |
 | `markers.lua` | `[[CAPTION:]]`, `[[STYLE:]]` and orientation markers. On slides the caption also becomes the alt text pandoc prints under a picture | PowerPoint, HTML, PDF |
 | `center-div.lua` | Centring divs become LaTeX centring | PDF |
 | `site.lua` | Admonitions or asides, figures as picture and caption, no repeated title | site pages |
 
-Word handles the markers itself after pandoc (`pandoc/docx_post.py`), because they become Word fields, sections and styles. `pandoc/docx_finish.py` merges the result into the template, fills the placeholders and applies the layout. `pandoc/pptx_finish.py` keeps embedded fonts, fills the version token and the footer, and warns about crowded slides. The finished file is moved to its output path, and the build report records it with its checksum.
+Word handles the markers itself after pandoc (`pandoc/docx_post.py`), because they become Word fields, sections and styles. `pandoc/docx_finish.py` merges the result into the template, fills the placeholders and applies the layout. It merges with docxcompose one list at a time, because docxcompose gives every list in one table the same numbering. Tables get the template's table style, centred at full width. An ordinary table with a header row is autofit, its header repeated and its rows kept whole. A `[[LISTTABLE]]` (found by its bookmark) or a table without a header keeps pandoc's column widths in a fixed layout. Its body rows may break across pages, and its pictures are made no wider than their cells. `pandoc/pptx_finish.py` keeps embedded fonts, fills the version token and the footer, and warns about crowded slides. The finished file is moved to its output path, and the build report records it with its checksum.
 
 ## Around the documents
 

docs/architecture/04_sites_and_imports.md

@@ -17,7 +17,7 @@ The site package keeps what all site generators share in `site/pages.py`, and wh
 
 ## Imports
 
-`docx_import.py` walks the Word body in order: paragraphs, tables and content controls. It keeps tracked insertions, drops deletions, and renders blocks to Markdown. `pptx_import.py` walks each slide's shapes in reading order, including those pandoc wraps in `mc:AlternateContent`. It writes the conventions the slides converter reads, so a deck makes a round trip to the same slides.
+`docx_import.py` walks the Word body in order: paragraphs, tables and content controls. It keeps tracked insertions, drops deletions, and renders blocks to Markdown. A table that a pipe table cannot hold (paragraphs or lists in a cell, merged cells, no header row) becomes a `[[LISTTABLE]]`. `pptx_import.py` walks each slide's shapes in reading order, including those pandoc wraps in `mc:AlternateContent`. It writes the conventions the slides converter reads, so a deck makes a round trip to the same slides.
 
 Both importers share picture conversion (Pillow for bitmaps, Inkscape in the container for EMF, WMF and SVG), escaping and the reading of document properties. Both count what they cannot carry over.
 

docs/guide/02_projects.md

@@ -193,7 +193,7 @@ The old commands `py-doc2docx`, `py-doc2slides` and `py-doclint` still work and
 `py-doc build --report build.json`, or `reports: {path: build.json}` in `py-doc.yml`, writes a JSON report of the build:
 
 - the py-doc version, the image tag and id, and the version of each tool used;
-- each document with its sources (words, sentences, paragraphs) and its outputs (size and SHA-256);
+- each document with its sources (words, sentences, paragraphs and `[[LISTTABLE]]` tables) and its outputs (size and SHA-256);
 - the icons each poster uses, with their pack and licence;
 - every diagram: type, file, line, and whether the cache served it;
 - template problems, and every warning with file and line.

docs/guide/03_writing.md

@@ -17,6 +17,7 @@ Pandoc reads the Markdown: its own flavour, with fenced divs, raw HTML, table ca
 | `[[CAPTION:Figure]] text` | A numbered caption for the picture, diagram or table that follows |
 | `[[LANDSCAPE]]`, `[[PORTRAIT]]` | A section of that orientation (Word), and picture widths in the other formats |
 | `[[STYLE:Name]] text` | The paragraph in that Word style |
+| `[[LISTTABLE]]` and `\| ... \|` rows | A table whose cells hold paragraphs, lists, merged cells and diagrams: see [Rich table cells](#rich-table-cells) |
 | `[[PAGEBREAK]]`, or `::: {.pagebreak}` and `:::` | A page break, or a new slide |
 | Fenced diagram blocks | A picture: see [Diagrams](#diagrams) |
 | Raw HTML | Kept in HTML and sites, dropped elsewhere by pandoc |
@@ -188,6 +189,42 @@ Word uses the `Div.note` to `Div.caution` paragraph styles. The built-in referen
 - **Styles**: `[[STYLE:Name]] text` applies a named Word paragraph style, and warns once if the template has none of that name. Other formats keep the text.
 - **Page breaks**: `[[PAGEBREAK]]` on its own line, `::: {.pagebreak}` followed by `:::`, or `page_break: true` on a source. In a deck it starts a new slide, and on a site it is left out.
 
+## Rich table cells
+
+A pipe table holds one line of text in each cell. For cells of several paragraphs, lists, merged cells or a diagram, put `[[LISTTABLE]]` on the line before the rows, and leave out the `|---|` line:
+
+````markdown
+[[LISTTABLE: header, widths=8,52,40]]
+| Step | What to do | Check |
+| 1 | [Prepare the node:]{.underline}<br>Fit the batteries.<br>Before you go to site:<br>- The firmware is version {{version}}<br>- The node is registered<br>  - with its own certificate | The light flashes green. |
+| 2 | Pair the node:<br>1. Hold the button for five seconds.<br>2. Choose the node in the app. | ^ |
+| 3 | The first reading goes this way:<br>[[BLOCK: reading]] | < |
+
+[[BLOCK: reading]]
+```dot
+digraph reading { rankdir=LR; Sensor -> Gateway -> Broker; }
+```
+[[ENDBLOCK]]
+````
+
+- **Rows**: each row is one line that starts and ends with `|`, straight after the marker. `\|` is a pipe in the text.
+- **Header**: `header` makes the first row the header row. Without it every row is body text.
+- **Widths**: `widths=8,52,40` gives the relative widths of the columns. Without it the columns are the same width.
+- **Paragraphs and lists**: `<br>` starts a new paragraph in a cell. A paragraph that starts with `- ` is a bullet and one that starts with `1. ` a numbered item, and items next to each other make one list. Two spaces before the `- ` or `1. ` put the item inside the one above.
+- **Merged cells**: a cell of only `<` joins the cell on its left, and a cell of only `^` the cell above. Merged cells must make a rectangle, and a body cell cannot join the header row. `\<` and `\^` are the characters themselves.
+- **Blocks**: a cell paragraph `[[BLOCK: name]]` is replaced by the lines between `[[BLOCK: name]]` and `[[ENDBLOCK]]` elsewhere in the same file. A block can hold a diagram, a picture, code or a callout. A picture or diagram in a cell has no caption of its own: caption the table. Plain pictures also work inline, as `![alt](picture.png)`.
+- **Formatting**: bold, italic, links, `[text]{.underline}` and `[text]{custom-style="Name"}` work as anywhere else. A `custom-style` is a character style of the Word template. Pandoc makes one with no formatting if the template has none of that name, so add the styles you use to the template.
+
+py-doc turns the table into a pandoc grid table after includes, conditions and variables, so rich tables work in included files and take `{{variables}}`. A marker in a code block is left alone. In Word the columns keep the widths given, the rows may break across pages, and a picture is made no wider than its cell. Only a header row is repeated on each page and printed as a header. HTML, PDF and sites show the same table. On a slide pandoc shows merged cells as separate cells, bullets as plain lines and no pictures.
+
+The build warns about a row with too many or too few cells, which is padded or cut. It warns about a merge that cannot be made, which is left empty, and a marker with no rows, which is left as written. It also warns about a block that is used but not defined, or defined but not used. A `[[BLOCK:]]` that is never closed with `[[ENDBLOCK]]` stops the build. `py-doc lint` and `py-doc check` report all of these with their line.
+
+Known limits:
+
+- A word wider than its column breaks the table, and the build says so. Widen the column or shorten the word.
+- Includes and conditions cannot go inside a cell, only around the whole table or inside a block.
+- A block belongs to its file: a table cannot use a block defined in another file.
+
 ## Includes, conditions and variables
 
 These lines are applied to each source before it is parsed. They are only recognised outside code, so a document can show the syntax in a fenced block or in backticks.

docs/guide/05_import.md

@@ -25,7 +25,8 @@ A file that is not a Word document or deck exits with status 2. The file's prope
 ## Word documents
 
 - Headings are moved up so that the first one is `#`. Title and Subtitle paragraphs become front matter.
-- Lists keep their nesting, tables become pipe tables (merged cells are flattened to empty cells), consecutive `Code` paragraphs become one fenced block, and `Quote` becomes a block quote.
+- Tables become pipe tables, or [rich tables](03_writing.md#rich-table-cells) (`[[LISTTABLE]]`) when a cell has several paragraphs or a list, cells are merged, or the table has no header row.
+- Lists keep their nesting, consecutive `Code` paragraphs become one fenced block, and `Quote` becomes a block quote.
 - `Note`, `Tip`, `Important`, `Warning` and `Caution` styles become `> [!NOTE]` callouts, and captions become `[[CAPTION:Figure]]` lines before their picture or table.
 - Tracked changes are accepted: insertions are kept and deletions dropped. The contents table is left out.
 - Pictures are saved as PNG in the order the document uses them. EMF, WMF and SVG go through Inkscape in the tool container. A picture that cannot be converted is named in a warning and shows as `(image not imported)`.

docs/guide/07_lint.md

@@ -73,7 +73,7 @@ lint:
 
 - **Structure rules** are project conventions as data. The rule types are `manifest-file-exists`, `regex-line`, `blank-line-between`, `path-exists`, `balanced-markers`, `requires-caption` and `cross-file-anchors`. Code blocks and inline code are never searched. Each rule has an `id`, a `severity` and a `message`, and `enabled: false` switches it off. A mistake in a rule is a configuration error that names the rule. `py-doc lint --print-example-rules` prints an example set.
 - **Cross-file anchors** work on the documents of `py-doc.yml`. All the files of a document share one set of heading anchors, as they do in pandoc. A repeated heading gets `-1`, `-2` and so on, and a `[x](#anchor)` link is checked against that set. Files pulled in with `[[INCLUDE:]]` count. A broken link and a repeated heading are reported with file and line.
-- **markdownlint** runs in the tool container (`markdownlint-cli2`), only when `lint.markdownlint` is set, and its findings appear with rules `MD001` and so on.
+- **markdownlint** runs in the tool container (`markdownlint-cli2`), only when `lint.markdownlint` is set, and its findings appear with rules `MD001` and so on. Its rule against raw HTML (`MD033`) reports the `<br>` of [rich tables](03_writing.md#rich-table-cells): allow it with `MD033: {allowed_elements: [br]}`.
 - **Business Process Model and Notation (BPMN) fences**: bpmn-text checks every ```` ```bpmnt ```` fence in the tool container. Its findings appear at their line of the Markdown file, with bpmn-text's own codes, such as `BT202` for a name that matches no element. `lint.rules` takes those codes like any other rule, such as `BT301: off`. The check runs only when the tool image is already built: without it, one warning says the fences were not checked, and `py-doc tools build` builds it. A broken fence still stops the build, with the same message.
 
 Without markdownlint or a `bpmnt` fence, lint needs no container.

docs/guide/14_writing_documents.md

@@ -96,6 +96,17 @@ Use a table to compare things or to look them up, with a caption on the line bef
 
 Keep tables narrow enough for the page: four or five columns in portrait. A wider table belongs in a landscape section (below) or split in two.
 
+When a cell needs more than one line, such as the steps of a procedure with the details of each, use a [rich table](03_writing.md#rich-table-cells). `<br>` starts a paragraph in a cell, `- ` a bullet and `1. ` a numbered item. A cell of `^` joins the cell above, and `<` the cell on its left:
+
+```markdown
+[[LISTTABLE: header, widths=8,62,30]]
+| Step | Action | Who |
+| 1 | Fit the controller.<br>Check before you start:<br>- The supply is isolated<br>- The enclosure is dry | Installer |
+| 2 | Connect the sensor cable to terminal S1. | ^ |
+```
+
+A diagram or picture can go in a cell too, as `[[BLOCK: name]]`, with the diagram between `[[BLOCK: name]]` and `[[ENDBLOCK]]` lines below the table.
+
 ## Pictures and diagrams
 
 Put a picture with its caption on the line before it: