From a8fab6b288633e4737b0a64991c254ff9ed275f7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:19:29 +0000 Subject: [PATCH 01/24] docs: Work identity, versions and Assets design (#60, #179) Design-only proposal for separating the logical Work, citeable Manifestations (Versions) and Assets (Files). Includes a current-state audit of the works model, ownership matrix, annotation/note/role/citation semantics, Asset lifecycle, deduplication, stable-ID and migration strategy, compatibility projection, downstream implications, rejected alternatives, implementation slices and open questions. No schema, migration, API or UI change. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/wiki/Domain-Model.md | 4 + docs/work-identity-model.md | 1351 +++++++++++++++++++++++++++++++++++ 2 files changed, 1355 insertions(+) create mode 100644 docs/work-identity-model.md diff --git a/docs/wiki/Domain-Model.md b/docs/wiki/Domain-Model.md index 5f408b98..c1e2f890 100644 --- a/docs/wiki/Domain-Model.md +++ b/docs/wiki/Domain-Model.md @@ -61,6 +61,10 @@ These records power the Research Graph and related detail/index views. Work source fields are not independent metadata. For video Works, source kind/provider/provider ID and URL semantics form an identity aggregate. The canonical reasoning and transition model are documented in [docs/work-source-identity.md](https://github.com/Fooftilly/PRKS/blob/master/docs/work-source-identity.md). +## Planned: Work identity, versions and files + +Today one Work row represents the intellectual work, one citeable publication, and at most one file. The proposed separation into Work → Version (manifestation) → File (asset) — covering editions, translations, revisions, duplicate detection and citation targets — is a design under review, not current behavior. See [docs/work-identity-model.md](https://github.com/Fooftilly/PRKS/blob/master/docs/work-identity-model.md). + ## IDs and offline creation Durable local-first creation requires IDs that can safely exist before a server round-trip. Domain-specific synchronization modules and `backend/entity_ids.py` handle the relevant boundaries. diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md new file mode 100644 index 00000000..fd52d8ab --- /dev/null +++ b/docs/work-identity-model.md @@ -0,0 +1,1351 @@ +# Work identity, editions, versions and Assets: design (#60) + +**Status: design proposal for review. Nothing here is implemented.** This +document does not add schema, migrations, API behavior or UI. It is the design +gate that #60 requires before large schema changes, and it follows the +architecture direction in #179. Once approved, the implementation slices in §17 +become separate focused issues and PRs. + +It follows the shape of [work-source-identity.md](work-source-identity.md): +audit the code first, then recommend. Every claim about current behavior +below was checked against `master` at `0e19b86` (schema version 16), not taken +from issue text. + +Contents: + +1. [Current-state audit](#1-current-state-audit) +2. [Problems with the overloaded Work row](#2-problems-with-the-overloaded-work-row) +3. [Terminology and proposed entities](#3-terminology-and-proposed-entities) +4. [Relationships and cardinality](#4-relationships-and-cardinality) +5. [Ownership matrix](#5-ownership-matrix) +6. [Annotations and notes](#6-annotations-and-notes) +7. [People and roles](#7-people-and-roles) +8. [Citations](#8-citations) +9. [Asset lifecycle and storage](#9-asset-lifecycle-and-storage) +10. [Identity and deduplication](#10-identity-and-deduplication) +11. [Stable IDs](#11-stable-ids) +12. [Migration and backfill](#12-migration-and-backfill) +13. [Compatibility and API projection](#13-compatibility-and-api-projection) +14. [Backup/restore and offline/durable operations](#14-backuprestore-and-offlinedurable-operations) +15. [Downstream roadmap implications](#15-downstream-roadmap-implications) +16. [Rejected alternatives and tradeoffs](#16-rejected-alternatives-and-tradeoffs) +17. [Proposed implementation slices](#17-proposed-implementation-slices) +18. [Open questions for the maintainer](#18-open-questions-for-the-maintainer) + +--- + +## 1. Current-state audit + +### 1.1 The `works` row + +One `works` row (`backend/db_schema.sql`) currently holds every concept below. +The table groups its 34 columns by what each one actually describes, which is +not the same as how the schema groups them. + +| What it describes | Columns | +| --- | --- | +| Identity | `id` (`W-` + 8 or 32 hex, see `backend/entity_ids.py`) | +| Display title | `title` (NOT NULL; also FTS-indexed) | +| User workflow | `status` (Progress groups), `last_opened_at` (Recent) | +| Bibliographic record | `doc_type`, `year`, `published_date`, `publisher`, `location`, `edition`, `journal`, `volume`, `issue`, `pages`, `isbn`, `doi`, `urldate`, `abstract` | +| Free-text credit | `author_text` (search, cards, video channel; loses to linked Author roles) | +| Source identity | `source_kind`, `file_path`, `source_url`, `provider`, `provider_id`, `source_mime` | +| Presentation | `thumb_url`, `thumb_page`, `hide_pdf_link_annotations` | +| Research content | `text_content` (the Research Note; the schema comment says "Markdown notes or OCR text", but it is only the note) | +| Private content | `private_notes` | +| Annotation materialization | `canonical_annotation_set_revision`, `materialized_pdf_annotation_revision` | +| Bookkeeping | `created_at`, `updated_at` | + +`works_fts` (FTS5, external content, kept current by triggers `works_ai`, +`works_ad`, `works_au`) indexes `title`, `abstract`, `text_content` and +`author_text`. + +### 1.2 Everything that holds a Work ID + +A redesign has to account for every one of these. The list is the reason §11 +keeps Work IDs unchanged. + +| Holder | Where | Kind of reference | +| --- | --- | --- | +| `roles.work_id` | main DB, FK CASCADE | Person↔Work credit | +| `annotations.work_id` | main DB, FK CASCADE | PDF annotation owner | +| `work_tags.work_id` | main DB, FK CASCADE | Tag attachment | +| `folder_files.work_id` | main DB, FK CASCADE | Folder membership (one folder per Work, `work-folder` scalar) | +| `playlist_items.work_id` | main DB, FK CASCADE, UNIQUE | One playlist per Work | +| `argument_sources.work_id` (+ `pages`) | main DB, FK CASCADE | Research Argument cites a Work, optionally with a page pinpoint | +| `processing_files.imported_work_id` | main DB, FK SET NULL | Import provenance | +| `sync_entity_revisions.scope_id` | main DB | `work-field/[work, field]`, `work-source/work`, `work-tag`, `work-folder`, `work-playlist`, `work-person-role`, `work-research-note`, `work-private-note`, `pdf-annotation/[work, annotation]` | +| `sync_operations.entity_id` | main DB | Durable-operation ledger (`entity_type` `work`) | +| `concept_mentions`, `argument_mentions`, `work_note_state` | `prks_research_index.db` (derived) | Research Note mentions | +| `work_text_index` | `prks_text_index.db` (derived) | PDF text, keyed by `work_id` + `source_ref_hash` of `file_path` | +| `thumbs/` filenames | derived | `prks_thumb_cache_stem(work_id, page)` | +| Research Note text | canonical text | `[[W-…]]` or `[[Title]]` wiki links (`resolve_wiki_links`) | +| URLs / workspace tabs | browser | `#/works/W-…`, persisted workspace layout | +| `prks.pdf.lastPage.` | browser `localStorage` | Last-read PDF page, per device | +| `prks-local-v1` (IndexedDB) | browser, durable | Pending operations carrying `work_id` and base revisions | +| Offline cache entities | browser, disposable | `work:*`, `work-annotations`, `works-browse:index`, `recent:index`, `recently-added:index`, summaries | +| Backups | archive | The main DB carries all of the above | + +These do **not** hold a Work ID: the managed PDF file name +(`_<8hex>_.pdf`, from `mint_managed_pdf_filename`), +`pending_pdf_cleanup` (keyed by basename), and the service-worker PDF cache +`prks-pdf-v1` (keyed by `/api/pdfs/`). + +### 1.3 How the model is used, by subsystem + +**Schema and migrations.** `backend/db_migrations.py` runs ordered migrations +up to `LATEST_SCHEMA_VERSION = 16`, validates the final shape +(`validate_current_schema`, `_assert_known_table_shapes`), and never touches +the filesystem. v15 added the two materialization revisions to `works`. v16 +added `pending_pdf_cleanup`. + +**`backend/db_manager.py`.** `add_work()` writes one row with every column and +canonicalizes the source through `_canonical_new_source` and +`effective_source_kind`. `get_work()` is a pure read (the stamp on +`last_opened_at` was moved into `mark_work_opened`); it attaches roles, tags, +annotations, and the one playlist and one folder. `update_work_metadata()` is +the PATCH path. `generate_bibtex()` builds a single entry from the row plus +roles. The summary projections (`_PRKS_WORK_SUMMARY_COLUMNS`, +`_prks_work_browse_select`, `finish_work_summary_rows`) read `works` columns +directly and `stat()` the managed PDF to produce `file_size_bytes`. + +**`*_sync.py`.** Each durable family declares a conflict unit keyed by +`work_id`: + +- `work_metadata_sync`: one revision **per field** (`work-field/[work, field]`), + covering `SYNCED_FIELDS`: status, title, source_url (provenance only), + doc_type, thumb_page, author_text, abstract, year, published_date, edition, + journal, volume, issue, pages, isbn, doi, publisher, location. +- `work_source_sync`: `SET_WORK_SOURCE` is **one aggregate** for video identity + (`source_kind` + `provider` + `provider_id` + `source_url`), because + `provider_id` outranks the URL in the viewer. +- `work_role_sync`: `work-person-role/[work, person, role]`; the role types are + `Author, Editor, Reviewer, Mentioned, Translator, Introduction, Foreword, + Afterword`. +- `work_note_sync`: Research Note (`text_content`) and Private Note + (`private_notes`) are separate whole-document aggregates. +- `work_tag_sync`, `folder_sync` (`SET_WORK_FOLDER`), `playlist_sync` + (`SET_WORK_PLAYLIST`), `work_open_sync` (a max-register on + `last_opened_at`), `work_lifecycle_sync` (`CREATE_WORK` for video only, and + `DELETE_WORK`). +- `pdf_annotation_sync`: one revisioned aggregate per annotation, scope + `pdf-annotation/[work, annotation]`. + +**HTTP.** `backend/server.py` handles `/api/works` (create, with upload or +adoption of an existing `/api/pdfs/`), `/api/works/:id` (GET/PATCH/ +DELETE), `/api/works/:id/pdf` (replace bytes, used by annotation +materialization), `/annotations*`, `*-state` endpoints for each sync family, +`/opened`, `/thumbnail`, `/api/bibtex/:id`, `/api/pdfs/:name`, +`/api/processing-files/*`, `/api/works/bulk`, and the backup endpoints. Only +Positions has typed DTOs so far (`backend/api_contract/`). + +**Processing Files / import.** `processing_files` is a staging copy of most +`works` bibliographic columns plus draft roles and tags. `import_processing_file` +checks the `%PDF-` header, streams the file into managed storage, linearizes +it, creates one Work with `add_work()`, and records `imported_work_id`. It +does **no duplicate check**. + +**Managed PDFs.** Bytes live in `pdfs/` under a minted, unique basename. +`works.file_path = "/api/pdfs/"` is the only link from the DB to the +bytes. Several Works may point at the same basename (creation can adopt an +existing managed name), which is why `work_pdf_replace` does +**copy-on-write** before it overwrites shared bytes. Deletion commits the row +and writes a `pending_pdf_cleanup` claim in the same transaction. Every write +follows `backend/fs_durability.py`. + +**Annotation bytes vs metadata.** The `annotations` table is canonical. The +managed PDF bytes are a **materialized** copy that embeds the same markup: +`POST /api/works/:id/pdf` **overwrites the managed file in place** with the +annotated export. `canonical_annotation_set_revision` > `materialized_…` +means the bytes are stale. Linearization also rewrites bytes in place. So +**PRKS does not keep the original bytes it ingested**, and the managed file's +hash changes every time annotations are saved. + +**Work source.** See [work-source-identity.md](work-source-identity.md). The +proposed invariants there (`video` ⇒ provider identity; `pdf` ⇒ identity is +`file_path`, and `source_url` is only provenance) hold across current data. + +**Bibliographic metadata.** It is flat on `works`. `author_text` is a fallback +credit. Publisher normalization (`publishers`, `publisher_aliases`) resolves +spellings for search only. There is no `language`, `subtitle`, arXiv ID, +generic identifier table, or provenance for any value. + +**People, roles and credit names.** `roles(person_id, work_id, role_type, +order_index, credit_name)`, with UNIQUE `(person_id, work_id, role_type)`. +`credit_name` is how that person is credited **on this Work**. `Mentioned` +roles are also written automatically by `_sync_mentioned_roles()` from +`[[Name]]` markup found **inside PDF bytes** on replace, so they come from an +asset. + +**Tags and folders.** Both attach to the Work. A Work is in at most one folder. +Folder and Tag vocabulary are separate entities. + +**Research Notes.** `works.text_content`. Concept and Argument mentions are +indexed per Work in the derived research index. Notes can contain +`[[pdf:]]` markers (`prksReplacePdfAnnotationWikiMarkers`) that +jump to a PDF annotation, and `[[W-…]]`/`[[Title]]` Work links. + +**Private Notes.** `works.private_notes`. Reminders are capped at 8,000 +characters in the UI and 64 KiB on the wire. They are not indexed. + +**Reading and open state.** `status` is the user's reading progress (Progress +page). `last_opened_at` is a server-side max-register (Recent). The last-read +PDF page lives only in the browser (`localStorage prks.pdf.lastPage.`, +behind the "remember last page" setting). + +**Playlists.** These are ordered Works, mainly videos. The UNIQUE index on +`playlist_items(work_id)` limits each Work to one playlist. + +**Citation and export.** `generate_bibtex()` is the only exporter. The cite +key is **computed, never stored**: first Author family name + year. Authors, +editors, translators, and introduction/foreword/afterword come from roles. The +entry type is `doc_type`, or, when that is empty, guessed from journal or +publisher. `url` is `source_url`. `urldate` is taken from **`updated_at`**, not +from the `urldate` column. A server-wide profile selects which fields are +exported. + +**Duplicates and source identity.** Nothing detects duplicates today. Video +identity is canonicalized, so two spellings of one YouTube ID are recognized +as the same **within one Work**, but not across Works. + +**Offline and durable operation.** Durable intent in `prks-local-v1` is kept +apart from the disposable acknowledged cache. Server revisions are +per-scope. PDF binary ingestion and replacement require a connection. +Annotation metadata can be edited offline when the PDF bytes are cached +(`docs/local-first-sync.md`, "PDF annotations (V2 local-first boundary)"). + +**Backup and restore.** A backup is the main DB plus `files/pdfs/*` plus +person images, with a manifest (`FORMAT_VERSION = 1`, `db_schema_version`). +Restore refuses a backup with a newer schema. `audit_managed_pdfs` counts +referenced-but-missing PDFs by scanning `works.file_path`. Thumbnails and both +indexes are derived and are rebuilt, not backed up. + +**Frontend.** `prksInferWorkSourceKind` picks the viewer. Work cards, browse +lists, Folder/Person/Playlist summaries, and the metadata/role/source/notes/ +tag editors all consume the flat Work shape. Every `work-*-state.js` module +mirrors one backend sync family. + +### 1.4 Current invariants to preserve + +These hold today. The replacement design must keep each one or say explicitly +why it changes. + +1. **I1: Work IDs are immutable.** They are never rewritten. Legacy 8-hex IDs + remain valid forever. +2. **I2:** `get_work()` is a pure read. Opening a Work is an explicit + max-register operation. +3. **I3: Source identity is an aggregate.** For video, `provider_id` outranks + `source_url`. Identity is never split across independent field scopes. +4. **I4: The PDF text index is derived.** Any change to a Work's managed PDF + identity resynchronizes it or relies on reconciliation. +5. **I5: Managed bytes are never lost to a crash.** A basename is minted + unique, published durably, never trusted as a path + (`safe_pdf_path_under_dir`), copy-on-written when shared, and claimed in + `pending_pdf_cleanup` inside the delete transaction. +6. **I6: Annotation metadata is canonical.** The bytes are a materialization. + Semantic ACK never waits on bytes. Canonical > materialized means "rebuild", + never a user-facing binary conflict. +7. **I7: Conflict units are deliberately chosen.** Metadata conflicts per field, + notes per whole document, roles per (person, role), annotations per + annotation. Merging them into a coarser unit is a regression. +8. **I8: One folder and one playlist per Work.** +9. **I9: One role row per (person, work, role_type).** +10. **I10: Migrations touch SQLite only.** Managed files are never modified by a + migration. +11. **I11: Backups are self-contained.** Every referenced managed file travels + with the DB that references it. +12. **I12: Durable intent is not acknowledged state.** An offline edit never + mutates the acknowledged cache before ACK. +13. **I13: PDF ingestion is online-only.** + +--- + +## 2. Problems with the overloaded Work row + +1. **One row means four different things**: the intellectual work ("Plato's + *Republic*"), one citeable publication ("Grube/Reeve, Hackett 1992"), one + file ("this scan"), and the user's relationship to it (status, notes, + tags). Adding a second translation today means either creating an + unrelated Work, which scatters notes, tags and research links, or + overwriting the first translation's citation metadata. +2. **`title` means two things.** It is both the name the user files the work + under and the citation title. A Serbian edition's title "Država" cannot + coexist with "Republic". +3. **`source_url` means three things**: video identity (§3 of the source + report), PDF download provenance, and the BibTeX `url`. The current UI + already exposes only the safe meaning. +4. **`file_path` means three things**: storage locator, file identity, and + "this Work is a PDF". It is the only record of which bytes belong to the + Work, which is why deletion needs `pending_pdf_cleanup`. +5. **Annotation state is keyed by Work but depends on one file's pagination.** + `annotations.page_index`, `geometry_json` and the two materialization + revisions are only meaningful for one specific PDF. A second PDF on the + same Work would make every coordinate ambiguous. +6. **The managed file is both the original and the annotated export.** + Materialization and linearization overwrite it in place. PRKS cannot say + what it originally ingested, and a content hash of the current file is not + a stable duplicate key. +7. **Roles mix intellectual and edition credits.** Author (the work) and + Translator/Editor/Foreword (one edition) sit in one table, and + `credit_name` ("Platon" vs "Plato") is really a per-edition fact. +8. **`argument_sources.pages` is a pinpoint with no edition.** "Republic, + p. 45" means nothing once two paginations exist. +9. **`thumb_page`, `hide_pdf_link_annotations` and the browser's last-page key** + describe one file and are stored against the Work. +10. **`updated_at` is also the BibTeX `urldate`**, so any tag or status edit + changes a citation's access date. +11. **`year` and `published_date` overlap.** BibTeX takes `year`, then falls + back to a prefix of `published_date`. +12. **`edition` is free text** ("2nd rev. ed."), not version identity. +13. **Nothing detects duplicates.** Import creates a new Work even when the + same bytes already exist. + +--- + +## 3. Terminology and proposed entities + +Three core entities, plus a few relationship entities. The internal names are +precise. The UI names are chosen so that users with one paper and one PDF never +see the internal ones (§13.4). + +### 3.1 Work (logical Work): internal `work`, UI "Work" + +This is the intellectual identity the user reasons about: what they file, +tag, annotate in notes, argue with, and read about. It is the **existing +`works` entity with its existing ID** (§11). A Work always has at least one +Manifestation, and exactly one of them is **primary**. + +### 3.2 Manifestation: internal `manifestation`, UI "Version" + +This is **one citeable form of a Work**: a specific edition, revision, +translation, or publication form with its own bibliographic record. +Examples: arXiv v1; arXiv v2; the accepted manuscript; the journal version; +the 1968 Bloom translation; the Serbian translation; the Greek text in the +OCT; a web page as captured; a YouTube video. It owns citation metadata and +edition-scoped credits. + +A Manifestation has a `kind`, which is a descriptive classification and not +a separate entity type: + +``` +kind ∈ { unspecified, preprint, accepted_manuscript, published, edition, + translation, reprint, web_page, video, other } +``` + +Two orthogonal fields cover translation identity: `language` (BCP 47) and the +relations in §3.4. `kind` never decides behavior. It labels the version in the +UI and gives the citation layer a hint. + +**Why not "Edition" or "Version" internally?** "Edition" is wrong for arXiv +revisions and for translations. "Version" is the right *user* word but is +ambiguous in code, where it collides with schema version, revision and asset +replacement. "Manifestation" is the library-science term for "a specific +embodiment someone can cite", and it is used here **without** adopting the +rest of FRBR/LRM. See §16 for the rejected Expression layer. + +### 3.3 Asset: internal `asset`, UI "File" + +This is **one concrete content object that belongs to one Manifestation**: +a managed PDF, a web snapshot, a future attachment, or an external +provider-hosted stream (a YouTube video). It owns the storage locator, media +type, byte size, content hashes, provenance of the bytes, and every piece of +state defined by that content's own coordinates: PDF annotations, page +positions and page thumbnails. + +`asset.kind ∈ { managed_file, external_stream, external_link }`: + +- `managed_file`: bytes stored by PRKS under managed storage (PDFs today, + web snapshots and attachments later). +- `external_stream`: identified by a provider and a provider ID (YouTube). + This is where today's `SET_WORK_SOURCE` aggregate moves. +- `external_link`: a URL PRKS does not store (for example "the publisher's + page"). This is optional and may never be needed. §18 Q8 asks. + +### 3.4 Relationship and supporting entities + +| Entity | Purpose | +| --- | --- | +| `manifestation_relations(from_id, to_id, relation)` | Typed links between Manifestations of the **same** Work: `revision_of`, `published_version_of`, `new_edition_of`, `translation_of`, `reprint_of`. These carry "arXiv v2 revises v1", "journal version of the accepted manuscript", and "Serbian translation of the Greek text". They carry no page correspondence (#61 owns assisted correspondence). | +| `manifestation_identifiers(manifestation_id, scheme, value, normalized)` | DOI, ISBN, arXiv ID (versioned and versionless), PMID, and similar. `works.doi`/`isbn` backfill into these later. This is the entity duplicate detection keys on. | +| `roles` gaining a nullable `manifestation_id` | Edition-scoped credit (§7). | +| `argument_sources` gaining a nullable `manifestation_id` | A pinpoint citation is pinned to a pagination (§8.5). | +| `work_lifecycle(work_id, state, target_work_id)` | Redirects left by merges, modeled exactly like `sync_tag_lifecycle` (§10.4). | +| `duplicate_decisions` | Durable record of "not a duplicate" answers, so a declined suggestion does not return (§10.5). | +| Metadata provenance | A per-field record of detected, retrieved or user-edited origin on Manifestation fields (#42). This design only reserves the ownership: provenance belongs to the Manifestation field it describes, never to the Work. | + +--- + +## 4. Relationships and cardinality + +``` + ┌─────────────────────────────┐ + tags, folder, playlist ─┤ Work ├─ Research Note, Private Note + status, last_opened │ W-… (existing ID, kept) │ Research links (Concepts, + Work-level roles │ canonical title, abstract │ Arguments via mentions) + (Author, Mentioned) └──────┬───────────────▲──────┘ + │ 1..* │ primary_manifestation_id (exactly 1) + ▼ │ + ┌──────────────────────────────┴──┐ manifestation_relations + edition roles ───┤ Manifestation ├──── (revision_of, translation_of, + (Translator, │ MF-… kind, language, citation │ published_version_of, …) + Editor, Foreword)│ metadata, identifiers │ + └──────┬───────────────▲──────────┘ argument_sources.pages + │ 0..* │ primary_asset_id (0 or 1) pinned here + ▼ │ + ┌──────────────────────┴──────────┐ + annotations ─────┤ Asset ├──── managed bytes /pdfs/ + page position │ AS-… kind, locator, media │ or provider identity + page thumbnails │ type, size, hashes, state │ derived: text index, thumbs + materialization └─────────────────────────────────┘ +``` + +| Relationship | Cardinality | Notes | +| --- | --- | --- | +| Work → Manifestation | 1 → 1..* | Never zero. A notes-only or metadata-only Work still has one empty primary Manifestation, so projections and citations always resolve. | +| Work → primary Manifestation | exactly 1 | `works.primary_manifestation_id`, NOT NULL after backfill, and must belong to that Work. | +| Manifestation → Asset | 1 → 0..* | Zero means "metadata, no file". | +| Manifestation → primary Asset | 0 or 1 | `manifestations.primary_asset_id`. It is required to be non-NULL when the Manifestation has any active Asset. | +| Asset → Manifestation | exactly 1 | An Asset is never shared by two Manifestations. Shared **bytes** are a storage fact handled below the Asset (§9.3). | +| Annotation → Asset | exactly 1 | §6. | +| Manifestation relations | same-Work only | Cross-Work links are research relationships, not version relationships. | + +### Worked cases + +| Case | Shape | +| --- | --- | +| Ordinary single-PDF paper | 1 Work → 1 Manifestation → 1 Asset. This is every existing PDF Work after backfill. | +| arXiv v1 → v2 | 1 Work → MF(v1, preprint) + MF(v2, preprint, `revision_of` v1). Each has its own Asset. Annotations stay with their own PDF. | +| Preprint → accepted manuscript → journal | 1 Work → 3 MFs chained by `published_version_of`. The journal MF is usually primary and the citation target. | +| First → revised edition | 1 Work → 2 MFs (`new_edition_of`), each with its own ISBN, date and pages. | +| Original → translations | 1 Work → MF(grc, published) + MF(en, translation, `translation_of` grc) + MF(sr, translation, `translation_of` grc). The Translator role is on each translation MF. | +| Two English translations | Two `translation` MFs with `language = en`, each with its own translator. | +| Several scans of one exact edition | 1 MF → several Assets. One of them is primary. | +| Publisher PDF + locally annotated copy | 1 MF → Asset(publisher original) + Asset(annotated export, `derived_from_asset_id` = original). Today these are the same file (§9.4). | +| Future Web Work | 1 Work → MF(web_page, url, site, accessed date) → Asset(sanitized HTML snapshot) and optionally a readable-text Asset. A re-capture is a new Asset in the same MF. | +| Metadata but no file | 1 Work → 1 MF → 0 Assets. | +| Video | 1 Work → MF(video) → Asset(external_stream: youtube, provider_id). | +| Exact duplicate file | Two Assets with the same `ingest_sha256`. This is detected (§10.2) and never merged silently. | +| Probable bibliographic duplicate | Two Works whose MFs share a normalized DOI/ISBN or have similar title/author/year. They become a **candidate**, and only the user decides (§10.3). | +| Genuinely separate works | Two Works. A declined candidate is recorded in `duplicate_decisions`. | + +--- + +## 5. Ownership matrix + +Legend: **W** = Work, **M** = Manifestation, **A** = Asset, **J** = +relationship/join entity, **D** = dedicated domain object. "Backfill" +describes where today's value goes; §12 covers how. + +| Concept (current column/table) | Owner | Backfill of current value | Rationale / tradeoff | +| --- | --- | --- | --- | +| Canonical title (`title`) | **W** | stays on `works.title` | The name the user files the work under. | +| Publication title | **M** (`manifestations.title`, NULL = inherit Work title) | NULL (inherit) | For one version, editing the title changes both, as today. A translation sets its own. Inheritance avoids copying every title. | +| Subtitle | **M** (new `subtitle`) | NULL | Differs by edition. There is no current column. | +| Document type (`doc_type`) | **M** | copied | This is a BibTeX/CSL entry type: a preprint is `misc`/`online`, a journal version is `article`. The Types page derives from the primary M. **Tradeoff:** the Types page may later want a Work-level "kind of intellectual work" (monograph, paper, lecture). It is not needed now. | +| Year / date (`year`, `published_date`) | **M** | copied | Publication date of this form. An optional Work-level `original_date` (for example "c. 375 BC") is a future addition. | +| Edition statement (`edition`) | **M** | copied | Free text on the version it describes. | +| Language | **M** (new `language`) | NULL | Translation identity is the M's language plus a `translation_of` relation. A Work-level `original_language` is optional and future. | +| Translation identity | **J** (`manifestation_relations.translation_of`) + M.`language` + M-scoped Translator role | none | §3.4. There is no Expression entity (§16). | +| Publisher, location (`publisher`, `location`) | **M** | copied | Per edition. | +| Journal, volume, issue, pages (`journal`, `volume`, `issue`, `pages`) | **M** | copied | Per publication form. | +| DOI (`doi`) | **M** (+ `manifestation_identifiers`) | copied | DOIs identify a publication form. A preprint and the journal version have different DOIs. | +| ISBN (`isbn`) | **M** (+ identifiers) | copied | Per edition and binding. | +| URL for citation (`source_url` on a non-video Work) | **M** (`url`) | copied for non-video Works | This is the BibTeX `url` today. **Tradeoff:** it is also "where I downloaded the PDF from" (Asset provenance). Keeping one value on M preserves the current single field. Asset `origin_url` is populated going forward by capture/import, not by the backfill. | +| URL as video identity (`source_url` + `provider*` on a video Work) | **A** (external_stream aggregate) | moved into the Asset | Keeps I3: `SET_WORK_SOURCE` becomes an Asset-scoped aggregate. The M's citation `url` is derived from it for `@online`. | +| Access date (`urldate`) | **M** | copied | This is a citation fact about the version. The current BibTeX use of `updated_at` is a quirk the projection preserves until #41 decides (§8.6). | +| Abstract (`abstract`) | **W**, with optional **M** override (NULL = inherit) | stays on `works.abstract` | **Genuinely ambiguous.** Usually one summary of the intellectual work, but preprint and published abstracts can differ, and translations are in another language. Keeping it on W avoids an FTS rewrite (`works_fts` indexes it). The override covers the cases where it differs. | +| Free-text author (`author_text`) | **W** | stays | Search/card/channel fallback for the work's creators. Edition credits come from roles. | +| People / roles (`roles`) | **J**, scoped to W (NULL `manifestation_id`) or M | by role type, §7.3 | §7. | +| Credit names (`roles.credit_name`) | **J** row. For a W-level role, M can override. | stays on the row | "Platon" on the Serbian edition and "Plato" on the English one. §7.2. | +| Tags (`work_tags`) | **W** | unchanged | Topical classification of the intellectual work. **Tradeoff:** a user tag like "needs-OCR" is really about a file. That is acceptable. File-level status belongs to Asset state, not to Tags. | +| Folders (`folder_files`) | **W** | unchanged | One folder per Work (I8). | +| Playlists (`playlist_items`) | **W** | unchanged | Ordered viewing sequence. One per Work (I8). | +| Research Notes (`text_content`) | **W** | unchanged | §6.4. | +| Private Notes (`private_notes`) | **W** | unchanged | §6.5, with an open question. | +| Annotations / comments (`annotations`) | **A** | `annotations.asset_id` = the Work's backfilled Asset | §6.1. Coordinates belong to one file. | +| PDF page references in notes (`[[pdf:]]`) | resolve through the annotation's **A** | nothing to move | Annotation IDs are globally unique, so existing markers keep resolving. The viewer opens the owning Asset. | +| Pinpoint pages (`argument_sources.pages`) | **J**, pinned to **M** | `manifestation_id` set to the default M when `pages` is non-empty | §8.5. | +| Reading progress (`status`) | **W** | unchanged | "Have I read this work?" **Tradeoff:** someone who has read translation A but not the original may want per-version status. Deferred, §18 Q6. | +| Last-read page (`localStorage prks.pdf.lastPage.`) | **A** (per device) | client falls back from the Asset key to the legacy Work key for the default Asset | Page N means nothing in another file. #61 needs an independent location per version. | +| Open/recent (`last_opened_at`) | **W** (max-register) | unchanged | Recent lists Works. An Asset-level `last_opened_at` may be added so "open" reopens the last-used version (§18 Q6). | +| Thumbnail page (`thumb_page`) | **A** | copied to the default Asset | It is a page of that file. The Work card uses the primary Asset. | +| Remote thumbnail (`thumb_url`) | **A** | copied to the default Asset | Presentation of the video asset. | +| Hide link annotations (`hide_pdf_link_annotations`) | **W** (viewer preference) | unchanged | A display preference for the sidebar list that applies to whichever PDF is shown. Low stakes. | +| Citation / cite key | **M** (a stored `cite_key` is optional future) | none (it is computed today) | §8. | +| Source / provenance | **A** for bytes (`origin`, `origin_url`, `imported_from_processing_file_id`). **M** field provenance for metadata. | `origin = 'legacy'` | Bytes and metadata have different provenance. | +| File path (`file_path`) | **A** (`storage_locator`) | copied | The Work projection keeps `file_path` (§13). | +| MIME / media type (`source_mime`) | **A** (`media_type`) | `application/pdf` for managed PDFs, NULL otherwise. `source_mime` is never populated today. | | +| Byte size | **A** (`byte_size`) | filled by the fingerprint pass (§12.4) | It replaces the per-row `stat()` in `finish_work_summary_rows` once trusted. | +| Content hash | **A** (`ingest_sha256`, `content_sha256`) | NULL until the fingerprint pass | §9.4 and §10.2. | +| Derived text / OCR / thumbnails | **A**, derived | the rebuilt index is keyed by `asset_id` | They stay derived and are never backed up (I4). | +| Annotation materialization revisions | **A** | copied from `works` | They describe one file's bytes. | +| Source kind (`source_kind`) | **A** (`kind` + `media_type`). The Work projection derives it. | derived | `prksInferWorkSourceKind` keeps working on the projection. | +| `created_at` / `updated_at` | each entity has its own | copied | `works.updated_at` stays the Work's. | +| Revision state (`sync_entity_revisions`) | the entity that owns the conflict unit | §14.2 | Scopes move with ownership, and revision counters are carried over. | +| Mentioned roles auto-extracted from PDF bytes | **J** at W scope, with provenance from **A** | unchanged | §7.4. | + +--- + +## 6. Annotations and notes + +### 6.1 Annotations belong to an Asset + +Every PDF annotation has `page_index` and geometry measured in one file's +coordinate space. Those only make sense against the exact bytes they were drawn +on (strictly, against the original bytes plus the materializations PRKS +itself produced). They therefore attach to the **Asset**, never to the +Manifestation or the Work: + +- `annotations.asset_id` NOT NULL (after backfill), FK to `assets`. +- `annotations.work_id` is kept during the transition as a denormalized value + equal to the Asset's Work, so existing queries and the + `pdf-annotation/[work, annotation]` scope keep working (§14.2). +- `canonical_annotation_set_revision` and `materialized_pdf_annotation_revision` + move to the Asset. + +The viewer shows the annotations of **the Asset it is displaying, and only +those**. When #61 puts two versions side by side, each pane shows its own set. + +### 6.2 What happens to annotations in each scenario + +The rule: **page-specific state never moves to a different pagination +implicitly.** Carrying annotations over is allowed only when PRKS can prove the +coordinate space is identical. Anything else is an explicit, reviewed action. + +| Scenario | What happens | Why | +| --- | --- | --- | +| Replace the PDF with an **exact byte duplicate** of its original | No new Asset. The upload is recognized as the same content (§10.2) and nothing changes. | The coordinate space is identical. There is nothing to migrate. | +| Replace with a **newer revision** (arXiv v2) | A **new Manifestation** (`revision_of`) with a new Asset. The old Asset and all its annotations remain, still readable. v2 may become primary. | The pagination changed. Moving coordinates would silently point highlights at the wrong text. | +| Preprint → publisher version | The same: a new M (`published_version_of`) and a new A. The preprint annotations stay on the preprint. | The same reason. The PDFs are completely different. | +| **Another scan** of the same edition | A new Asset in the **same** Manifestation. Annotations stay on the scan they were drawn on. | Same edition, but the scans differ in page offsets, cropping and rotation. | +| **Merging** duplicate Works | Assets move **whole** with their Manifestation, and their annotations go with them unchanged. | An annotation's Asset never changes, so its coordinates stay valid. | +| Merging two Assets that are **exact duplicates** (same `ingest_sha256`) | The user chooses: keep one Asset and **union** the annotation sets (the coordinates are provably identical), or keep both. | Content identity proves the coordinates are compatible. It is still never automatic. | +| Replacing a file **in place** (a "correct scan" in the same M) | A new Asset that `supersedes` the old one. The old Asset's annotations stay on it, and the UI offers "copy annotations to the new file" only as an assisted action (below). | | + +**Assisted annotation transfer (future, #61):** a copy is created with +text-anchor matching (for example a quoted passage plus prefix/suffix +captured from the source text layer) and a review list of matched, uncertain and unmatched items. +It is never a move, so the originals remain on their Asset. It is never offered +for translations. + +### 6.3 Materialized bytes and "the annotated copy" + +Today the managed file **is** the annotated export (§1.3). The design keeps +that behavior for the default Asset: materialization still rewrites that +Asset's bytes (copy-on-write when shared). §9.4 explains why `ingest_sha256` +exists, and §18 Q3 asks whether PRKS should start keeping the pristine +original as a separate Asset. Either way, the annotated export and the +original are different **bytes** of one Asset lineage, and their annotations +are the same canonical metadata rows. + +### 6.4 Research Notes belong to the Work + +The Research Note is the user's thinking about the intellectual work. Concept +and Argument mentions, the Research Graph and backlinks all reason about Works, +and the note already survives any change of file. It stays on `works` +(`text_content`, scope `work-research-note/`) with **no change**. + +Version-specific passages inside a note are expressed as references that carry +their own target: + +- `[[pdf:]]` already resolves through the annotation to its + Asset. With several Assets, the link opens that Asset. It never opens "the + current file". +- Any future page-pinpoint syntax (for example `[[page:MF-…#p45]]`) **must name + a Manifestation or Asset**. A bare page number inside a Work-level note is + ambiguous and must not be introduced. + +Per-version notes ("this translation mistranslates §4") are not supported by +this design. §18 Q5 asks whether they are needed. If they are, the answer is a +separate, small Manifestation-level note field, not a change of scope for the +Research Note. + +### 6.5 Private Notes belong to the Work (with a stated tradeoff) + +Private Notes are reminders ("chapter 3 is relevant to X", "ask Y about +this"). Most of them concern the work, so they stay on the Work, unchanged. +**Tradeoff:** some reminders are file facts ("this scan is missing pp. 40–45"). +The right home for those is Asset state (a short Asset `note` or condition +flag), not moving Private Notes. The notes deliberately do not share a scope: +Research and Private Notes stay separate conflict units on the Work, as they +are today. + +--- + +## 7. People and roles + +### 7.1 Scope by role type + +A role describes either **who made the intellectual work** or **who made +this edition**. The default scope for each current role type: + +| Role type | Default scope | Reasoning | +| --- | --- | --- | +| Author | **Work** | The creator of the intellectual work, the same for every translation. | +| Editor | **Manifestation** | Usually edition-specific: a critical edition's editor, or a volume editor of one printing. **Ambiguous for anthologies**, where the editor is the creator of the compiled work. The user can move it to Work scope. | +| Translator | **Manifestation** | Identifies one translation. | +| Introduction / Foreword / Afterword | **Manifestation** | They belong to one edition's apparatus. | +| Reviewer | **Manifestation** | A review, or a peer review, addresses a specific publication form. **Ambiguous in PRKS today.** The role's current product meaning is not documented, so §18 Q7 asks. | +| Mentioned | **Work** | "Person discussed in this work" is a research relationship. See §7.4 about its asset-derived origin. | + +These are **defaults applied at backfill and at role creation**. Scope is +stored per row, so a user can override a default. + +### 7.2 Storage + +- `roles` gains a nullable `manifestation_id` (FK to `manifestations`, CASCADE). + NULL means Work scope. Keeping one table, rather than adding + `manifestation_roles`, means every current `roles` reader (People pages, + credit helper, Research Graph, BibTeX) keeps working with a + filter instead of a UNION. +- The uniqueness (I9) becomes + `UNIQUE(person_id, work_id, role_type, COALESCE(manifestation_id, ''))`, so + one person can translate two different translations of one Work. +- **Credit-name override:** the credit on a Work-scoped Author row is the + default. A Manifestation may override how that person is credited on it + through `manifestation_credit_overrides(manifestation_id, person_id, + role_type, credit_name)`. This is optional and ships only when a UI needs it. + Backfill writes none. + +### 7.3 Backfill rule + +Every existing Work has exactly one Manifestation, so scope is invisible today. +The backfill therefore sets `manifestation_id` on the edition-scoped role types +(Editor, Translator, Introduction, Foreword, Afterword, Reviewer) and leaves +Author and Mentioned at Work scope. **Nothing visible changes.** The legacy +projection (§13) includes Work-scoped roles plus the primary Manifestation's +roles, which is exactly today's set. The first time a user adds a second +Manifestation, Translator and Editor correctly do **not** appear on it. + +### 7.4 Mentioned roles from PDF bytes + +`_sync_mentioned_roles` adds `Mentioned` roles from `[[Name]]` markup found in +a PDF when its bytes are replaced. In the new model the extraction runs per +Asset and records the Asset as provenance. The resulting role stays at Work +scope, and it is **never removed automatically** because another Asset lacks +the markup. This matches today's add-only behavior. + +### 7.5 What a Manifestation's full credit list is + +``` +credits(M) = Work-scoped roles of M.work (with M's credit overrides applied) + ∪ roles where manifestation_id = M +``` + +This is the single definition the citation projection (§8), the Work detail +people panel, and future CSL output all use. + +--- + +## 8. Citations + +Coordinated with #41. This design fixes **what a citation points to**. #41 +owns formatting, CSL and Citation.js. + +### 8.1 Ownership + +All citation metadata is owned by the **Manifestation**: entry type, title and +subtitle (inherited from the Work when NULL), date, edition, publisher, +location, journal, volume, issue, pages, identifiers, URL and access date, +language, and edition-scoped credits. The Work supplies only what the +Manifestation inherits (title, abstract) and its Work-scoped creators. + +### 8.2 What a citation points to + +**A citation targets a Manifestation**, never the abstract Work. A pinpoint +("p. 45") is a citation target plus a locator, and it is only meaningful +together with that Manifestation's pagination. + +### 8.3 Preferred Manifestation + +`works.primary_manifestation_id` is the version shown in lists, opened by +default, and **cited by default**. The single pointer is deliberately simple: + +- Copy citation / BibTeX / CSL on a Work cites its primary Manifestation. +- An explicit "cite this version" on any Manifestation cites that one. +- Batch export resolves **each Work to exactly one Manifestation** (the primary, + unless the caller names another) and never flattens several bibliographic + records into one entry. +- If users need a separate citation default (read the translation but cite the + original), a nullable `works.citation_manifestation_id` (NULL = primary) can + be added without migration pain. §18 Q2 asks whether it is needed now. + +### 8.4 Coexisting citeable Manifestations + +Each Manifestation produces its own complete entry. Two translations produce +two entries with different titles, translators, publishers and ISBNs. Cite +keys must be unique across Manifestations. The current computed key +(``) already collides across Works and would collide across +versions. #41 decides between a stored `cite_key` per Manifestation (stable for +LaTeX users) and a deterministic key with disambiguation. This design reserves +a nullable `manifestations.cite_key` and nothing else. + +### 8.5 Research citations inside PRKS + +`argument_sources(argument_id, work_id, pages)` is PRKS's own citation. It +gains a nullable `manifestation_id`: + +- `pages` empty → the Argument cites the **Work**. `manifestation_id` stays + NULL and renders against the primary. +- `pages` non-empty → the pinpoint was written against one pagination, so the + backfill **pins** it to the default Manifestation, and new pinpoints are + pinned to the Manifestation the user is looking at. Changing the primary + Manifestation later never moves an existing pinpoint. + +### 8.6 BibTeX compatibility projection + +During the migration `/api/bibtex/:id` keeps its URL, which takes a Work ID, +and its output. The generator is split into: + +``` +citation_record(manifestation_id) -> plain dict (entry type, fields, credits) +bibtex_from_record(record, export_profile) -> str +/api/bibtex/ = bibtex_from_record(citation_record(primary(work_id))) +``` + +The record builder is where #41's CSL-JSON adapter will attach. A +**byte-identical output test** over a fixture library proves parity before the +old generator is deleted. The current `urldate`-from-`updated_at` behavior is +preserved **deliberately** in the projection, documented as a known quirk, and +left for #41 to change with its own review, because changing citation output is +not a #60 concern. + +--- + +## 9. Asset lifecycle and storage + +### 9.1 Asset record + +| Field | Meaning | +| --- | --- | +| `id` | `AS-` + 32 hex (§11). Stable for the Asset's life. It survives merges and moves. | +| `manifestation_id` | Owner (FK). Moving an Asset to another Manifestation is an explicit operation. | +| `kind` | `managed_file` \| `external_stream` \| `external_link`. | +| `role` | `original` \| `annotated_export` \| `snapshot` \| `readable_text` \| `attachment` \| `other`. It describes what the content is, not how it is stored. | +| `storage_locator` | For `managed_file`: the managed **basename** under its storage area (`pdfs/` today). **Never an absolute path**, the same reasoning as `pending_pdf_cleanup`. The legacy `/api/pdfs/` form stays in the Work projection. | +| `provider`, `provider_id`, `url` | For `external_stream` / `external_link`. The `SET_WORK_SOURCE` aggregate moves here unchanged (I3). | +| `media_type` | `application/pdf`, `text/html`, `video/*` hint, … | +| `byte_size` | Size of the current bytes. Filled at write time, or by the fingerprint pass. | +| `ingest_sha256` | SHA-256 of the bytes **as first ingested**, before linearization or materialization. Immutable once set. NULL for legacy Assets (§9.4). | +| `content_sha256` | SHA-256 of the **current** bytes. Updated whenever PRKS rewrites them. Serves integrity checks and client cache validation. | +| `origin` | `upload` \| `processing_import` \| `adopted` \| `web_capture` \| `materialized` \| `legacy`. | +| `origin_url`, `origin_ref` | Where the bytes came from (a download URL, or a Processing File ID). This is provenance, not a citation. | +| `derived_from_asset_id` | Lineage, for example an annotated export of a publisher PDF. | +| `supersedes_asset_id` | "This file replaces that one" within one Manifestation. | +| `state` | `active` \| `trashed` (reserved for #57). Deletion removes the row (§9.2). | +| `thumb_page`, `thumb_url` | Presentation (from `works`). | +| `canonical_annotation_set_revision`, `materialized_annotation_revision` | From `works` (§6.1). | +| `captured_at` | For snapshots (Web Works). | +| `created_at`, `updated_at` | | + +**PDF bytes never go into SQLite.** Web snapshots are also stored as managed +files, not as DB blobs. + +### 9.2 Lifecycle + +``` + ingest (upload / import / capture / adopt) + │ bytes durable first (fs_durability), then row commits + ▼ + ┌─────────────┐ materialize / linearize: content_sha256 changes, + │ active │ ingest_sha256 does not + └──┬───────┬──┘ + detach/trash │ │ delete (Asset, or cascade from M/Work) + (#57, future) ▼ ▼ + ┌─────────┐ row deleted + pending_pdf_cleanup claim in the SAME txn + │ trashed │ (only if no other Asset/row references the basename) + └────┬────┘ │ + restore ◄────┘ purge ───────┤ + ▼ + bytes removed post-commit, retried by the existing + bounded retry pass (unchanged rules, AGENTS.md) +``` + +- **Availability is observed, not canonical.** "Missing file" means the + locator resolves to nothing on disk. It is computed by the same checks that + produce `file_size_bytes` today, may be cached as derived state, and **never + deletes or rewrites an Asset row**. A missing Asset still owns its + annotations. Backup reports it (as `managed_pdfs_missing` does now), and + restoring the bytes heals it. +- **Deletion keeps the cleanup model.** The reference check behind + `pending_pdf_cleanup` (three-valued, re-asked under the basename lock, and + settled in one transaction with the check) is extended to count + `assets.storage_locator` references in addition to `works.file_path` while + both exist. The claim still stores a basename. +- **Replacement.** "Replace file" in the UI creates a new Asset. The old one + becomes non-primary and remains, or is trashed if the user chooses. Only + materialization and linearization rewrite bytes in place, and both are PRKS + operations on the same content. + +### 9.3 Shared bytes + +Today, creation may adopt an existing managed basename, so two Works can share +bytes, and `work_pdf_replace` does copy-on-write. In the new model this is +**two Assets with the same `storage_locator`**. Copy-on-write stays exactly +where it is: before rewriting a locator that another Asset references, the +writing Asset is retargeted to a fresh minted name. Shared locators are +allowed but never created by new features. New ingestion always mints a +name, and duplicates are detected by hash instead (§10.2). + +### 9.4 Why two hashes + +PRKS rewrites managed PDFs in place when it linearizes them and when it +materializes annotations (§1.3). A single "content hash" would therefore +change every time the user saves a highlight, making it useless as an +exact-duplicate key. So: + +- `ingest_sha256` is the identity of what was brought in. It is computed on + the incoming stream **before** linearization, and it is what "have I already + got this file?" compares against. +- `content_sha256` is the identity of what is stored now. It is used for + integrity, backup verification, and client cache validation (#52, #58). + +Legacy Assets never had their original bytes preserved, so their +`ingest_sha256` stays **NULL**. PRKS must not invent it from bytes that have +since been materialized. Exact-duplicate detection against legacy files uses +`content_sha256` and is documented as weaker: it matches an unannotated legacy +file, and misses one whose bytes PRKS has rewritten. + +### 9.5 Backup and restore + +- The archive layout is unchanged: the DB plus `files/pdfs/*`. Asset rows live + in the canonical DB and are covered by it. No new filesystem component, so + the backup inventory needs no new classification. When Web Works add a + second managed area (for example `snapshots/`), it must be classified as + canonical in the inventory in the same PR. +- `audit_managed_pdfs` switches from `works.file_path` to the `assets` locators + when Assets become authoritative (slice D), and during the transition it + checks both. +- Restore verification may also check `content_sha256` for each Asset in the + archive. That is a stronger integrity check than today's presence check, and + it is optional. +- Derived per-Asset artifacts (the text index keyed by `asset_id`, thumbnails) + stay derived and are rebuilt after restore. + +--- + +## 10. Identity and deduplication + +Three different questions, answered by three different mechanisms, and **none +of them merges anything by itself**. + +``` + incoming file ──► ingest_sha256 ──► exact Asset duplicate? (certain) + │ + metadata ──► normalized identifiers (DOI / ISBN-13 / arXiv) ──► strong candidate + │ + └────► normalized title + creators + year (RapidFuzz later) ─► weak candidate + │ + every candidate ──► user decision ◄─────────────┘ + merge Works │ attach as Version │ attach as File │ not a duplicate +``` + +### 10.1 Levels + +| Level | Evidence | Confidence | Allowed automation | +| --- | --- | --- | --- | +| Exact Asset duplicate | equal `ingest_sha256` (or `content_sha256` for legacy files) | Certain for the content | **Warn before import**, showing the existing Work. Never auto-link or auto-delete. | +| Identifier match | equal normalized DOI, ISBN-13, or arXiv ID (versionless → same work, versioned → same revision) | Strong for "same publication", but **not** proof of "same Work" (a DOI identifies one form) | Suggest. The user decides. | +| Metadata similarity | normalized title, creator family names, year, publisher/journal | Weak | Suggest in a review list only. RapidFuzz may rank candidates (#42/#179). **Never decides.** | + +### 10.2 Exact duplicates + +- Computed at ingestion (`POST /api/works` upload, Processing import, and future + capture) **before** the Work is created, so the warning can offer choices: + - open the existing Work; + - import anyway as a separate Work; + - add as another File of an existing Version; + - add as a new Version of an existing Work. +- Identical content inside **one Manifestation** is harmless and allowed, and + is shown as "identical copy". +- There is no UNIQUE constraint on hashes: users may legitimately keep + duplicates. + +### 10.3 Bibliographic candidates + +- Normalization rules are **pure functions** with unit tests: DOI is lower-cased + and stripped of `https://doi.org/` and `doi:`; ISBN-10 is converted to + ISBN-13 with hyphens removed; an arXiv ID is split into versionless and + version parts; titles are NFC-normalized, case-folded, and have punctuation + and whitespace collapsed. +- Candidates are computed on demand or by a bounded pass. They are **derived**, + never canonical, and can be rebuilt. +- A DOI match between a *preprint* and a *journal* Manifestation is the + canonical "attach as Version" suggestion, not "merge". + +### 10.4 User-confirmed operations + +Each is a canonical domain command in the #182/#199 style, with a preview +before mutation. None of them is silent. + +| Operation | Effect | IDs | +| --- | --- | --- | +| `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed. `last_opened_at` is max. | The source `W-…` gets `work_lifecycle(state = merged, target)`. Old links, tabs, `[[W-…]]` and pending operations resolve through the redirect, like `sync_tag_lifecycle`. | +| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. If the old Work is left with no Manifestation, it is merged into the target or deleted, and the user chooses. | +| `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." Annotations stay with the Asset. | `AS-…` unchanged. | +| `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | + +### 10.5 Declined suggestions + +`duplicate_decisions` is canonical (it is a user decision), is backed up, and is +consulted by every candidate generator. A decision is on an ordered pair of +entity IDs, so it survives unrelated edits. It is revisited only if an ID is +merged away. + +--- + +## 11. Stable IDs + +### 11.1 Decision: `W-…` keeps meaning "the logical Work" + +Every existing Work ID continues to identify **the same user-facing thing**: +the item in lists, the target of tags, folders, notes, playlists, argument +sources, research mentions, URLs and workspace tabs. After backfill each Work +has exactly one Manifestation and at most one Asset. Therefore: + +- every Work-owned reference (tags, folder, playlist, notes, status, open + state, research links) **needs no change**; +- every reference to Manifestation- or Asset-owned state that currently goes + through the Work (annotations, pages pinpoints, file path, materialization, + text index, thumbnails, last-read page) is re-pointed **once, at backfill**, + to the Work's single Manifestation or Asset. That is a 1:1 mapping, so no + meaning changes for existing data. + +The alternative, reusing `W-…` as the Manifestation ID, is rejected in §16. + +### 11.2 New IDs + +| Entity | Format | Minted by | +| --- | --- | --- | +| Manifestation | `MF-` + 32 hex (distributed form per `entity_ids`) | server, or an offline client later | +| Asset | `AS-` + 32 hex | server (binary ingestion is online-only, I13) | +| Relation, identifier and decision rows | natural composite keys | — | + +`MF` and `AS` are unused. `M` is taken by publisher aliases and `A` by +Arguments, and `entity_ids` warns against inferring type from prefix. APIs +always pair an ID with its entity type. + +**Backfill IDs are deterministic**: `MF-` + `uuid5(NS_PRKS_BACKFILL, +"manifestation:" + work_id)`, and `AS-` + `uuid5(…, "asset:" + work_id)`, +upper-case hex. This format matches `entity_ids.is_distributed`. Consequences: + +- the migration is **reproducible**: restoring the same pre-migration backup + twice, or migrating two copies, yields identical IDs; +- the mapping from a legacy `work_id`-keyed operation to "the Manifestation + it originally meant" is **computable forever**, even after the Work gains + more versions (§14.2); +- clients must still **never derive** these IDs themselves. The server returns + them. Deterministic IDs are a migration property, not a protocol. + +--- + +## 12. Migration and backfill + +### 12.1 Principles + +- **Additive first.** New tables and new nullable columns. No column is dropped + or renamed until every reader has moved (the strangler approach from #179). +- **One authority per field at every moment.** At any given schema version, + each field has exactly one canonical home. The other copy is a projection + kept current **in the same transaction**, by triggers or by the single + canonical mutation boundary. Moving authority is its own migration. +- **No filesystem work in migrations** (I10). Hashing is a separate bounded, + resumable pass (§12.4). +- **No user classification.** Every existing Work becomes Work → 1 Manifestation + → 0 or 1 Asset automatically. +- Each step leaves `master` deployable, and every step follows `backend/AGENTS.md` + "Database schema changes" (one migration, a matching `db_schema.sql`, and + fresh-DB plus upgraded-DB tests). + +### 12.2 The common case + +``` +before (v16) after backfill +──────────── ────────────── +works W-1 works W-1 (Work-owned columns authoritative; + title, status, notes, tags… legacy bibliographic/source columns still present) + doi, publisher, year, … primary_manifestation_id = MF-uuid5(W-1) + file_path=/api/pdfs/x.pdf │ + canonical/materialized revs ▼ +annotations(work_id=W-1) manifestations MF-uuid5(W-1) +roles(W-1, Translator) work_id=W-1, kind='unspecified', title=NULL (inherit) +argument_sources(W-1, pages='45') doi, publisher, year, … (copied) + primary_asset_id = AS-uuid5(W-1) + │ + ▼ + assets AS-uuid5(W-1) + kind='managed_file', storage_locator='x.pdf', + media_type='application/pdf', origin='legacy', + ingest_sha256=NULL, content_sha256=NULL (pass fills), + materialization revisions (copied), thumb_page (copied) + annotations.asset_id = AS-uuid5(W-1) + roles(W-1, Translator).manifestation_id = MF-uuid5(W-1) + argument_sources(W-1,'45').manifestation_id = MF-uuid5(W-1) +``` + +The backfill rules: + +- A **video** Work gets an `external_stream` Asset carrying `provider`, + `provider_id`, `source_url` and `thumb_url`. +- A Work with **no file and no video** gets a Manifestation and **no** Asset. +- A non-video `source_url` goes to `manifestations.url`. +- A **legacy inferred-video row** (kind NULL, no file, a URL) is classified with + `effective_source_kind()`, the same rule every reader uses. When the URL + cannot be parsed, the row gets no Asset and keeps its URL on the + Manifestation. Nothing is guessed (see work-source-identity.md §10). + +### 12.3 Order of migrations + +The versions below are illustrative. Each is one migration in one PR (see +§17). + +1. **vN: create the entities.** Create `manifestations`, `assets`, + `manifestation_relations`, and `manifestation_identifiers`. Add + `works.primary_manifestation_id`, `annotations.asset_id`, + `roles.manifestation_id`, and `argument_sources.manifestation_id`. Backfill + deterministically, then validate (every Work has exactly one primary + Manifestation that belongs to it, and so on). Install **mirror triggers** + so that `works` (still authoritative for everything) keeps the default + Manifestation and Asset current on INSERT/UPDATE, and the child FKs + cascade on DELETE. **No reader changes.** +2. **vN+1: Asset authority.** Asset-owned state (locator, source aggregate, + materialization revisions, thumbnail page, annotation ownership) becomes + authoritative on `assets`. The migration drops those mirror triggers and + installs the reverse projection (`assets` → `works.file_path` and the other + legacy columns) for readers not yet migrated. Only one direction exists at + any version. The text index moves to `asset_id` keys, which is derived and + rebuilt, not migrated. +3. **vN+2: Manifestation authority.** The same move for bibliographic fields. + The migration **copies revision counters** from `work-field/[work, f]` to + `manifestation-field/[MF-uuid5(work), f]` (§14.2). +4. **vN+3: Roles scope enforced.** Swap the uniqueness index to include + `manifestation_id`, and set role scope by role type (§7.3). +5. **Later: retire legacy columns.** Drop them only after every reader uses the + projection module. Because SQLite `DROP COLUMN` has restrictions (FTS + triggers, indexes), this may require a table rebuild. It is optional, and the + columns can simply stay as the maintained projection. + +### 12.4 Fingerprint pass (not a migration) + +This is a bounded, resumable pass modeled on `retry_pending_pdf_cleanup`: a +fixed number of Assets per run, at startup or on demand. It streams each +managed file through SHA-256, records `content_sha256`, `byte_size` and a +`fingerprinted_at`, and skips missing files, which are reported, never +treated as empty (the same rule as the text index). It never modifies bytes. +It stores hashes in the canonical DB because they describe canonical bytes, and +they are re-verifiable, so a restore can recompute them. + +### 12.5 Rollback and recovery + +- PRKS has **no down migrations**, and this design does not add any. A newer + schema is refused by older code and by restore (`schema_newer`). So the + recovery path after an upgrade is the **pre-upgrade backup**. Slice A should + make sure the upgrade flow recommends or creates one (§18 Q9). +- Each migration is transactional (`_run_in_transaction`). A failure leaves the + previous version intact, and a crash mid-migration rolls back. +- Additive steps are **forward-fixable**: a bad backfill can be corrected by a + later migration, because the legacy columns were not removed. +- Mirror and reverse triggers are validated by `validate_current_schema`. A + parity test asserts, for every Work in fixture libraries, that the legacy + columns equal the projection of the new entities. + +### 12.6 Old and new code coexisting + +Within one PRKS server there is exactly one code version, so "coexistence" +means **old call sites next to new call sites**, in two forms: + +- Unmigrated readers keep reading `works` columns, which the triggers or + projection keep correct. +- Browser clients running an older cached app shell keep sending legacy + operations. The server keeps accepting them (§14.2) until the durable + outbox of every supported client has drained. A new client never sends a + legacy operation for a Work with more than one Manifestation. + +--- + +## 13. Compatibility and API projection + +### 13.1 One projection module + +A new focused module, for example `backend/work_projection.py`, owns: + +``` +legacy_work(work_id) = Work + + primary Manifestation (fields; title/abstract inherited) + + primary Asset (file_path, source_kind, provider…, thumb_*) + + credits(primary M) → today's Work dict +legacy_work_summary(rows) = same, for list/browse/recent/folder/person/playlist rows +``` + +Every endpoint that returns a Work today moves onto it, **one family at a time**, +with a parity test showing the JSON is identical to the pre-migration output +for the fixture library. The legacy dict gains only **additive** fields: +`primary_manifestation_id`, `primary_asset_id`, `manifestation_count`, +`asset_count`. Clients that do not know these fields ignore them. + +### 13.2 Writes through the legacy shape + +| Legacy write | Routed to | +| --- | --- | +| `PATCH /api/works/:id` with a Work-owned field | the Work | +| … with a Manifestation-owned field | the **primary** Manifestation. The editor shows the primary, so this is what the user sees. The response names the `manifestation_id` it wrote. | +| … with a source field | the primary Asset's source aggregate (`SET_WORK_SOURCE` semantics unchanged) | +| `POST /api/works` | creates Work + Manifestation + Asset in one transaction | +| `POST /api/works/:id/pdf` (materialization) | the **primary Asset**. The client must send `asset_id` once more than one exists. Without it the request is refused (`ASSET_AMBIGUOUS`) rather than guessed. | +| Annotation endpoints | the annotation's own Asset. A new annotation needs the displayed `asset_id`, which defaults to the primary only while exactly one PDF Asset exists. | +| `/api/bibtex/:id` | `citation_record(primary)`, §8.6 | + +### 13.3 New API surface (typed, #185 pattern) + +New endpoints use Pydantic request/response DTOs, the OpenAPI fragment, and +openapi-core tests from the start. They are the surface MCP (#47) and Android +(#52) consume: + +``` +GET /api/works/:id/manifestations list (with assets) +GET /api/manifestations/:id detail + citation record +POST /api/works/:id/manifestations add a Version +PATCH/DELETE /api/manifestations/:id +POST /api/manifestations/:id/assets add a File (online; binary) +PATCH/DELETE /api/assets/:id +POST /api/works/:id/primary-manifestation set primary +GET /api/assets/:id/content serves bytes (the /api/pdfs route stays) +``` + +Nothing here is implemented in this PR. + +### 13.4 Keeping the simple case simple + +- A Work with one Manifestation and at most one Asset **looks exactly as it does + today**. No "Versions" UI, no new fields in the editor, no new words. +- The richer model appears only when it has something to show: + - a "Versions" section appears on Work detail once a second Manifestation or + Asset exists, or from an explicit "Add another version / file…" action; + - a version picker appears in Copy citation and in the viewer only when more + than one choice exists; + - duplicate warnings appear at the moment of import; + - #61's Secondary Reading View offers "open another version". +- UI words are "Version" (Manifestation) and "File" (Asset). "Manifestation", + "Asset" and "FRBR" stay out of the UI. + +--- + +## 14. Backup/restore and offline/durable operations + +### 14.1 Backup/restore + +- Backup `FORMAT_VERSION` stays 1. The manifest already carries + `db_schema_version`, which is what distinguishes pre- and post-#60 archives. +- A **pre-#60 backup restored into post-#60 PRKS** is migrated on open like any + older DB. Deterministic backfill IDs mean the result is identical to what the + original library became. +- A **post-#60 backup into pre-#60 PRKS** is refused (`schema_newer`). This is + the existing rule. +- `pending_pdf_cleanup` claims travel with the DB, and after restore they are + re-evaluated against a catalogue that now includes Asset locators. +- The fingerprint pass reruns after restore for Assets with a NULL or suspect + hash. + +### 14.2 Durable operations and revisions + +The architecture must not assume a permanent local-first target (#179), but the +existing durable-operation guarantees (I7, I12) are compatibility constraints +and are kept. + +**Scopes follow ownership, and revisions are carried, not reset.** + +| Current scope | Future scope | Migration | +| --- | --- | --- | +| `work-field/[W, f]` for M-owned `f` | `manifestation-field/[MF, f]` | Copy the revision to `MF-uuid5(W)` in the authority migration. | +| `work-field/[W, f]` for W-owned `f` (status, title, abstract, author_text) | unchanged | — | +| `work-source/W` | `asset-source/AS` | Copy to `AS-uuid5(W)`. | +| `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `AS-uuid5(W)`. Deletion tombstones are copied too, so resurrection protection survives. | +| `work-person-role/[W, P, r]` | unchanged for W-scope; `manifestation-person-role/[MF, P, r]` for M-scope | Copy for backfilled M-scoped roles. | +| notes, tags, folder, playlist, open | unchanged | — | + +**Legacy operations after the move.** A pending `SET_WORK_METADATA_FIELD(W, +doi, base=7)` enqueued before the upgrade is applied to +**`MF-uuid5(W)`**, the Manifestation the user was editing when they enqueued +it, **not to "whatever is primary now"**. If that Manifestation has since +moved to another Work or been deleted, the operation is refused with an +explicit conflict code. It is never retargeted. The same rule applies to +annotation operations (→ `AS-uuid5(W)`) and to `SET_WORK_SOURCE`. + +**New durable families** (Manifestation fields, M-scoped roles, set-primary, +relations) each declare their shape per "Adding a family" in +`docs/local-first-sync.md`. Asset creation stays online-only (I13). Durable +file ingestion is a separate future design, not assumed here. + +**Client caches.** `work:*` and the browse projections stay keyed by Work and +carry the additive fields. New cache entities are keyed by `MF-`/`AS-` IDs. +The service-worker PDF cache is already keyed by basename, which is per +Asset. The `prks.pdf.lastPage.` key becomes +`prks.pdf.lastPage.asset.`. The default Asset reads the legacy key +once as a fallback. + +**Merges and durable operations.** A pending operation that names a merged +`W-…` resolves through `work_lifecycle` for Work-owned state, the same way +`sync_tag_lifecycle` redirects Tags. Operations on the source Work's +Manifestations and Assets need no redirect, because those IDs survived the +merge. + +--- + +## 15. Downstream roadmap implications + +These are the constraints each epic should adopt, so that none of them invents +its own identity or asset model. + +| Issue | Implications | +| --- | --- | +| **#41 Citation V2** | Cite a **Manifestation**. Default to the Work's primary. Batch export resolves each Work to exactly one Manifestation. Build CSL-JSON from `citation_record(manifestation_id)` (§8.6), not from `works` columns. Decide cite-key persistence (§8.4) and the `urldate` quirk. Do not add citation fields to `works`. | +| **#42 Ingestion / Web Works** | Every ingest creates, or attaches to, Work → Manifestation → Asset through one canonical command. Web Work = Manifestation(`web_page`) + snapshot Asset(s) under a new managed area, classified as canonical in the backup inventory. Duplicate warnings use §10 (hash, then identifiers, then optional RapidFuzz ranking). Provenance attaches to Manifestation fields and Asset origin. BibTeX/RIS import creates Manifestations and uses identifier normalization. | +| **#57 History / Trash** | Trash granularity is Work, Manifestation, or Asset. Asset `state = trashed` keeps bytes until purge, and purge uses the `pending_pdf_cleanup` rules. Merge and move are history events with previews. `work_lifecycle` redirects are the durable record of merges. | +| **#58 Offline UX** | "Available offline" is a property of **Assets** (bytes) plus the Work's metadata. Default to the primary Asset. Validate cached bytes with `content_sha256`. Pending edits stay keyed to the entity they were made on (§14.2). | +| **#59 Capture** | Capture sends intent (URL, selection, snapshot) to the same ingestion command as #42. Dedup by normalized canonical URL, then the snapshot hash. No capture-specific Work shape. | +| **#61 Reading Workflow** | The Secondary Reading View opens **another Asset**, of the same Manifestation or of another Manifestation of the same Work, and reads `manifestation_relations` for labels. Reading location is per Asset. Each pane shows its own Asset's annotations. "Open corresponding passage" and annotation transfer are assisted, reviewed actions (§6.2). Never assume page correspondence. | +| **#47 MCP** | Tools take Work IDs for Work-level operations and **must** report which Manifestation or Asset a citation, quote or page came from. Use the typed endpoints (§13.3) rather than the legacy projection for anything version-sensitive. | +| **#52 Android** | Consume the typed API. Cache files by Asset ID, validated by `content_sha256`, never by `file_path`. Treat the legacy Work shape as display-only. | + +**Standing rule until slices D and E land** (restating #60's #179 note): new +feature work must not attach new file- or edition-specific state to the `works` +row. + +--- + +## 16. Rejected alternatives and tradeoffs + +1. **Reuse `W-…` as the Manifestation ID and mint new logical Work IDs.** This + matches the fact that today's rows are mostly bibliographic records, and it + would keep annotations and BibTeX attached for free. **Rejected:** tags, + folders, notes, playlists, research mentions, argument sources, URLs, + workspace tabs, `[[W-…]]` links and pending durable operations all mean "the + thing the user files". Every one of them would have to be re-pointed to a new + ID, and each re-pointing is a chance to lose data or break an offline client. + Keeping `W-…` as the Work re-points only the file-specific references, and + those are the ones that have to change semantically anyway. +2. **Give the backfilled Manifestation the same ID as its Work.** This would make + legacy scopes trivially compatible. **Rejected:** once a Manifestation moves + to another Work (merge), `W-123` would name a Work in one table and a + Manifestation of a different Work in another. Deterministic uuid5 IDs give + the same compatibility without the ambiguity. +3. **A mandatory FRBR/LRM four-level model (Work → Expression → Manifestation → + Item).** "Expression" (a specific translation or text) is real, but for + PRKS's use cases it is carried by `language` + `translation_of` + + M-scoped Translator. Two editions of the same translation are + `new_edition_of` links between two translation Manifestations. **Deferred, + not ruled out:** if users need to group "all editions of the Bloom + translation", a nullable `expression_key` on Manifestation can be added + without restructuring. +4. **Embed the default Manifestation in the `works` row permanently** (only + extra versions get rows). **Rejected:** it creates two kinds of + Manifestation with different code paths forever, and every consumer would + have to handle both. +5. **Separate `manifestation_roles` table.** **Rejected** in favor of a nullable + scope column (§7.2), so current role readers keep working with a filter. +6. **Annotations on the Manifestation.** Scans of one edition have different + coordinates. **Rejected.** Annotations go on the Asset. +7. **Research Notes on the Manifestation.** This would scatter one line of + thought across versions and break Concept/Argument mentions. **Rejected.** +8. **One content hash of the current bytes as identity.** This breaks on + PRKS's own materialization and linearization (§9.4). **Rejected** in favor + of `ingest_sha256` + `content_sha256`. +9. **Automatic merge on DOI/ISBN or high fuzzy score.** A DOI identifies one + publication form, and fuzzy matches are wrong often enough to destroy data. + **Rejected.** Every merge is user-confirmed with a preview. +10. **Store bytes in SQLite, or content-addressed storage replacing basenames.** + Blobs in SQLite are ruled out by #60. Content-addressed storage conflicts + with in-place materialization and the existing COW, cleanup and backup + model. **Rejected for now.** Hashes are metadata, and basenames stay the + locator. +11. **Dual-writing new and old columns from application code.** Every missed + write path would silently diverge. **Rejected** in favor of one authority + per field, with triggers or the single boundary maintaining the other copy + in the same transaction (§12.1). +12. **An ORM or a new sync framework for the new entities.** Out of scope per + #179 and this issue's boundaries. + +--- + +## 17. Proposed implementation slices + +These are derived from the repository's constraints: the migration rules in +`backend/AGENTS.md`, per-family durable scopes, the strangler pattern of +#182/#185/#199, and I10's ban on filesystem work in migrations. Each slice is +one PR (or a small sequence) and leaves `master` deployable with no +user-visible change until slice H. + +**Gating rule:** no UI or API may create a **second** Manifestation or Asset for +a Work until slices D, E and F have moved every M- and A-owned state off the +Work. Until then the 1:1 backfill guarantees that the legacy behavior is +exactly right. + +| Slice | Content | User-visible? | Depends on | +| --- | --- | --- | --- | +| **A. Entities + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1). `db_schema.sql`, schema validation, fresh and upgraded tests, and a parity test (legacy columns = the new rows) over fixture libraries, including video, inferred-video, no-file and shared-basename rows. No readers. | No | this design | +| **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | +| **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | +| **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `AS-uuid5(W)`. Browser last-page key migration. | No | C | +| **E. Manifestation authority** | vN+2: bibliographic fields owned by `manifestations`. `manifestation-field` scopes with carried revisions. `SET_WORK_METADATA_FIELD` mapped to `MF-uuid5(W)`. `argument_sources.manifestation_id` pinning enforced for new pinpoints. Identifier table populated from DOI/ISBN. | No | C | +| **F. Role scope** | vN+3: role-type scope defaults, the uniqueness swap, `credits(M)`, and the role-sync scope for M-scoped roles. | No | E | +| **G. Typed Version/File API** | §13.3 read endpoints first, then mutations as canonical domain commands (#182/#199 pattern) with OpenAPI and openapi-core tests. New durable families declared per the sync guide. | API only | D, E, F | +| **H. Versions UI** | "Add another version / file", set primary, open a specific Version or File, and a version picker for Copy citation, all behind §13.4's progressive disclosure. Follow `DESIGN.md` for Work detail composition. #61's secondary view builds on this. | **Yes** | G | +| **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256` (and `content_sha256` for legacy files). Offers the four choices in §10.2. | Yes | B, H | +| **J. Identifier candidates + decline** | Normalized DOI/ISBN/arXiv candidates, a review list, and `duplicate_decisions`. | Yes | E, I | +| **K. Merge / move workflow** | `MERGE_WORKS`, `MOVE_MANIFESTATION`, `MOVE_ASSET` with previews, and `work_lifecycle` redirects. Coordinate with #57. | Yes | J | +| **L. RapidFuzz evaluation** | A separate research/evaluation issue (dependency review, ranking quality on synthetic data). It only ranks; it never decides. | — | J | + +A, B and C can proceed in parallel after A's migration merges. D and E are +the risky slices, because they move durable-operation scopes. Each should run +the affected sync feature E2E groups before merge, and the full E2E gate once. + +--- + +## 18. Open questions for the maintainer + +Each question has a recommendation. Nothing in slice A depends on the answers +except Q1 and Q9. + +1. **Naming.** Internal `manifestation` / `asset`, UI "Version" / "File". + *Recommendation:* approve as proposed. Q1 blocks slice A, because the table + names become permanent. +2. **Separate citation default.** Is `works.primary_manifestation_id` enough, + or is `citation_manifestation_id` needed from the start? *Recommendation:* + primary only. Add the second pointer when #41 shows a need. +3. **Keep pristine originals?** Today materialization overwrites the ingested + PDF. Should new ingestion keep the original as its own Asset + (`role = original`) and materialize into a derived `annotated_export` + Asset? That costs up to twice the storage, but gives the user back a + "publisher PDF + my annotated copy" pair and a stable duplicate key. + *Recommendation:* yes for new ingestion, behind a setting. Legacy files stay + as they are. +4. **Abstract ownership.** Work-level with a Manifestation override (proposed), + or Manifestation-only? *Recommendation:* as proposed, which keeps FTS + unchanged. +5. **Per-version notes.** Are Manifestation- or Asset-level notes wanted + ("this translation is unreliable", "scan missing pp. 40–45")? + *Recommendation:* not now. If needed, add a small Version note field. Never + move Research or Private Notes. +6. **Per-version reading state.** Should `status` or "last opened" also exist per + Manifestation or Asset? *Recommendation:* Work-level `status` only. Add an + Asset-level `last_opened_at` in slice H so "open" returns to the last-used + file. +7. **Reviewer and Editor semantics.** What does the `Reviewer` role mean in + PRKS today, and should Editor default to Manifestation scope? + *Recommendation:* both default to Manifestation scope. Revisit if anthologies + are common in real libraries. +8. **`external_link` Assets.** Is a non-managed "link to the publisher page" + Asset useful, or is `manifestations.url` enough? *Recommendation:* the + `url` field is enough. Keep the `external_link` kind reserved but + unimplemented. +9. **Upgrade safety net.** Should the first #60 migration refuse to run, or + force an automatic backup, when no recent backup exists? *Recommendation:* + create an automatic pre-migration backup into the existing backup location + before running vN, and log only counts. This needs your call because it + writes to storage at startup. +10. **Cite keys.** Stored per Manifestation, or computed with disambiguation? + This is owned by #41, and the design only reserves the column. +11. **Retiring legacy `works` columns.** Keep them indefinitely as the + maintained projection, or plan a table rebuild once every reader has moved? + *Recommendation:* keep them until Android and MCP ship on the typed API, + then decide. From db0a4f44e1056227df8c1874859690c3e89fa9b0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:23:34 +0000 Subject: [PATCH 02/24] =?UTF-8?q?docs(#60):=20address=20review=20=E2=80=94?= =?UTF-8?q?=20argument=20source=20identity,=20merged-work=20writes,=20sour?= =?UTF-8?q?ce=5Fmime=20backfill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 54 ++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 10 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index fd52d8ab..4d9c8e0d 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -368,8 +368,8 @@ positions and page thumbnails. | `manifestation_relations(from_id, to_id, relation)` | Typed links between Manifestations of the **same** Work: `revision_of`, `published_version_of`, `new_edition_of`, `translation_of`, `reprint_of`. These carry "arXiv v2 revises v1", "journal version of the accepted manuscript", and "Serbian translation of the Greek text". They carry no page correspondence (#61 owns assisted correspondence). | | `manifestation_identifiers(manifestation_id, scheme, value, normalized)` | DOI, ISBN, arXiv ID (versioned and versionless), PMID, and similar. `works.doi`/`isbn` backfill into these later. This is the entity duplicate detection keys on. | | `roles` gaining a nullable `manifestation_id` | Edition-scoped credit (§7). | -| `argument_sources` gaining a nullable `manifestation_id` | A pinpoint citation is pinned to a pagination (§8.5). | -| `work_lifecycle(work_id, state, target_work_id)` | Redirects left by merges, modeled exactly like `sync_tag_lifecycle` (§10.4). | +| `argument_sources` gaining a nullable `manifestation_id`, with row identity `(argument_id, work_id, manifestation_id)` | A pinpoint citation is pinned to a pagination, and one Argument can cite several Versions of one Work (§8.5). | +| `work_lifecycle(work_id, state, target_work_id)` | Merge record, modeled like `sync_tag_lifecycle`: it redirects **reads** and refuses **writes** with `WORK_MERGED` (§10.4, §14.2). | | `duplicate_decisions` | Durable record of "not a duplicate" answers, so a declined suggestion does not return (§10.5). | | Metadata provenance | A per-field record of detected, retrieved or user-edited origin on Manifestation fields (#42). This design only reserves the ownership: provenance belongs to the Manifestation field it describes, never to the Work. | @@ -474,7 +474,7 @@ describes where today's value goes; §12 covers how. | Citation / cite key | **M** (a stored `cite_key` is optional future) | none (it is computed today) | §8. | | Source / provenance | **A** for bytes (`origin`, `origin_url`, `imported_from_processing_file_id`). **M** field provenance for metadata. | `origin = 'legacy'` | Bytes and metadata have different provenance. | | File path (`file_path`) | **A** (`storage_locator`) | copied | The Work projection keeps `file_path` (§13). | -| MIME / media type (`source_mime`) | **A** (`media_type`) | `application/pdf` for managed PDFs, NULL otherwise. `source_mime` is never populated today. | | +| MIME / media type (`source_mime`) | **A** (`media_type`) | Copy `source_mime` when it is non-empty. Otherwise infer `application/pdf` for managed PDFs, and leave it NULL for everything else. | `POST /api/works` accepts and stores a caller-supplied `source_mime`, so a user database may hold values even though the audited library had none. The backfill must never overwrite them. | | Byte size | **A** (`byte_size`) | filled by the fingerprint pass (§12.4) | It replaces the per-row `stat()` in `finish_work_summary_rows` once trusted. | | Content hash | **A** (`ingest_sha256`, `content_sha256`) | NULL until the fingerprint pass | §9.4 and §10.2. | | Derived text / OCR / thumbnails | **A**, derived | the rebuilt index is keyed by `asset_id` | They stay derived and are never backed up (I4). | @@ -692,6 +692,19 @@ gains a nullable `manifestation_id`: pinned to the Manifestation the user is looking at. Changing the primary Manifestation later never moves an existing pinpoint. +**Row identity must change with it.** Today the primary key is +`(argument_id, work_id)`, so a nullable column alone would still stop one +Argument from citing two Manifestations of the same Work (for example, a +passage in the Greek text and the same passage in a translation). It would also +make `MERGE_WORKS` collide when an Argument already cites both the source and +the target Work. The citation identity becomes +`(argument_id, work_id, COALESCE(manifestation_id, ''))`, enforced by a unique +index. The table is rebuilt in the same migration that adds the column (SQLite +cannot alter a primary key). The `argument-sources` durable scope is the whole +source list of one Argument, so it is unaffected. On merge, if re-pointing +would produce two rows with the same identity, the preview shows both and the +user keeps one or both pinpoints. Neither is dropped silently. + ### 8.6 BibTeX compatibility projection During the migration `/api/bibtex/:id` keeps its URL, which takes a Work ID, @@ -880,7 +893,7 @@ before mutation. None of them is silent. | Operation | Effect | IDs | | --- | --- | --- | -| `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed. `last_opened_at` is max. | The source `W-…` gets `work_lifecycle(state = merged, target)`. Old links, tabs, `[[W-…]]` and pending operations resolve through the redirect, like `sync_tag_lifecycle`. | +| `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. Pending **writes** to the source are refused with `WORK_MERGED` + `target_work_id`, like `TAG_MERGED`, and are never retargeted (§14.2). | | `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. If the old Work is left with no Manifestation, it is merged into the target or deleted, and the user chooses. | | `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." Annotations stay with the Asset. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | @@ -989,6 +1002,8 @@ The backfill rules: `provider_id`, `source_url` and `thumb_url`. - A Work with **no file and no video** gets a Manifestation and **no** Asset. - A non-video `source_url` goes to `manifestations.url`. +- `assets.media_type` takes the stored `source_mime` when one is present, and + is inferred only when it is absent. - A **legacy inferred-video row** (kind NULL, no file, a URL) is classified with `effective_source_kind()`, the same rule every reader uses. When the URL cannot be parsed, the row gets no Asset and keeps its URL on the @@ -1002,7 +1017,8 @@ The versions below are illustrative. Each is one migration in one PR (see 1. **vN: create the entities.** Create `manifestations`, `assets`, `manifestation_relations`, and `manifestation_identifiers`. Add `works.primary_manifestation_id`, `annotations.asset_id`, - `roles.manifestation_id`, and `argument_sources.manifestation_id`. Backfill + `roles.manifestation_id`, and `argument_sources.manifestation_id` (a table + rebuild that widens its key, §8.5). Backfill deterministically, then validate (every Work has exactly one primary Manifestation that belongs to it, and so on). Install **mirror triggers** so that `works` (still authoritative for everything) keeps the default @@ -1183,11 +1199,29 @@ Asset. The `prks.pdf.lastPage.` key becomes `prks.pdf.lastPage.asset.`. The default Asset reads the legacy key once as a fallback. -**Merges and durable operations.** A pending operation that names a merged -`W-…` resolves through `work_lifecycle` for Work-owned state, the same way -`sync_tag_lifecycle` redirects Tags. Operations on the source Work's -Manifestations and Assets need no redirect, because those IDs survived the -merge. +**Merges and durable operations.** A pending operation's `base_revision` +belongs to a scope keyed by the **source** Work. It says nothing about the +target Work's scope, so retargeting the operation would either accept a stale +write when the counters happen to match, or report a false conflict when they +do not. The rule therefore follows what `TAG_MERGED` already does: + +- A pending write to Work-owned state (notes, status, tags, folder, playlist, + Work-scoped roles, Work fields) that names a merged `W-…` is **refused** + with `409 WORK_MERGED` and `target_work_id`. It is never retargeted. The + client keeps the user's intended value and presents it as an explicit merge + conflict against the target ("apply to the merged Work?"). If the user + accepts, the client sends a **new** operation based on the target's current + revision. +- The merge itself does not rebase or combine revision counters. Source scopes + are tombstoned, so a stale device cannot resurrect them, and target scopes + advance normally for whatever the merge changed. +- `MARK_WORK_OPENED` is a max-register with no base revision (I7), so it may be + applied to the target without conflict. +- Operations on the source Work's Manifestations and Assets are unaffected. + Their scopes are keyed by `MF-…`/`AS-…` IDs, which survive the merge, so their + base revisions are still meaningful. +- **Reads** (links, tabs, `[[W-…]]`, cached detail requests) follow the redirect + to the target. --- From c196eeac536e243bad4db40dce1e702e3e917177 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:34:42 +0000 Subject: [PATCH 03/24] docs(#60): revise design per maintainer review - Record maintainer decisions D1-D11 (naming, no auto full backup, pristine originals as Asset storage slots, citation_manifestation_id, Work-level abstract, role scopes incl. Work-scoped Reviewer, no per-Version notes/status, no external_link, cite keys to #41, legacy columns kept). - Specify DB-enforced same-owner integrity: composite ownership FKs on new and rebuilt leaf tables, narrow triggers for works pointers. - Redesign argument_sources: key (argument_id, order_index), citation identity (work, version, pinpoint); rebuild/backfill and merge rules. - Replace uuid5-recomputed legacy mapping with immutable origin_work_id. - Refuse every pending operation on a merged Work with WORK_MERGED; merge transaction order; no implicit rebase. - Preserve source_mime; define Slice A acceptance criteria. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/wiki/Domain-Model.md | 2 +- docs/work-identity-model.md | 903 ++++++++++++++++++++++++++---------- 2 files changed, 647 insertions(+), 258 deletions(-) diff --git a/docs/wiki/Domain-Model.md b/docs/wiki/Domain-Model.md index c1e2f890..f7f5bb7c 100644 --- a/docs/wiki/Domain-Model.md +++ b/docs/wiki/Domain-Model.md @@ -63,7 +63,7 @@ Work source fields are not independent metadata. For video Works, source kind/pr ## Planned: Work identity, versions and files -Today one Work row represents the intellectual work, one citeable publication, and at most one file. The proposed separation into Work → Version (manifestation) → File (asset) — covering editions, translations, revisions, duplicate detection and citation targets — is a design under review, not current behavior. See [docs/work-identity-model.md](https://github.com/Fooftilly/PRKS/blob/master/docs/work-identity-model.md). +Today one Work row represents the intellectual work, one citeable publication, and at most one file. The proposed separation into Work → Version (manifestation) → File or Source (asset) — covering editions, translations, revisions, duplicate detection and citation targets — is a design under review, not current behavior. See [docs/work-identity-model.md](https://github.com/Fooftilly/PRKS/blob/master/docs/work-identity-model.md). ## IDs and offline creation diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 4d9c8e0d..70ba0348 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1,10 +1,13 @@ # Work identity, editions, versions and Assets: design (#60) -**Status: design proposal for review. Nothing here is implemented.** This +**Status: design proposal, revised after maintainer review. Nothing here is +implemented.** The three-level model has been approved in direction, and the +maintainer decisions are recorded in [§0](#0-maintainer-decisions). This document does not add schema, migrations, API behavior or UI. It is the design gate that #60 requires before large schema changes, and it follows the architecture direction in #179. Once approved, the implementation slices in §17 -become separate focused issues and PRs. +become separate focused issues and PRs. Slice A must not start until this +revision is approved. It follows the shape of [work-source-identity.md](work-source-identity.md): audit the code first, then recommend. Every claim about current behavior @@ -13,6 +16,7 @@ from issue text. Contents: +0. [Maintainer decisions](#0-maintainer-decisions) 1. [Current-state audit](#1-current-state-audit) 2. [Problems with the overloaded Work row](#2-problems-with-the-overloaded-work-row) 3. [Terminology and proposed entities](#3-terminology-and-proposed-entities) @@ -30,10 +34,29 @@ Contents: 15. [Downstream roadmap implications](#15-downstream-roadmap-implications) 16. [Rejected alternatives and tradeoffs](#16-rejected-alternatives-and-tradeoffs) 17. [Proposed implementation slices](#17-proposed-implementation-slices) -18. [Open questions for the maintainer](#18-open-questions-for-the-maintainer) +18. [Remaining open questions](#18-remaining-open-questions) --- +## 0. Maintainer decisions + +These were decided in the review of this PR. The rest of the document is +written to match them. + +| # | Topic | Decision | +| --- | --- | --- | +| D1 | Naming | Internal entities and tables are `manifestation`/`manifestations` and `asset`/`assets`. In the UI a Manifestation is a **Version**. An Asset is not called "File" everywhere: a managed file is a **File**, and an external stream such as YouTube is a **Source** (or a stream, depending on context). | +| D2 | Upgrade backup | Slice A does **not** create or require a full automatic backup before migrating. The migration adds tables and columns only, runs inside one SQLite transaction, and changes nothing on the filesystem. An automatic upgrade snapshot that covers only the DB, sized to what the upgrade touches, may be evaluated separately. PRKS never silently packs the whole library at startup. | +| D3 | Pristine originals | The architecture must allow keeping pristine originals. PRKS's own materialized (annotated) PDF is **not** a separate user-visible Asset. An Asset is one stable file lineage, and its original bytes and its generated materialization are storage slots inside that Asset (§9.4). No extra entity is added for this in Slice A. | +| D4 | Citation default | Add a nullable `works.citation_manifestation_id` from the start. NULL means "cite the primary". `primary_manifestation_id` is the Version the user normally reads and sees; `citation_manifestation_id` is the preferred Version for citing. There is no UI for it yet, because #41 owns that. | +| D5 | Abstract | The canonical abstract lives on the Work. A Manifestation may override it. | +| D6 | Roles | Author and Mentioned are Work-scoped. Translator, Introduction, Foreword and Afterword default to Manifestation scope. Editor defaults to Manifestation scope, and each row can be overridden (for anthologies and compiled works). **Reviewer stays Work-scoped by default**, because its current meaning is not documented well enough to assign every existing Reviewer to one Version. | +| D7 | Notes and reading state | No per-Version Research Notes, Private Notes or Progress/status for now. All three stay on the Work. Page and reading location belong to the Asset. An Asset-level "last opened" or last-used-Version mechanism may come later with #61. | +| D8 | `external_link` Assets | Not implemented. `manifestations.url` is enough. | +| D9 | Cite keys | Whether cite keys are stored, and how they are disambiguated, stays with #41. | +| D10 | Legacy `works` columns | They stay as a maintained compatibility projection until the typed API and future external clients are established. When to remove them is decided later. | +| D11 | Review fixes | (A) `argument_sources` gets a new row identity (§8.5). (B) The backfill keeps `source_mime` (§5, §12.2). (C) After a merge, pending operations are refused with `WORK_MERGED` and are never redirected (§10.4, §14.2). (D) Ownership consistency is enforced by the database (§4.1). | + ## 1. Current-state audit ### 1.1 The `works` row @@ -313,7 +336,8 @@ see the internal ones (§13.4). This is the intellectual identity the user reasons about: what they file, tag, annotate in notes, argue with, and read about. It is the **existing `works` entity with its existing ID** (§11). A Work always has at least one -Manifestation, and exactly one of them is **primary**. +Manifestation. Exactly one of them is **primary**, meaning the Version the user +normally reads and sees. Optionally, one is the **citation** Version (D4). ### 3.2 Manifestation: internal `manifestation`, UI "Version" @@ -343,23 +367,34 @@ replacement. "Manifestation" is the library-science term for "a specific embodiment someone can cite", and it is used here **without** adopting the rest of FRBR/LRM. See §16 for the rejected Expression layer. -### 3.3 Asset: internal `asset`, UI "File" +### 3.3 Asset: internal `asset`, UI "File" or "Source" + +This is **one stable source or file lineage that belongs to one +Manifestation**: a PDF the user added, a web snapshot, a future attachment, +or an external provider-hosted stream (a YouTube video). It owns: + +- the storage slots for its bytes (§9.4); +- media type, byte size and content hashes; +- provenance of the bytes; +- every piece of state defined by that content's own coordinates: PDF + annotations, page positions and page thumbnails. -This is **one concrete content object that belongs to one Manifestation**: -a managed PDF, a web snapshot, a future attachment, or an external -provider-hosted stream (a YouTube video). It owns the storage locator, media -type, byte size, content hashes, provenance of the bytes, and every piece of -state defined by that content's own coordinates: PDF annotations, page -positions and page thumbnails. +An Asset is **one lineage, not one byte string**. When PRKS linearizes a PDF or +materializes annotations into it, the bytes change but the Asset does not: no +new Asset is created, and nothing new appears to the user (D3). A new Asset +exists only when the user brings in another file or source: another scan, a +file from elsewhere, a re-captured snapshot. -`asset.kind ∈ { managed_file, external_stream, external_link }`: +`asset.kind ∈ { managed_file, external_stream }`: - `managed_file`: bytes stored by PRKS under managed storage (PDFs today, - web snapshots and attachments later). + web snapshots and attachments later). The UI calls it a **File**. - `external_stream`: identified by a provider and a provider ID (YouTube). - This is where today's `SET_WORK_SOURCE` aggregate moves. -- `external_link`: a URL PRKS does not store (for example "the publisher's - page"). This is optional and may never be needed. §18 Q8 asks. + This is where today's `SET_WORK_SOURCE` aggregate moves. The UI calls it a + **Source** or stream (D1). + +Links that PRKS does not store are **not** Assets. The Manifestation's `url` is +enough (D8). ### 3.4 Relationship and supporting entities @@ -367,9 +402,9 @@ positions and page thumbnails. | --- | --- | | `manifestation_relations(from_id, to_id, relation)` | Typed links between Manifestations of the **same** Work: `revision_of`, `published_version_of`, `new_edition_of`, `translation_of`, `reprint_of`. These carry "arXiv v2 revises v1", "journal version of the accepted manuscript", and "Serbian translation of the Greek text". They carry no page correspondence (#61 owns assisted correspondence). | | `manifestation_identifiers(manifestation_id, scheme, value, normalized)` | DOI, ISBN, arXiv ID (versioned and versionless), PMID, and similar. `works.doi`/`isbn` backfill into these later. This is the entity duplicate detection keys on. | -| `roles` gaining a nullable `manifestation_id` | Edition-scoped credit (§7). | -| `argument_sources` gaining a nullable `manifestation_id`, with row identity `(argument_id, work_id, manifestation_id)` | A pinpoint citation is pinned to a pagination, and one Argument can cite several Versions of one Work (§8.5). | -| `work_lifecycle(work_id, state, target_work_id)` | Merge record, modeled like `sync_tag_lifecycle`: it redirects **reads** and refuses **writes** with `WORK_MERGED` (§10.4, §14.2). | +| `roles` gaining a nullable `manifestation_id` | Edition-scoped credit (§7). The owner is enforced by a composite FK (§4.1). | +| `argument_sources` rebuilt with key `(argument_id, order_index)` plus a nullable `manifestation_id` | One Argument can cite the Work in general **and** several Versions of it, each with its own pinpoint (§8.5). | +| `work_lifecycle(work_id, state, target_work_id)` | Merge record, modeled like `sync_tag_lifecycle`: it redirects **reads** and refuses **every pending operation** naming the merged Work with `WORK_MERGED` (§10.4, §14.2). | | `duplicate_decisions` | Durable record of "not a duplicate" answers, so a declined suggestion does not return (§10.5). | | Metadata provenance | A per-field record of detected, retrieved or user-edited origin on Manifestation fields (#42). This design only reserves the ownership: provenance belongs to the Manifestation field it describes, never to the Work. | @@ -384,6 +419,7 @@ positions and page thumbnails. Work-level roles │ canonical title, abstract │ Arguments via mentions) (Author, Mentioned) └──────┬───────────────▲──────┘ │ 1..* │ primary_manifestation_id (exactly 1) + │ │ citation_manifestation_id (0 or 1) ▼ │ ┌──────────────────────────────┴──┐ manifestation_relations edition roles ───┤ Manifestation ├──── (revision_of, translation_of, @@ -402,9 +438,10 @@ positions and page thumbnails. | Relationship | Cardinality | Notes | | --- | --- | --- | | Work → Manifestation | 1 → 1..* | Never zero. A notes-only or metadata-only Work still has one empty primary Manifestation, so projections and citations always resolve. | -| Work → primary Manifestation | exactly 1 | `works.primary_manifestation_id`, NOT NULL after backfill, and must belong to that Work. | +| Work → primary Manifestation | exactly 1 | `works.primary_manifestation_id`. The Version the user normally reads and sees. It must belong to the same Work (§4.1). | +| Work → citation Manifestation | 0 or 1 | `works.citation_manifestation_id`. NULL means "cite the primary". If set, it must belong to the same Work (§4.1). | | Manifestation → Asset | 1 → 0..* | Zero means "metadata, no file". | -| Manifestation → primary Asset | 0 or 1 | `manifestations.primary_asset_id`. It is required to be non-NULL when the Manifestation has any active Asset. | +| Manifestation → primary Asset | 0 or 1 | `manifestations.primary_asset_id`. It must be non-NULL whenever the Manifestation has an active Asset, and it must belong to that Manifestation (§4.1). | | Asset → Manifestation | exactly 1 | An Asset is never shared by two Manifestations. Shared **bytes** are a storage fact handled below the Asset (§9.3). | | Annotation → Asset | exactly 1 | §6. | | Manifestation relations | same-Work only | Cross-Work links are research relationships, not version relationships. | @@ -420,7 +457,8 @@ positions and page thumbnails. | Original → translations | 1 Work → MF(grc, published) + MF(en, translation, `translation_of` grc) + MF(sr, translation, `translation_of` grc). The Translator role is on each translation MF. | | Two English translations | Two `translation` MFs with `language = en`, each with its own translator. | | Several scans of one exact edition | 1 MF → several Assets. One of them is primary. | -| Publisher PDF + locally annotated copy | 1 MF → Asset(publisher original) + Asset(annotated export, `derived_from_asset_id` = original). Today these are the same file (§9.4). | +| Publisher PDF + PRKS annotations | 1 MF → **1** Asset. The pristine publisher bytes (when kept) and PRKS's annotated materialization are two storage slots of that one Asset, not two Assets (D3, §9.4). | +| Publisher PDF + a copy annotated in another tool | 1 MF → 2 Assets. The user brought in a second file, and `derived_from_asset_id` may record where it came from. | | Future Web Work | 1 Work → MF(web_page, url, site, accessed date) → Asset(sanitized HTML snapshot) and optionally a readable-text Asset. A re-capture is a new Asset in the same MF. | | Metadata but no file | 1 Work → 1 MF → 0 Assets. | | Video | 1 Work → MF(video) → Asset(external_stream: youtube, provider_id). | @@ -428,6 +466,131 @@ positions and page thumbnails. | Probable bibliographic duplicate | Two Works whose MFs share a normalized DOI/ISBN or have similar title/author/year. They become a **candidate**, and only the user decides (§10.3). | | Genuinely separate works | Two Works. A declined candidate is recorded in `duplicate_decisions`. | +### 4.1 Ownership integrity (enforced by the database) + +The compatibility model keeps legacy parent IDs (`work_id`) next to the new +entity IDs (`manifestation_id`, `asset_id`). If those were separate FKs, every +referenced row could exist while the combination is impossible, for example a +role on `W1` scoped to a Manifestation of `W2`. The legacy projection, cascades, +moves, merges, sync scopes and cleanup would then disagree about who owns what. +So **same-owner consistency is a database invariant**. Canonical commands still +validate it first so they can return a useful error, but the database is the +final guard. + +**Mechanism.** SQLite gives two tools, and the design uses each where it +fits: + +1. **Composite ownership FKs** wherever the child table can be created or + rebuilt safely: the new tables, and the **leaf** tables `roles`, + `annotations` and `argument_sources`, which nothing references. A child + refers to the pair `(entity id, owner id)`, and that pair only exists on the + parent row if the ownership is real. +2. **Narrow integrity triggers** on `works`. Adding a composite FK to `works` + means rebuilding it, and it is the parent of about ten tables. PRKS + migrations run inside one `BEGIN IMMEDIATE` transaction with + `foreign_keys = ON`, and SQLite cannot turn foreign keys off inside a + transaction. Its `DROP TABLE works` step would therefore perform an + implicit cascading delete of every child row. `ALTER TABLE ADD COLUMN` + cannot add a table-level composite FK, so the two pointer columns are added + plainly and guarded by triggers instead. + +```sql +-- parent pairs (new tables) +manifestations: UNIQUE (id, work_id) +assets: UNIQUE (id, manifestation_id), UNIQUE (id, work_id) + -- assets.work_id is a denormalized copy of its Manifestation's Work + +-- composite ownership FKs +assets FOREIGN KEY (manifestation_id, work_id) REFERENCES manifestations(id, work_id) + ON UPDATE CASCADE ON DELETE CASCADE +manifestations FOREIGN KEY (primary_asset_id, id) REFERENCES assets(id, manifestation_id) + ON UPDATE RESTRICT ON DELETE RESTRICT DEFERRABLE INITIALLY DEFERRED +-- leaf tables, rebuilt in the same migration (SQLite cannot add a table constraint in place) +annotations FOREIGN KEY (asset_id, work_id) REFERENCES assets(id, work_id) + ON UPDATE CASCADE ON DELETE CASCADE +roles FOREIGN KEY (manifestation_id, work_id) REFERENCES manifestations(id, work_id) + ON UPDATE CASCADE ON DELETE CASCADE +argument_sources + FOREIGN KEY (manifestation_id, work_id) REFERENCES manifestations(id, work_id) + ON UPDATE CASCADE ON DELETE RESTRICT + +-- works pointers: ALTER TABLE ADD COLUMN + triggers +works.primary_manifestation_id TEXT +works.citation_manifestation_id TEXT +``` + +The existing single-column FKs (`roles.work_id → works`, +`annotations.work_id → works`, and so on) stay as they are. + +| Trigger | Rule | +| --- | --- | +| `works_manifestation_pointers_owned` (BEFORE UPDATE OF `primary_manifestation_id`, `citation_manifestation_id` ON `works`) | A non-NULL pointer must name a Manifestation whose `work_id` is this Work (`MANIFESTATION_OWNER_MISMATCH`). The primary may not be set back to NULL (`WORK_PRIMARY_MANIFESTATION_REQUIRED`). The one exception is a Work already recorded in `work_lifecycle` as `merged`, whose row the same merge transaction is about to delete (§10.4). | +| `works_manifestation_pointers_insert` (BEFORE INSERT ON `works`) | A Work is inserted with NULL pointers. Its Manifestation cannot exist before the Work, because `manifestations.work_id` references it. | +| `manifestations_pointer_target_move` (BEFORE UPDATE OF `work_id` ON `manifestations`) | A Manifestation that its Work names as primary or citation cannot be moved away (`MANIFESTATION_IS_POINTER_TARGET`). The pointer must change first, in the same transaction. | +| `manifestations_pointer_target_delete` (BEFORE DELETE ON `manifestations`) | The same rule for deletion. A whole-Work delete is unaffected: the Work row is gone before its Manifestations cascade. | +| `manifestation_origin_immutable`, `asset_origin_immutable` (BEFORE UPDATE OF `origin_work_id`) | The legacy-operation mapping (§11.2) can never be rewritten. | +| `manifestation_primary_asset_kept` (BEFORE UPDATE OF `primary_asset_id` ON `manifestations`, BEFORE UPDATE OF `state` ON `assets`) | `primary_asset_id` may not be cleared while the Manifestation still has an `active` Asset, and the primary Asset may not be trashed until the pointer has moved. Deleting it is already refused by the deferred FK. | + +What each required invariant maps to: + +| Invariant | Enforced by | +| --- | --- | +| `works.primary_manifestation_id` belongs to that Work | `works_manifestation_pointers_owned` plus the two `manifestations_pointer_target_*` triggers, so neither side of the pair can drift | +| `works.citation_manifestation_id`, if set, belongs to that Work | the same triggers. NULL means "use primary". | +| `manifestations.primary_asset_id`, if set, belongs to that Manifestation | composite FK `(primary_asset_id, id) → assets(id, manifestation_id)` | +| `roles(work_id, manifestation_id)` has a matching owner | composite FK. A NULL `manifestation_id` is a Work-scoped role, and the FK does not apply. | +| `argument_sources(work_id, manifestation_id)` has a matching owner | composite FK. A NULL `manifestation_id` is a Work-level citation. | +| `annotations(work_id, asset_id)`: the Asset's Manifestation belongs to that Work | `annotations → assets(id, work_id)`, and `assets(manifestation_id, work_id) → manifestations(id, work_id)`. The chain carries ownership from the Asset through its Manifestation. | +| An Asset's denormalized `work_id` matches its Manifestation's Work | composite FK on `assets` | + +**How the cascade and restrict choices behave:** + +- **Moves keep owners consistent.** Moving a Manifestation to another Work is + one `UPDATE manifestations SET work_id`. `ON UPDATE CASCADE` carries the new + `work_id` to that Manifestation's Assets, then to their annotations, and to + its scoped roles and pinned argument sources. No application code has to + re-copy owner columns. +- **A pointer target cannot leave silently.** A Manifestation that is its + Work's primary or citation Version cannot be moved away or deleted until the + pointer changes. The same holds for a primary Asset (deferred FK, checked at + commit). +- **A pinned citation blocks deleting a Version.** Deleting a Manifestation that + an Argument pins is refused (`RESTRICT`). The canonical command reports it as + in use, like `ARGUMENT_IN_USE` today. Deleting a whole Work still cascades + cleanly: every child row goes with it. + +**What SQLite cannot enforce at commit.** SQLite has no deferred triggers or +CHECKs, so "every Work *has* a primary Manifestation" and "a Manifestation with +an active Asset *has* a primary Asset" cannot be required at the moment a row +is inserted. Both are set by the single statement or canonical command that +creates the rows. In Slice A the mirror trigger does it: it inserts the +Manifestation (and Asset) and sets the pointers inside the Work's own INSERT. +The triggers above then make sure a pointer, once set, can never be cleared, +moved or pointed at a foreign row. An **integrity query** (Works with a NULL +primary, Manifestations with active Assets but no primary Asset) runs in the +schema tests, after the backfill, and in backup verification. + +All of the above was prototyped against SQLite 3.45 (the version used in +development). Contradictory rows for each invariant were refused, and legal +moves, pointer swaps and whole-Work deletes left `PRAGMA foreign_key_check` +clean. Adding the columns and triggers inside a transaction with foreign keys +on also left existing child rows intact. PRKS already enables +`PRAGMA foreign_keys` on every connection (`db_manager`, `db_migrations`). + +**Validation.** `validate_current_schema` / `_assert_known_table_shapes` gain +the composite FK tuples and the triggers, so a DB missing any of them fails +validation (the FK-tuple checks already exist). Slice A's tests cover fresh and +upgraded DBs: + +- each invariant's contradictory insert or update is refused; +- legal moves cascade their owner columns; +- a pointer target cannot be deleted or moved away while referenced; +- after the backfill, `PRAGMA foreign_key_check` reports nothing beyond + the legacy violations counted by the preflight (on clean fixtures, nothing at + all), and the integrity query returns no rows; +- the backfill preflight (§12.3) reports rows it cannot map rather than + inserting them. + --- ## 5. Ownership matrix @@ -461,22 +624,23 @@ describes where today's value goes; §12 covers how. | Folders (`folder_files`) | **W** | unchanged | One folder per Work (I8). | | Playlists (`playlist_items`) | **W** | unchanged | Ordered viewing sequence. One per Work (I8). | | Research Notes (`text_content`) | **W** | unchanged | §6.4. | -| Private Notes (`private_notes`) | **W** | unchanged | §6.5, with an open question. | +| Private Notes (`private_notes`) | **W** | unchanged | §6.5. No per-Version notes (D7). | | Annotations / comments (`annotations`) | **A** | `annotations.asset_id` = the Work's backfilled Asset | §6.1. Coordinates belong to one file. | | PDF page references in notes (`[[pdf:]]`) | resolve through the annotation's **A** | nothing to move | Annotation IDs are globally unique, so existing markers keep resolving. The viewer opens the owning Asset. | -| Pinpoint pages (`argument_sources.pages`) | **J**, pinned to **M** | `manifestation_id` set to the default M when `pages` is non-empty | §8.5. | -| Reading progress (`status`) | **W** | unchanged | "Have I read this work?" **Tradeoff:** someone who has read translation A but not the original may want per-version status. Deferred, §18 Q6. | +| Pinpoint pages (`argument_sources.pages`) | **J**, pinned to **M** | `manifestation_id` set to the default M when `pages` is non-empty. The table is rebuilt with a new key. | §8.5. | +| Reading progress (`status`) | **W** | unchanged | "Have I read this work?" No per-Version status (D7). **Tradeoff:** someone who has read translation A but not the original cannot record that yet. | | Last-read page (`localStorage prks.pdf.lastPage.`) | **A** (per device) | client falls back from the Asset key to the legacy Work key for the default Asset | Page N means nothing in another file. #61 needs an independent location per version. | -| Open/recent (`last_opened_at`) | **W** (max-register) | unchanged | Recent lists Works. An Asset-level `last_opened_at` may be added so "open" reopens the last-used version (§18 Q6). | +| Open/recent (`last_opened_at`) | **W** (max-register) | unchanged | Recent lists Works. An Asset-level "last opened" or last-used-Version value may come later with #61 (D7). | | Thumbnail page (`thumb_page`) | **A** | copied to the default Asset | It is a page of that file. The Work card uses the primary Asset. | | Remote thumbnail (`thumb_url`) | **A** | copied to the default Asset | Presentation of the video asset. | | Hide link annotations (`hide_pdf_link_annotations`) | **W** (viewer preference) | unchanged | A display preference for the sidebar list that applies to whichever PDF is shown. Low stakes. | -| Citation / cite key | **M** (a stored `cite_key` is optional future) | none (it is computed today) | §8. | +| Citation target default (new) | **W** pointer `citation_manifestation_id` → **M** | NULL (= primary) | Deliberately separate from `primary_manifestation_id` (D4, §8.3). | +| Citation / cite key | **M** (whether a `cite_key` is stored belongs to #41, D9) | none (it is computed today) | §8. | | Source / provenance | **A** for bytes (`origin`, `origin_url`, `imported_from_processing_file_id`). **M** field provenance for metadata. | `origin = 'legacy'` | Bytes and metadata have different provenance. | -| File path (`file_path`) | **A** (`storage_locator`) | copied | The Work projection keeps `file_path` (§13). | +| File path (`file_path`) | **A** (`storage_locator`, the working/served bytes) | copied | The Work projection keeps `file_path` (§13). `source_locator` (pristine bytes) stays NULL for legacy files (§9.4). | | MIME / media type (`source_mime`) | **A** (`media_type`) | Copy `source_mime` when it is non-empty. Otherwise infer `application/pdf` for managed PDFs, and leave it NULL for everything else. | `POST /api/works` accepts and stores a caller-supplied `source_mime`, so a user database may hold values even though the audited library had none. The backfill must never overwrite them. | | Byte size | **A** (`byte_size`) | filled by the fingerprint pass (§12.4) | It replaces the per-row `stat()` in `finish_work_summary_rows` once trusted. | -| Content hash | **A** (`ingest_sha256`, `content_sha256`) | NULL until the fingerprint pass | §9.4 and §10.2. | +| Content hash | **A** (`ingest_sha256` for the source bytes, `content_sha256` for the working bytes) | `content_sha256` from the fingerprint pass. `ingest_sha256` stays NULL for legacy files. | §9.4 and §10.2. | | Derived text / OCR / thumbnails | **A**, derived | the rebuilt index is keyed by `asset_id` | They stay derived and are never backed up (I4). | | Annotation materialization revisions | **A** | copied from `works` | They describe one file's bytes. | | Source kind (`source_kind`) | **A** (`kind` + `media_type`). The Work projection derives it. | derived | `prksInferWorkSourceKind` keeps working on the projection. | @@ -496,10 +660,12 @@ on (strictly, against the original bytes plus the materializations PRKS itself produced). They therefore attach to the **Asset**, never to the Manifestation or the Work: -- `annotations.asset_id` NOT NULL (after backfill), FK to `assets`. -- `annotations.work_id` is kept during the transition as a denormalized value - equal to the Asset's Work, so existing queries and the - `pdf-annotation/[work, annotation]` scope keep working (§14.2). +- `annotations.asset_id` NOT NULL (after backfill), with the composite FK + `(asset_id, work_id) → assets(id, work_id)` (§4.1). +- `annotations.work_id` is kept as a denormalized value that must equal the + Asset's Work. The composite FK enforces this, and moves keep it current by + cascade. Existing queries and the `pdf-annotation/[work, annotation]` scope + keep working (§14.2). - `canonical_annotation_set_revision` and `materialized_pdf_annotation_revision` move to the Asset. @@ -530,13 +696,20 @@ for translations. ### 6.3 Materialized bytes and "the annotated copy" -Today the managed file **is** the annotated export (§1.3). The design keeps -that behavior for the default Asset: materialization still rewrites that -Asset's bytes (copy-on-write when shared). §9.4 explains why `ingest_sha256` -exists, and §18 Q3 asks whether PRKS should start keeping the pristine -original as a separate Asset. Either way, the annotated export and the -original are different **bytes** of one Asset lineage, and their annotations -are the same canonical metadata rows. +Today the managed file **is** the annotated export (§1.3). In the new model, +PRKS's materialization is **derived storage of the same Asset**, not a peer +Asset (D3): + +- the annotations are the canonical rows on that Asset; +- the materialized PDF is those annotations rendered into the Asset's working + bytes (`storage_locator`); +- the pristine bytes, when kept, sit in the same Asset's `source_locator`. + +The user sees one File, with or without its original preserved. Keeping +pristine originals is permitted by this model but is not required by any slice +(§9.4). A copy that the user annotated in **another** tool and brought in is a +different matter: it is a genuinely separate file, and therefore a separate +Asset. ### 6.4 Research Notes belong to the Work @@ -555,18 +728,18 @@ their own target: a Manifestation or Asset**. A bare page number inside a Work-level note is ambiguous and must not be introduced. -Per-version notes ("this translation mistranslates §4") are not supported by -this design. §18 Q5 asks whether they are needed. If they are, the answer is a -separate, small Manifestation-level note field, not a change of scope for the -Research Note. +Per-Version notes ("this translation mistranslates §4") are out of scope for +now (D7). If they are ever added, they will be a separate small +Manifestation-level field, never a change of scope for the Research Note. ### 6.5 Private Notes belong to the Work (with a stated tradeoff) Private Notes are reminders ("chapter 3 is relevant to X", "ask Y about this"). Most of them concern the work, so they stay on the Work, unchanged. **Tradeoff:** some reminders are file facts ("this scan is missing pp. 40–45"). -The right home for those is Asset state (a short Asset `note` or condition -flag), not moving Private Notes. The notes deliberately do not share a scope: +If PRKS ever needs to record those, they belong in Asset state (for +example a condition flag), not in moved Private Notes. Nothing like that is +planned now (D7). The notes deliberately do not share a scope: Research and Private Notes stay separate conflict units on the Work, as they are today. @@ -582,10 +755,10 @@ this edition**. The default scope for each current role type: | Role type | Default scope | Reasoning | | --- | --- | --- | | Author | **Work** | The creator of the intellectual work, the same for every translation. | -| Editor | **Manifestation** | Usually edition-specific: a critical edition's editor, or a volume editor of one printing. **Ambiguous for anthologies**, where the editor is the creator of the compiled work. The user can move it to Work scope. | +| Editor | **Manifestation** (D6) | Usually edition-specific: a critical edition's editor, or a volume editor of one printing. **Ambiguous for anthologies**, where the editor is the creator of the compiled work. Each row's scope can be overridden, so such an Editor moves to Work scope. | | Translator | **Manifestation** | Identifies one translation. | | Introduction / Foreword / Afterword | **Manifestation** | They belong to one edition's apparatus. | -| Reviewer | **Manifestation** | A review, or a peer review, addresses a specific publication form. **Ambiguous in PRKS today.** The role's current product meaning is not documented, so §18 Q7 asks. | +| Reviewer | **Work** (D6) | The role's current product meaning in PRKS is not documented well enough to assign every existing Reviewer to one publication Version. A user may explicitly create or move a Reviewer role to Manifestation scope when that is what the relationship means. | | Mentioned | **Work** | "Person discussed in this work" is a research relationship. See §7.4 about its asset-derived origin. | These are **defaults applied at backfill and at role creation**. Scope is @@ -593,8 +766,9 @@ stored per row, so a user can override a default. ### 7.2 Storage -- `roles` gains a nullable `manifestation_id` (FK to `manifestations`, CASCADE). - NULL means Work scope. Keeping one table, rather than adding +- `roles` gains a nullable `manifestation_id` with the composite FK + `(manifestation_id, work_id) → manifestations(id, work_id)` (§4.1). A role + cannot be scoped to another Work's Version. NULL means Work scope. Keeping one table, rather than adding `manifestation_roles`, means every current `roles` reader (People pages, credit helper, Research Graph, BibTeX) keeps working with a filter instead of a UNION. @@ -610,9 +784,11 @@ stored per row, so a user can override a default. ### 7.3 Backfill rule Every existing Work has exactly one Manifestation, so scope is invisible today. -The backfill therefore sets `manifestation_id` on the edition-scoped role types -(Editor, Translator, Introduction, Foreword, Afterword, Reviewer) and leaves -Author and Mentioned at Work scope. **Nothing visible changes.** The legacy +Slice A adds the `manifestation_id` column (NULL everywhere) and the composite +FK. The **role-scope migration** (§12.3 step 4), which runs together with the +role-sync scope change, then sets `manifestation_id` on the edition-scoped role types +(Editor, Translator, Introduction, Foreword, Afterword). It leaves Author, +Mentioned and Reviewer at Work scope (D6). **Nothing visible changes.** The legacy projection (§13) includes Work-scoped roles plus the primary Manifestation's roles, which is exactly today's set. The first time a user adds a second Manifestation, Translator and Editor correctly do **not** appear on it. @@ -656,19 +832,29 @@ Manifestation inherits (title, abstract) and its Work-scoped creators. ("p. 45") is a citation target plus a locator, and it is only meaningful together with that Manifestation's pagination. -### 8.3 Preferred Manifestation +### 8.3 Primary and citation Versions + +A Work has two pointers, deliberately kept separate so that "what I read and +open" is not tied to "what I cite" (D4): -`works.primary_manifestation_id` is the version shown in lists, opened by -default, and **cited by default**. The single pointer is deliberately simple: +| Pointer | Meaning | NULL | +| --- | --- | --- | +| `works.primary_manifestation_id` | The Version normally shown, listed and opened. | Never NULL (§4.1). | +| `works.citation_manifestation_id` | The preferred Version for citing (for example, read the translation but cite the critical edition). | Means "use the primary". The backfill leaves it NULL everywhere, so citation output is unchanged. | -- Copy citation / BibTeX / CSL on a Work cites its primary Manifestation. -- An explicit "cite this version" on any Manifestation cites that one. -- Batch export resolves **each Work to exactly one Manifestation** (the primary, - unless the caller names another) and never flattens several bibliographic - records into one entry. -- If users need a separate citation default (read the translation but cite the - original), a nullable `works.citation_manifestation_id` (NULL = primary) can - be added without migration pain. §18 Q2 asks whether it is needed now. +``` +citation_target(work) = COALESCE(works.citation_manifestation_id, + works.primary_manifestation_id) +``` + +- Copy citation, BibTeX and CSL on a Work cite `citation_target(work)`. +- An explicit "cite this Version" on any Manifestation cites that one. +- Batch export resolves **each Work to exactly one Manifestation** + (`citation_target`, unless the caller names another) and never flattens + several bibliographic records into one entry. +- Both pointers must belong to the Work (§4.1). A Manifestation that either + pointer names cannot be deleted or moved away until the pointer is changed. +- No UI sets the citation pointer yet. #41 owns that. ### 8.4 Coexisting citeable Manifestations @@ -676,34 +862,109 @@ Each Manifestation produces its own complete entry. Two translations produce two entries with different titles, translators, publishers and ISBNs. Cite keys must be unique across Manifestations. The current computed key (``) already collides across Works and would collide across -versions. #41 decides between a stored `cite_key` per Manifestation (stable for -LaTeX users) and a deterministic key with disambiguation. This design reserves -a nullable `manifestations.cite_key` and nothing else. - -### 8.5 Research citations inside PRKS - -`argument_sources(argument_id, work_id, pages)` is PRKS's own citation. It -gains a nullable `manifestation_id`: - -- `pages` empty → the Argument cites the **Work**. `manifestation_id` stays - NULL and renders against the primary. -- `pages` non-empty → the pinpoint was written against one pagination, so the - backfill **pins** it to the default Manifestation, and new pinpoints are - pinned to the Manifestation the user is looking at. Changing the primary - Manifestation later never moves an existing pinpoint. - -**Row identity must change with it.** Today the primary key is -`(argument_id, work_id)`, so a nullable column alone would still stop one -Argument from citing two Manifestations of the same Work (for example, a -passage in the Greek text and the same passage in a translation). It would also -make `MERGE_WORKS` collide when an Argument already cites both the source and -the target Work. The citation identity becomes -`(argument_id, work_id, COALESCE(manifestation_id, ''))`, enforced by a unique -index. The table is rebuilt in the same migration that adds the column (SQLite -cannot alter a primary key). The `argument-sources` durable scope is the whole -source list of one Argument, so it is unaffected. On merge, if re-pointing -would produce two rows with the same identity, the preview shows both and the -user keeps one or both pinpoints. Neither is dropped silently. +Versions. Whether keys are stored, and how they are disambiguated, is owned +by #41 (D9). This design adds no cite-key column. + +### 8.5 Research citations inside PRKS: `argument_sources` identity + +`argument_sources` is PRKS's own citation. What its call sites do today: + +| Call site | What it does | +| --- | --- | +| `research_network._replace_sources_on_conn` | Replaces an Argument's **whole list**: validates, then DELETEs all rows and INSERTs them again with `order_index = position`. It refuses a repeated `work_id` ("Duplicate source Work"). | +| `argument_sync` (`current_sources`, `set_sources_on_conn`) | The durable unit is `argument-sources/`, the **whole ordered list**. The wire shape is `[{work_id, pages}]`, ordered by `order_index, work_id`. | +| `research_network._argument_sources` | Reads the list for Argument detail, joining Work title and Authors. | +| `research_graph` | Reads `(argument_id, work_id, pages)` to build Argument→Work edges. | +| migration v11→v12 | Created legacy rows with `order_index = 0`, so several rows of one Argument can share an `order_index`. | + +No call site addresses a single row. A citation is an **item in an ordered +list** that is always replaced as a whole. + +**New row model:** + +```sql +CREATE TABLE argument_sources ( + argument_id TEXT NOT NULL REFERENCES arguments(id) ON DELETE CASCADE, + order_index INTEGER NOT NULL, -- position in the list + work_id TEXT NOT NULL REFERENCES works(id) ON DELETE CASCADE, + manifestation_id TEXT, -- NULL = the Work in general + pages TEXT NOT NULL DEFAULT '', -- pinpoint / locator + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (argument_id, order_index), + FOREIGN KEY (manifestation_id, work_id) REFERENCES manifestations(id, work_id) + ON UPDATE CASCADE ON DELETE RESTRICT +); +CREATE UNIQUE INDEX idx_argument_sources_citation + ON argument_sources(argument_id, work_id, COALESCE(manifestation_id, ''), pages); +CREATE INDEX idx_argument_sources_work_id ON argument_sources(work_id); -- kept +``` + +- **The row key is the list position**, `(argument_id, order_index)`. That is + the identity the list-replacement semantics already give a row. +- **Citation identity** is `(work, Version or none, pinpoint)`, and only + **exact duplicates** are refused. This meets every requirement: + - one Argument may cite the Work in general (`manifestation_id` NULL); + - it may also, or instead, cite several Versions of that Work, each with its + own pinpoint; + - it may give several pinpoints for the same Version or Work; + - an identical repeated row carries no information. +- **Work-level rows may carry a locator.** Classical citations often use + locators that do not depend on the edition (Stephanus "514a", Bekker + numbers), so there is no rule that "pages ⇒ Version". +- The owner is enforced by the composite FK (§4.1). Deleting a Version that an + Argument pins is refused and reported as in use, never silently unpinned. + +**Rejected: a stable surrogate row ID.** Rows are rewritten on every list +replacement, so an ID would not stay stable unless the protocol were extended +to carry it, and nothing references an individual citation today. If a future +feature needs one (for example, quote anchors inside a citation), it can be +added then without changing the citation identity above. + +**Backfill (table rebuild, in the Slice A migration).** SQLite cannot change a +primary key in place, so the migration rebuilds the table: + +- It copies rows in the canonical read order `(order_index, work_id)` and + renumbers `order_index` as `0..n-1` for each Argument. This fixes the legacy + `0, 0, 0` rows **without changing the order any reader sees**, so the + `argument-sources` list value, and therefore its revision, is unchanged. +- It **pins** each row with non-empty `pages` to the Work's backfilled + Manifestation, because that pinpoint was written against that pagination. + It leaves rows with empty `pages` at Work level. A user can later unpin a + locator that does not depend on the edition. +- It cannot produce collisions, because the old key already made + `(argument_id, work_id)` unique. + +**Legacy wire compatibility.** Clients that send `[{work_id, pages}]` keep +working: + +- An entry that matches an existing row's `(work_id, pages)` keeps that row's + `manifestation_id`. +- A new entry with empty `pages` is Work-level. +- A new entry with `pages` is pinned to the Work's only Manifestation when it + has exactly one. When it has several, the entry is **refused** with + `ARGUMENT_SOURCE_VERSION_REQUIRED`, never guessed. +- A legacy list that would silently drop Version-specific rows it cannot + express is refused the same way. + +New clients send an optional `manifestation_id` on each entry. The "Duplicate +source Work" validation becomes "duplicate identical citation". + +**Merge behavior.** `MERGE_WORKS` never discards a non-identical citation: + +- Pinned rows move with their Manifestation (the owner cascades), keep their + `manifestation_id`, and cannot collide. +- Work-level rows of the source Work are re-pointed to the target. A row that + becomes **identical** to an existing row (same work, both Work-level, same + `pages`) is collapsed, keeping the earlier position. This loses nothing, + because the rows are equal. +- Any other row is kept, and positions are renumbered in list order. +- Every affected Argument's `argument-sources` revision advances, so an + offline edit based on the pre-merge list conflicts instead of overwriting + it (§14.2). +- The merge preview lists each Argument whose citations change. + +`research_graph` keeps one Argument→Work edge per `(argument, work)`, +collapsing several citations of the same Work into one edge. ### 8.6 BibTeX compatibility projection @@ -713,7 +974,7 @@ and its output. The generator is split into: ``` citation_record(manifestation_id) -> plain dict (entry type, fields, credits) bibtex_from_record(record, export_profile) -> str -/api/bibtex/ = bibtex_from_record(citation_record(primary(work_id))) +/api/bibtex/ = bibtex_from_record(citation_record(citation_target(work_id))) ``` The record builder is where #41's CSL-JSON adapter will attach. A @@ -732,18 +993,21 @@ not a #60 concern. | Field | Meaning | | --- | --- | | `id` | `AS-` + 32 hex (§11). Stable for the Asset's life. It survives merges and moves. | -| `manifestation_id` | Owner (FK). Moving an Asset to another Manifestation is an explicit operation. | -| `kind` | `managed_file` \| `external_stream` \| `external_link`. | -| `role` | `original` \| `annotated_export` \| `snapshot` \| `readable_text` \| `attachment` \| `other`. It describes what the content is, not how it is stored. | -| `storage_locator` | For `managed_file`: the managed **basename** under its storage area (`pdfs/` today). **Never an absolute path**, the same reasoning as `pending_pdf_cleanup`. The legacy `/api/pdfs/` form stays in the Work projection. | -| `provider`, `provider_id`, `url` | For `external_stream` / `external_link`. The `SET_WORK_SOURCE` aggregate moves here unchanged (I3). | +| `manifestation_id` | Owner. Moving an Asset to another Manifestation is an explicit operation. | +| `work_id` | A denormalized copy of the Manifestation's Work, kept equal by the composite FK and by cascade (§4.1). It exists so annotations can prove their owner. | +| `origin_work_id` | The legacy Work whose row this Asset was backfilled or mirrored from. It is immutable, and it maps legacy Work-keyed operations (§11.2, §14.2). NULL for Assets created by new commands. | +| `kind` | `managed_file` \| `external_stream` (D8: no `external_link`). | +| `role` | `document` \| `snapshot` \| `readable_text` \| `attachment` \| `other`. It describes what the content is for the user. PRKS's own materialized export is **not** a role (D3, §9.4). | +| `storage_locator` | For `managed_file`: the managed **basename** of the **working bytes**, the bytes PRKS serves and may rewrite (linearization, annotation materialization). **Never an absolute path**, the same reasoning as `pending_pdf_cleanup`. The legacy `/api/pdfs/` form stays in the Work projection. | +| `source_locator` | Optional managed basename of the **pristine source bytes**, exactly as ingested and never rewritten. NULL when no original is preserved, which is true of every legacy Asset. | +| `provider`, `provider_id`, `url` | For `external_stream`. The `SET_WORK_SOURCE` aggregate moves here unchanged (I3). | | `media_type` | `application/pdf`, `text/html`, `video/*` hint, … | | `byte_size` | Size of the current bytes. Filled at write time, or by the fingerprint pass. | -| `ingest_sha256` | SHA-256 of the bytes **as first ingested**, before linearization or materialization. Immutable once set. NULL for legacy Assets (§9.4). | -| `content_sha256` | SHA-256 of the **current** bytes. Updated whenever PRKS rewrites them. Serves integrity checks and client cache validation. | -| `origin` | `upload` \| `processing_import` \| `adopted` \| `web_capture` \| `materialized` \| `legacy`. | +| `ingest_sha256` | SHA-256 of the source bytes **as first ingested**, before linearization or materialization. Immutable once set. It is the content identity of the lineage. NULL for legacy Assets (§9.4). | +| `content_sha256` | SHA-256 of the **working** bytes at `storage_locator`. Updated whenever PRKS rewrites them. Serves integrity checks and client cache validation. | +| `origin` | `upload` \| `processing_import` \| `adopted` \| `web_capture` \| `legacy`. | | `origin_url`, `origin_ref` | Where the bytes came from (a download URL, or a Processing File ID). This is provenance, not a citation. | -| `derived_from_asset_id` | Lineage, for example an annotated export of a publisher PDF. | +| `derived_from_asset_id` | A user-visible relation between **separate** Assets, for example a copy annotated in another tool that the user brought in. It is never used for PRKS's own materialization. | | `supersedes_asset_id` | "This file replaces that one" within one Manifestation. | | `state` | `active` \| `trashed` (reserved for #57). Deletion removes the row (§9.2). | | `thumb_page`, `thumb_url` | Presentation (from `works`). | @@ -783,8 +1047,9 @@ files, not as DB blobs. - **Deletion keeps the cleanup model.** The reference check behind `pending_pdf_cleanup` (three-valued, re-asked under the basename lock, and settled in one transaction with the check) is extended to count - `assets.storage_locator` references in addition to `works.file_path` while - both exist. The claim still stores a basename. + `assets.storage_locator` **and** `assets.source_locator` references, in + addition to `works.file_path` while both exist. Deleting an Asset claims + both of its basenames. A claim still stores a basename. - **Replacement.** "Replace file" in the UI creates a new Asset. The old one becomes non-primary and remains, or is trashed if the user chooses. Only materialization and linearization rewrite bytes in place, and both are PRKS @@ -800,29 +1065,59 @@ writing Asset is retargeted to a fresh minted name. Shared locators are allowed but never created by new features. New ingestion always mints a name, and duplicates are detected by hash instead (§10.2). -### 9.4 Why two hashes +### 9.4 Asset identity versus source and working bytes -PRKS rewrites managed PDFs in place when it linearizes them and when it -materializes annotations (§1.3). A single "content hash" would therefore -change every time the user saves a highlight, making it useless as an -exact-duplicate key. So: +An Asset is **one lineage**: "this scan", "this publisher PDF", "this +snapshot". Its bytes can change without a new Asset being created: -- `ingest_sha256` is the identity of what was brought in. It is computed on - the incoming stream **before** linearization, and it is what "have I already - got this file?" compares against. -- `content_sha256` is the identity of what is stored now. It is used for - integrity, backup verification, and client cache validation (#52, #58). +``` +Asset AS-… (one File in the UI; owns annotations, page state, thumbnails) + ├─ source slot source_locator pristine bytes as ingested ingest_sha256 + │ (optional; never rewritten) + └─ working slot storage_locator bytes PRKS serves and rewrites content_sha256 + (linearized; annotations materialized) +``` -Legacy Assets never had their original bytes preserved, so their -`ingest_sha256` stays **NULL**. PRKS must not invent it from bytes that have -since been materialized. Exact-duplicate detection against legacy files uses -`content_sha256` and is documented as weaker: it matches an unannotated legacy -file, and misses one whose bytes PRKS has rewritten. +- **Materialization is derived storage, not a peer Asset** (D3). The + canonical annotation rows belong to the Asset. The materialized PDF is those + rows rendered into the working slot. Its freshness is still tracked by the + canonical and materialized revision pair (I6), now held on the Asset. +- **Pristine originals are permitted, not required.** When `source_locator` is + set, the working bytes can always be regenerated from it (source bytes → + linearize → materialize the current annotations). "Download original" then + returns the source slot. When it is NULL, which is true of every legacy + Asset and of any Asset ingested while preservation is off, the working + slot is the only copy, exactly as today. Whether new ingestion keeps + originals, and under what setting, is a later decision (§18). The + schema makes it possible without a new entity. +- **The Asset has two hashes because PRKS rewrites the working bytes.** + Linearization and every annotation save change them (§1.3), so their hash + cannot be a duplicate key. + - `ingest_sha256` is the lineage's content identity. It is computed on the + incoming stream before linearization and compared against for "have I + already got this file?". + - `content_sha256` identifies the current working bytes. It is used for + integrity, backup verification, and client cache validation (#52, #58). +- **Legacy Assets** never had their original bytes preserved, so their + `ingest_sha256` stays **NULL**. PRKS must not invent it from bytes that have + since been materialized. Exact-duplicate detection against legacy files uses + `content_sha256` and is documented as weaker: it matches an unannotated legacy + file, and misses one whose bytes PRKS has rewritten. +- **When a separate Asset is correct:** a different file the user brings in + (another scan, another download, a copy annotated elsewhere, a re-captured + snapshot). Such Assets have their own annotations and page state, and the + user may relate them with `derived_from_asset_id` or `supersedes_asset_id`. + +This does not need a separate "stored blob" entity. Slice A adds the +`source_locator` column (NULL for all backfilled rows) so that preserving +originals later is data, not schema, work. ### 9.5 Backup and restore - The archive layout is unchanged: the DB plus `files/pdfs/*`. Asset rows live - in the canonical DB and are covered by it. No new filesystem component, so + in the canonical DB and are covered by it. Preserved source bytes are ordinary + managed files under `pdfs/`, so they are backed up and audited like working + bytes. `audit_managed_pdfs` counts both locators. No new filesystem component, so the backup inventory needs no new classification. When Web Works add a second managed area (for example `snapshots/`), it must be classified as canonical in the inventory in the same PR. @@ -893,11 +1188,29 @@ before mutation. None of them is silent. | Operation | Effect | IDs | | --- | --- | --- | -| `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. Pending **writes** to the source are refused with `WORK_MERGED` + `target_work_id`, like `TAG_MERGED`, and are never retargeted (§14.2). | +| `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | | `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. If the old Work is left with no Manifestation, it is merged into the target or deleted, and the user chooses. | | `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." Annotations stay with the Asset. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | +**`MERGE_WORKS` transaction order.** The §4.1 constraints depend on this +order, and all steps are one transaction: + +1. Insert `work_lifecycle(source, 'merged', target)`. +2. Release the source Work's primary and citation pointers. The trigger allows + this only because step 1 exists. +3. `UPDATE manifestations SET work_id = target` for the moved Manifestations. + The owner columns of their Assets, annotations, scoped roles and pinned + argument sources follow by cascade. +4. Re-point Work-level rows: tags, Work-scoped roles, and Work-level argument + sources. Only exactly identical citation rows are collapsed (§8.5), and the + user's choices from the preview are applied. +5. Delete the source Work row, and tombstone its sync scopes. +6. Advance the revisions the merge changed, then commit. The deferred FKs are + checked at this point. + +A failure at any step rolls back the whole merge. + ### 10.5 Declined suggestions `duplicate_decisions` is canonical (it is a user decision), is backed up, and is @@ -940,15 +1253,29 @@ always pair an ID with its entity type. **Backfill IDs are deterministic**: `MF-` + `uuid5(NS_PRKS_BACKFILL, "manifestation:" + work_id)`, and `AS-` + `uuid5(…, "asset:" + work_id)`, -upper-case hex. This format matches `entity_ids.is_distributed`. Consequences: +upper-case hex. This format matches `entity_ids.is_distributed`. So the +migration is **reproducible**: restoring the same pre-migration backup twice, +or migrating two copies, yields identical IDs. + +**Legacy mapping is stored, not recomputed.** Rows created after the migration +by Slice A's mirror trigger get random IDs (`'MF-' || hex(randomblob(16))`), +because a SQL trigger cannot compute uuid5. So the mapping from a legacy +`work_id`-keyed operation to "the Manifestation or Asset it meant" is kept in +an immutable column instead: + +``` +origin_MF(W) = the manifestation WHERE origin_work_id = W (UNIQUE, immutable) +origin_AS(W) = the asset WHERE origin_work_id = W (UNIQUE, immutable) +``` + +Every backfilled or mirrored row carries its `origin_work_id`. Rows created +later by new canonical commands leave it NULL, because no legacy operation can +refer to them. The mapping survives the Work gaining more Versions, and it +survives moves: `origin_work_id` does not change when the row moves. So a +legacy operation for `W` finds `origin_MF(W)`, sees that its current `work_id` +is no longer `W`, and is refused (§14.2). -- the migration is **reproducible**: restoring the same pre-migration backup - twice, or migrating two copies, yields identical IDs; -- the mapping from a legacy `work_id`-keyed operation to "the Manifestation - it originally meant" is **computable forever**, even after the Work gains - more versions (§14.2); -- clients must still **never derive** these IDs themselves. The server returns - them. Deterministic IDs are a migration property, not a protocol. +Clients must **never derive** these IDs themselves. The server returns them. --- @@ -973,73 +1300,109 @@ upper-case hex. This format matches `entity_ids.is_distributed`. Consequences: ### 12.2 The common case ``` -before (v16) after backfill -──────────── ────────────── +before (v16) after the Slice A migration +──────────── ─────────────────────────── works W-1 works W-1 (Work-owned columns authoritative; title, status, notes, tags… legacy bibliographic/source columns still present) doi, publisher, year, … primary_manifestation_id = MF-uuid5(W-1) + source_mime (maybe set) citation_manifestation_id = NULL (= primary) file_path=/api/pdfs/x.pdf │ canonical/materialized revs ▼ -annotations(work_id=W-1) manifestations MF-uuid5(W-1) +annotations(work_id=W-1) manifestations MF-uuid5(W-1) origin_work_id = W-1 roles(W-1, Translator) work_id=W-1, kind='unspecified', title=NULL (inherit) -argument_sources(W-1, pages='45') doi, publisher, year, … (copied) +argument_sources(A, W-1, '45') doi, publisher, year, … (copied) primary_asset_id = AS-uuid5(W-1) │ ▼ - assets AS-uuid5(W-1) - kind='managed_file', storage_locator='x.pdf', - media_type='application/pdf', origin='legacy', - ingest_sha256=NULL, content_sha256=NULL (pass fills), + assets AS-uuid5(W-1) origin_work_id = W-1 + work_id=W-1, kind='managed_file', role='document', + storage_locator='x.pdf', source_locator=NULL, + media_type=COALESCE(source_mime, 'application/pdf'), + origin='legacy', ingest_sha256=NULL, + content_sha256=NULL (fingerprint pass fills), materialization revisions (copied), thumb_page (copied) - annotations.asset_id = AS-uuid5(W-1) - roles(W-1, Translator).manifestation_id = MF-uuid5(W-1) - argument_sources(W-1,'45').manifestation_id = MF-uuid5(W-1) + annotations(…, asset_id = AS-uuid5(W-1)) rebuilt + roles(…, manifestation_id = NULL) rebuilt; + scope set later by the role-scope migration + argument_sources(A, 0, W-1, MF-uuid5(W-1), '45') rebuilt ``` The backfill rules: - A **video** Work gets an `external_stream` Asset carrying `provider`, `provider_id`, `source_url` and `thumb_url`. -- A Work with **no file and no video** gets a Manifestation and **no** Asset. +- A Work with **no file, no video and no annotations** gets a Manifestation and + **no** Asset. +- A Work that has **annotations but no usable file reference** (a legacy + inconsistency) still gets a `managed_file` Asset, with a NULL + `storage_locator`, which reads as missing. Its annotations are never dropped + or left without an owner. - A non-video `source_url` goes to `manifestations.url`. -- `assets.media_type` takes the stored `source_mime` when one is present, and - is inferred only when it is absent. +- **MIME type:** if `works.source_mime` is non-empty, `assets.media_type` gets + **that exact value**. `application/pdf` is inferred only when it is absent + and the Asset is a managed PDF. Otherwise `media_type` is NULL. Existing MIME + metadata is never discarded or overwritten. - A **legacy inferred-video row** (kind NULL, no file, a URL) is classified with `effective_source_kind()`, the same rule every reader uses. When the URL cannot be parsed, the row gets no Asset and keeps its URL on the Manifestation. Nothing is guessed (see work-source-identity.md §10). +- `argument_sources` rows are renumbered and pinned as described in §8.5. ### 12.3 Order of migrations The versions below are illustrative. Each is one migration in one PR (see §17). -1. **vN: create the entities.** Create `manifestations`, `assets`, - `manifestation_relations`, and `manifestation_identifiers`. Add - `works.primary_manifestation_id`, `annotations.asset_id`, - `roles.manifestation_id`, and `argument_sources.manifestation_id` (a table - rebuild that widens its key, §8.5). Backfill - deterministically, then validate (every Work has exactly one primary - Manifestation that belongs to it, and so on). Install **mirror triggers** - so that `works` (still authoritative for everything) keeps the default - Manifestation and Asset current on INSERT/UPDATE, and the child FKs - cascade on DELETE. **No reader changes.** +1. **vN (Slice A): create the entities and the integrity layer.** In one + transaction: + 1. **Preflight.** Classify rows the backfill cannot map. Some legacy DBs + contain rows that already violate their existing `work_id` FK, such as + annotations or roles whose Work no longer exists; backup already warns + about these as `fk_violations`. Such rows are **carried over unchanged + with NULL new columns**: no owner is invented for them, and they do not + block the upgrade. A NULL child column exempts them from the composite + FKs, so the existing violation is neither repaired nor worsened. Only + their count is logged, which is privacy-safe. Anything else that cannot + be mapped **aborts** the migration with a precise reason code, and the + transaction rolls back. + 2. Create `manifestations`, `assets`, `manifestation_relations` and + `manifestation_identifiers`, with the parent pairs and composite FKs of + §4.1. + 3. Add `works.primary_manifestation_id` and `works.citation_manifestation_id` + with `ALTER TABLE ADD COLUMN`. + 4. Rebuild the leaf tables `annotations`, `roles` and `argument_sources` with + their new columns and composite FKs (create new, copy, drop old, rename). + Their indexes are recreated, and `argument_sources` gets its new key + (§8.5). + 5. Backfill deterministically (§12.2). + 6. Install the §4.1 triggers and the **mirror triggers**. `works` stays + authoritative for everything, and on INSERT/UPDATE the mirror triggers + keep `origin_MF(W)` and `origin_AS(W)` current. They also fill + `annotations.asset_id` and pin `argument_sources` rows with pages when + legacy code writes them without the new columns, reproducing the + backfill rules. The child FKs cascade on DELETE. + 7. Run the integrity query, and compare `PRAGMA foreign_key_check` with the + preflight: **no new** violation may appear. Either check failing aborts + the migration. + + **No reader changes.** No filesystem work. D2 applies: no automatic + full-library backup is made or required. 2. **vN+1: Asset authority.** Asset-owned state (locator, source aggregate, materialization revisions, thumbnail page, annotation ownership) becomes - authoritative on `assets`. The migration drops those mirror triggers and - installs the reverse projection (`assets` → `works.file_path` and the other - legacy columns) for readers not yet migrated. Only one direction exists at - any version. The text index moves to `asset_id` keys, which is derived and - rebuilt, not migrated. + authoritative on `assets`, and `annotations.asset_id` becomes NOT NULL. The + migration drops those mirror triggers and installs the reverse projection + (`assets` → `works.file_path` and the other legacy columns) for readers not + yet migrated. Only one direction exists at any version. The text index moves + to `asset_id` keys, which is derived and rebuilt, not migrated. 3. **vN+2: Manifestation authority.** The same move for bibliographic fields. - The migration **copies revision counters** from `work-field/[work, f]` to - `manifestation-field/[MF-uuid5(work), f]` (§14.2). -4. **vN+3: Roles scope enforced.** Swap the uniqueness index to include - `manifestation_id`, and set role scope by role type (§7.3). -5. **Later: retire legacy columns.** Drop them only after every reader uses the - projection module. Because SQLite `DROP COLUMN` has restrictions (FTS - triggers, indexes), this may require a table rebuild. It is optional, and the - columns can simply stay as the maintained projection. + The migration **copies revision counters** from `work-field/[W, f]` to + `manifestation-field/[origin_MF(W), f]` (§14.2). +4. **vN+3: Role scope.** Set role scope by role type (§7.3), swap the + uniqueness index to include `manifestation_id`, and copy role revisions to + `manifestation-person-role` scopes for the rows that became + Manifestation-scoped. +5. **Later: retire legacy columns.** This is decided later (D10). Until then the + columns stay as the maintained projection. ### 12.4 Fingerprint pass (not a migration) @@ -1053,12 +1416,20 @@ they are re-verifiable, so a restore can recompute them. ### 12.5 Rollback and recovery +- **The upgrade is fail-safe by transaction, not by backup (D2).** Each + migration runs in one `BEGIN IMMEDIATE` transaction + (`_run_in_transaction`). A failed preflight, constraint, integrity query or + crash rolls back to the previous version with the library untouched. Slice A + performs no filesystem mutation, so there is nothing outside SQLite to undo. +- **No automatic full-library backup.** Slice A does not create or require a + `.prks-backup`. Packing a large PDF library at startup could double storage + use and fail under storage pressure. An automatic upgrade snapshot that + copies only the DB, proportional to the change, may be evaluated separately. + It is not part of this design. - PRKS has **no down migrations**, and this design does not add any. A newer - schema is refused by older code and by restore (`schema_newer`). So the - recovery path after an upgrade is the **pre-upgrade backup**. Slice A should - make sure the upgrade flow recommends or creates one (§18 Q9). -- Each migration is transactional (`_run_in_transaction`). A failure leaves the - previous version intact, and a crash mid-migration rolls back. + schema is refused by older code and by restore (`schema_newer`). Undoing a + *committed* upgrade therefore means restoring a backup the user already + made, the same as for every existing migration. - Additive steps are **forward-fixable**: a bad backfill can be corrected by a later migration, because the legacy columns were not removed. - Mirror and reverse triggers are validated by `validate_current_schema`. A @@ -1107,9 +1478,10 @@ for the fixture library. The legacy dict gains only **additive** fields: | … with a Manifestation-owned field | the **primary** Manifestation. The editor shows the primary, so this is what the user sees. The response names the `manifestation_id` it wrote. | | … with a source field | the primary Asset's source aggregate (`SET_WORK_SOURCE` semantics unchanged) | | `POST /api/works` | creates Work + Manifestation + Asset in one transaction | -| `POST /api/works/:id/pdf` (materialization) | the **primary Asset**. The client must send `asset_id` once more than one exists. Without it the request is refused (`ASSET_AMBIGUOUS`) rather than guessed. | +| `POST /api/works/:id/pdf` (materialization) | the **working slot** of the primary Asset (§9.4). The client must send `asset_id` once more than one exists. Without it the request is refused (`ASSET_AMBIGUOUS`) rather than guessed. | | Annotation endpoints | the annotation's own Asset. A new annotation needs the displayed `asset_id`, which defaults to the primary only while exactly one PDF Asset exists. | -| `/api/bibtex/:id` | `citation_record(primary)`, §8.6 | +| `/api/bibtex/:id` | `citation_record(citation_target(work))`, which is the primary while `citation_manifestation_id` is NULL (§8.3, §8.6) | +| Argument sources `[{work_id, pages}]` (HTTP and `argument-sources` sync) | kept pins are preserved. Otherwise a new entry is pinned to the only Manifestation, or refused with `ARGUMENT_SOURCE_VERSION_REQUIRED` when the Work has several (§8.5) | ### 13.3 New API surface (typed, #185 pattern) @@ -1141,8 +1513,10 @@ Nothing here is implemented in this PR. than one choice exists; - duplicate warnings appear at the moment of import; - #61's Secondary Reading View offers "open another version". -- UI words are "Version" (Manifestation) and "File" (Asset). "Manifestation", - "Asset" and "FRBR" stay out of the UI. +- UI words (D1): a Manifestation is a **Version**. An Asset is a **File** when + it is a managed file, and a **Source** or stream when it is external (for + example "YouTube source"). PRKS's own materialized PDF is never shown as a + separate File (D3). "Manifestation", "Asset" and "FRBR" stay out of the UI. --- @@ -1172,20 +1546,29 @@ and are kept. | Current scope | Future scope | Migration | | --- | --- | --- | -| `work-field/[W, f]` for M-owned `f` | `manifestation-field/[MF, f]` | Copy the revision to `MF-uuid5(W)` in the authority migration. | +| `work-field/[W, f]` for M-owned `f` | `manifestation-field/[MF, f]` | Copy the revision to `origin_MF(W)` in the authority migration. | | `work-field/[W, f]` for W-owned `f` (status, title, abstract, author_text) | unchanged | — | -| `work-source/W` | `asset-source/AS` | Copy to `AS-uuid5(W)`. | -| `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `AS-uuid5(W)`. Deletion tombstones are copied too, so resurrection protection survives. | +| `work-source/W` | `asset-source/AS` | Copy to `origin_AS(W)`. | +| `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `origin_AS(W)`. Deletion tombstones are copied too, so resurrection protection survives. | | `work-person-role/[W, P, r]` | unchanged for W-scope; `manifestation-person-role/[MF, P, r]` for M-scope | Copy for backfilled M-scoped roles. | | notes, tags, folder, playlist, open | unchanged | — | **Legacy operations after the move.** A pending `SET_WORK_METADATA_FIELD(W, doi, base=7)` enqueued before the upgrade is applied to -**`MF-uuid5(W)`**, the Manifestation the user was editing when they enqueued -it, **not to "whatever is primary now"**. If that Manifestation has since -moved to another Work or been deleted, the operation is refused with an -explicit conflict code. It is never retargeted. The same rule applies to -annotation operations (→ `AS-uuid5(W)`) and to `SET_WORK_SOURCE`. +**`origin_MF(W)`**, the Manifestation the user was editing when they enqueued +it, **not to "whatever is primary now"**. The mapping is looked up through +the stored `origin_work_id` (§11.2), never recomputed. The operation is never +retargeted: + +| State of `W` / `origin_MF(W)` at apply time | Result | +| --- | --- | +| `W` live, `origin_MF(W).work_id = W` | applied with the carried revision | +| `W` merged away | `409 WORK_MERGED` + `target_work_id` (below) | +| `origin_MF(W)` moved to another Work outside a merge | `409 MANIFESTATION_MOVED` + the current `work_id` | +| `origin_MF(W)` deleted | the family's existing deleted/absent result | + +The same rule applies to annotation operations (→ `origin_AS(W)`) and to +`SET_WORK_SOURCE`. **New durable families** (Manifestation fields, M-scoped roles, set-primary, relations) each declare their shape per "Adding a family" in @@ -1203,25 +1586,36 @@ once as a fallback. belongs to a scope keyed by the **source** Work. It says nothing about the target Work's scope, so retargeting the operation would either accept a stale write when the counters happen to match, or report a false conflict when they -do not. The rule therefore follows what `TAG_MERGED` already does: - -- A pending write to Work-owned state (notes, status, tags, folder, playlist, - Work-scoped roles, Work fields) that names a merged `W-…` is **refused** - with `409 WORK_MERGED` and `target_work_id`. It is never retargeted. The - client keeps the user's intended value and presents it as an explicit merge - conflict against the target ("apply to the merged Work?"). If the user - accepts, the client sends a **new** operation based on the target's current - revision. -- The merge itself does not rebase or combine revision counters. Source scopes - are tombstoned, so a stale device cannot resurrect them, and target scopes - advance normally for whatever the merge changed. -- `MARK_WORK_OPENED` is a max-register with no base revision (I7), so it may be - applied to the target without conflict. -- Operations on the source Work's Manifestations and Assets are unaffected. - Their scopes are keyed by `MF-…`/`AS-…` IDs, which survive the merge, so their - base revisions are still meaningful. +do not. `sync_tag_lifecycle` already follows this principle: `TAG_MERGED` is a +refusal, not a redirect. So: + +- **Every pending operation that names a merged-away `W-…` is refused** with + `409 WORK_MERGED` and `target_work_id`. The server never applies it to the + target, whatever the operation is. This covers Work fields, notes, status, + tags, folder, playlist, Work-scoped roles, argument sources naming the + Work, and legacy Work-keyed operations mapped through `origin_MF(W)` or + `origin_AS(W)`. +- **The client reconciles explicitly.** It keeps the user's intended value and + presents a merge conflict ("this Work was merged into …; apply your change + there?"). If the user accepts, the client sends a **new** operation against + the target's current revision. The one operation that needs no question is + `MARK_WORK_OPENED`, which is a max-register with no base revision. The client + may re-issue it for the target automatically, and that is still a new, + explicit operation, not a server-side redirect. +- **Operations that name a moved Manifestation or Asset by its own ID continue + normally.** Their scopes are keyed by `MF-…`/`AS-…`, the IDs are unchanged by + the merge, and so their base revisions are still meaningful. +- **The merge does not rebase or combine revision counters.** Source Work + scopes are tombstoned, so a stale device cannot resurrect them. Target scopes + advance for whatever the merge changed, including the `argument-sources` + revision of every Argument whose list changed (§8.5), so older edits + conflict instead of overwriting. +- **Any future rebase of pending Work mutations during a merge** must be an + explicit merge decision with defined revision reconciliation, shown in the + merge preview. It is never implicit redirect behavior, and it is not part of + this design. - **Reads** (links, tabs, `[[W-…]]`, cached detail requests) follow the redirect - to the target. + to the target. A read cannot accept stale state. --- @@ -1232,7 +1626,7 @@ its own identity or asset model. | Issue | Implications | | --- | --- | -| **#41 Citation V2** | Cite a **Manifestation**. Default to the Work's primary. Batch export resolves each Work to exactly one Manifestation. Build CSL-JSON from `citation_record(manifestation_id)` (§8.6), not from `works` columns. Decide cite-key persistence (§8.4) and the `urldate` quirk. Do not add citation fields to `works`. | +| **#41 Citation V2** | Cite a **Manifestation**. Default to `citation_target(work)` = `COALESCE(citation_manifestation_id, primary_manifestation_id)`, and own the UI for choosing the citation Version (D4). Batch export resolves each Work to exactly one Manifestation. Build CSL-JSON from `citation_record(manifestation_id)` (§8.6), not from `works` columns. Decide cite-key persistence (§8.4) and the `urldate` quirk. Do not add citation fields to `works`. | | **#42 Ingestion / Web Works** | Every ingest creates, or attaches to, Work → Manifestation → Asset through one canonical command. Web Work = Manifestation(`web_page`) + snapshot Asset(s) under a new managed area, classified as canonical in the backup inventory. Duplicate warnings use §10 (hash, then identifiers, then optional RapidFuzz ranking). Provenance attaches to Manifestation fields and Asset origin. BibTeX/RIS import creates Manifestations and uses identifier normalization. | | **#57 History / Trash** | Trash granularity is Work, Manifestation, or Asset. Asset `state = trashed` keeps bytes until purge, and purge uses the `pending_pdf_cleanup` rules. Merge and move are history events with previews. `work_lifecycle` redirects are the durable record of merges. | | **#58 Offline UX** | "Available offline" is a property of **Assets** (bytes) plus the Work's metadata. Default to the primary Asset. Validate cached bytes with `content_sha256`. Pending edits stay keyed to the entity they were made on (§14.2). | @@ -1261,8 +1655,8 @@ row. 2. **Give the backfilled Manifestation the same ID as its Work.** This would make legacy scopes trivially compatible. **Rejected:** once a Manifestation moves to another Work (merge), `W-123` would name a Work in one table and a - Manifestation of a different Work in another. Deterministic uuid5 IDs give - the same compatibility without the ambiguity. + Manifestation of a different Work in another. The stored, immutable + `origin_work_id` (§11.2) gives the same compatibility without the ambiguity. 3. **A mandatory FRBR/LRM four-level model (Work → Expression → Manifestation → Item).** "Expression" (a specific translation or text) is real, but for PRKS's use cases it is carried by `language` + `translation_of` + @@ -1298,6 +1692,24 @@ row. in the same transaction (§12.1). 12. **An ORM or a new sync framework for the new entities.** Out of scope per #179 and this issue's boundaries. +13. **PRKS's materialized annotated PDF as its own peer Asset.** This would + double every annotated file in the UI and split one lineage's annotations, + page state and duplicate identity across two Assets. **Rejected (D3).** + Materialization is derived storage inside the Asset (§9.4). +14. **A stable surrogate ID per `argument_sources` row.** Every list + replacement rewrites the rows, and nothing addresses a single citation. + **Rejected for now.** The row key is the list position, and citation + identity is `(work, Version, pinpoint)` (§8.5). +15. **Composite ownership FKs on `works` via a table rebuild.** `works` is the + parent of about ten tables. Migrations run in one transaction with foreign + keys on, so dropping the old table would cascade-delete child rows. + **Rejected** in favor of narrow pointer triggers (§4.1). The leaf tables + do get composite FKs. +16. **Application-only ownership validation.** A long-lived compatibility + layer needs the database as the final guard. **Rejected (D11-D).** +17. **An automatic full-library backup before migrating.** It can double + storage use and fail under storage pressure, and a transactional, + SQLite-only, additive migration does not need it. **Rejected (D2).** --- @@ -1316,17 +1728,17 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1). `db_schema.sql`, schema validation, fresh and upgraded tests, and a parity test (legacy columns = the new rows) over fixture libraries, including video, inferred-video, no-file and shared-basename rows. No readers. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests; (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins pages rows; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename and legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | -| **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `AS-uuid5(W)`. Browser last-page key migration. | No | C | -| **E. Manifestation authority** | vN+2: bibliographic fields owned by `manifestations`. `manifestation-field` scopes with carried revisions. `SET_WORK_METADATA_FIELD` mapped to `MF-uuid5(W)`. `argument_sources.manifestation_id` pinning enforced for new pinpoints. Identifier table populated from DOI/ISBN. | No | C | +| **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | +| **E. Manifestation authority** | vN+2: bibliographic fields owned by `manifestations`. `manifestation-field` scopes with carried revisions. `SET_WORK_METADATA_FIELD` mapped to `origin_MF(W)`. The `argument_sources` writer and sync accept `manifestation_id`, replacing the mirror trigger, and return `ARGUMENT_SOURCE_VERSION_REQUIRED` (§8.5). Identifier table populated from DOI/ISBN. | No | C | | **F. Role scope** | vN+3: role-type scope defaults, the uniqueness swap, `credits(M)`, and the role-sync scope for M-scoped roles. | No | E | -| **G. Typed Version/File API** | §13.3 read endpoints first, then mutations as canonical domain commands (#182/#199 pattern) with OpenAPI and openapi-core tests. New durable families declared per the sync guide. | API only | D, E, F | +| **G. Typed Version/File/Source API** | §13.3 read endpoints first, then mutations as canonical domain commands (#182/#199 pattern) with OpenAPI and openapi-core tests. New durable families declared per the sync guide. | API only | D, E, F | | **H. Versions UI** | "Add another version / file", set primary, open a specific Version or File, and a version picker for Copy citation, all behind §13.4's progressive disclosure. Follow `DESIGN.md` for Work detail composition. #61's secondary view builds on this. | **Yes** | G | | **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256` (and `content_sha256` for legacy files). Offers the four choices in §10.2. | Yes | B, H | | **J. Identifier candidates + decline** | Normalized DOI/ISBN/arXiv candidates, a review list, and `duplicate_decisions`. | Yes | E, I | -| **K. Merge / move workflow** | `MERGE_WORKS`, `MOVE_MANIFESTATION`, `MOVE_ASSET` with previews, and `work_lifecycle` redirects. Coordinate with #57. | Yes | J | +| **K. Merge / move workflow** | `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | | **L. RapidFuzz evaluation** | A separate research/evaluation issue (dependency review, ranking quality on synthetic data). It only ranks; it never decides. | — | J | A, B and C can proceed in parallel after A's migration merges. D and E are @@ -1335,51 +1747,28 @@ the affected sync feature E2E groups before merge, and the full E2E gate once. --- -## 18. Open questions for the maintainer - -Each question has a recommendation. Nothing in slice A depends on the answers -except Q1 and Q9. - -1. **Naming.** Internal `manifestation` / `asset`, UI "Version" / "File". - *Recommendation:* approve as proposed. Q1 blocks slice A, because the table - names become permanent. -2. **Separate citation default.** Is `works.primary_manifestation_id` enough, - or is `citation_manifestation_id` needed from the start? *Recommendation:* - primary only. Add the second pointer when #41 shows a need. -3. **Keep pristine originals?** Today materialization overwrites the ingested - PDF. Should new ingestion keep the original as its own Asset - (`role = original`) and materialize into a derived `annotated_export` - Asset? That costs up to twice the storage, but gives the user back a - "publisher PDF + my annotated copy" pair and a stable duplicate key. - *Recommendation:* yes for new ingestion, behind a setting. Legacy files stay - as they are. -4. **Abstract ownership.** Work-level with a Manifestation override (proposed), - or Manifestation-only? *Recommendation:* as proposed, which keeps FTS - unchanged. -5. **Per-version notes.** Are Manifestation- or Asset-level notes wanted - ("this translation is unreliable", "scan missing pp. 40–45")? - *Recommendation:* not now. If needed, add a small Version note field. Never - move Research or Private Notes. -6. **Per-version reading state.** Should `status` or "last opened" also exist per - Manifestation or Asset? *Recommendation:* Work-level `status` only. Add an - Asset-level `last_opened_at` in slice H so "open" returns to the last-used - file. -7. **Reviewer and Editor semantics.** What does the `Reviewer` role mean in - PRKS today, and should Editor default to Manifestation scope? - *Recommendation:* both default to Manifestation scope. Revisit if anthologies - are common in real libraries. -8. **`external_link` Assets.** Is a non-managed "link to the publisher page" - Asset useful, or is `manifestations.url` enough? *Recommendation:* the - `url` field is enough. Keep the `external_link` kind reserved but - unimplemented. -9. **Upgrade safety net.** Should the first #60 migration refuse to run, or - force an automatic backup, when no recent backup exists? *Recommendation:* - create an automatic pre-migration backup into the existing backup location - before running vN, and log only counts. This needs your call because it - writes to storage at startup. -10. **Cite keys.** Stored per Manifestation, or computed with disambiguation? - This is owned by #41, and the design only reserves the column. -11. **Retiring legacy `works` columns.** Keep them indefinitely as the - maintained projection, or plan a table rebuild once every reader has moved? - *Recommendation:* keep them until Android and MCP ship on the typed API, - then decide. +## 18. Remaining open questions + +The maintainer's earlier questions are answered in §0 (D1–D10). What remains +is listed below. **None of it blocks approving the design or starting Slice A.** +Items 1–3 are choices this revision made that should be confirmed during +review. + +1. **Work-level citations may carry a locator** (§8.5). This allows + edition-independent locators such as Stephanus or Bekker numbers, which is + why citation identity includes `pages`. The alternative, "pages always pins + a Version", is simpler, but it would force classical locators onto one + edition. +2. **Legacy FK-orphan rows are carried, not blocking** (§12.3 step 1.1). Rows + that already violate their `work_id` FK keep NULL new columns and are + counted. The alternative is to make the upgrade refuse to run on such a DB. +3. **Slice A rebuilds `roles` and `annotations`** as well as + `argument_sources`, so the composite FKs exist from the first migration. The + alternative is to add the columns without FKs in Slice A and rebuild later, + which leaves a window without DB enforcement. The recommendation is to + rebuild in Slice A as specified. +4. **The policy for preserving pristine originals.** The schema allows it + (`source_locator`, §9.4). When new ingestion should keep originals, and + behind what setting, is decided later. It does not affect Slice A. +5. **A proportional upgrade snapshot that covers only the DB** (D2). This may be + evaluated separately. It is not part of any slice here. From a7d40058f48155cf6ae2db33be048708ff8e2d10 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:35:47 +0000 Subject: [PATCH 04/24] docs(#60): rename merge table to sync_work_lifecycle; move re-keying, source routing, crash-safe hashes Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 40 ++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 70ba0348..ac8b0e6b 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -403,8 +403,8 @@ enough (D8). | `manifestation_relations(from_id, to_id, relation)` | Typed links between Manifestations of the **same** Work: `revision_of`, `published_version_of`, `new_edition_of`, `translation_of`, `reprint_of`. These carry "arXiv v2 revises v1", "journal version of the accepted manuscript", and "Serbian translation of the Greek text". They carry no page correspondence (#61 owns assisted correspondence). | | `manifestation_identifiers(manifestation_id, scheme, value, normalized)` | DOI, ISBN, arXiv ID (versioned and versionless), PMID, and similar. `works.doi`/`isbn` backfill into these later. This is the entity duplicate detection keys on. | | `roles` gaining a nullable `manifestation_id` | Edition-scoped credit (§7). The owner is enforced by a composite FK (§4.1). | -| `argument_sources` rebuilt with key `(argument_id, order_index)` plus a nullable `manifestation_id` | One Argument can cite the Work in general **and** several Versions of it, each with its own pinpoint (§8.5). | -| `work_lifecycle(work_id, state, target_work_id)` | Merge record, modeled like `sync_tag_lifecycle`: it redirects **reads** and refuses **every pending operation** naming the merged Work with `WORK_MERGED` (§10.4, §14.2). | +| `argument_sources` rebuilt with `PRIMARY KEY (argument_id, order_index)`, a nullable `manifestation_id`, and citation identity `UNIQUE (argument_id, work_id, COALESCE(manifestation_id, ''), pages)` | One Argument can cite the Work in general **and** several Versions of it, each with its own pinpoint (§8.5). | +| `sync_work_lifecycle(work_id, state, target_work_id)` | Merge record, named and modeled like `sync_tag_lifecycle`. It is distinct from the existing `work_lifecycle_sync` module (`CREATE_WORK`/`DELETE_WORK`). The table it redirects **reads** and refuses **every pending operation** naming the merged Work with `WORK_MERGED` (§10.4, §14.2). | | `duplicate_decisions` | Durable record of "not a duplicate" answers, so a declined suggestion does not return (§10.5). | | Metadata provenance | A per-field record of detected, retrieved or user-edited origin on Manifestation fields (#42). This design only reserves the ownership: provenance belongs to the Manifestation field it describes, never to the Work. | @@ -524,7 +524,7 @@ The existing single-column FKs (`roles.work_id → works`, | Trigger | Rule | | --- | --- | -| `works_manifestation_pointers_owned` (BEFORE UPDATE OF `primary_manifestation_id`, `citation_manifestation_id` ON `works`) | A non-NULL pointer must name a Manifestation whose `work_id` is this Work (`MANIFESTATION_OWNER_MISMATCH`). The primary may not be set back to NULL (`WORK_PRIMARY_MANIFESTATION_REQUIRED`). The one exception is a Work already recorded in `work_lifecycle` as `merged`, whose row the same merge transaction is about to delete (§10.4). | +| `works_manifestation_pointers_owned` (BEFORE UPDATE OF `primary_manifestation_id`, `citation_manifestation_id` ON `works`) | A non-NULL pointer must name a Manifestation whose `work_id` is this Work (`MANIFESTATION_OWNER_MISMATCH`). The primary may not be set back to NULL (`WORK_PRIMARY_MANIFESTATION_REQUIRED`). The one exception is a Work already recorded in `sync_work_lifecycle` as `merged`, whose row the same merge transaction is about to delete (§10.4). | | `works_manifestation_pointers_insert` (BEFORE INSERT ON `works`) | A Work is inserted with NULL pointers. Its Manifestation cannot exist before the Work, because `manifestations.work_id` references it. | | `manifestations_pointer_target_move` (BEFORE UPDATE OF `work_id` ON `manifestations`) | A Manifestation that its Work names as primary or citation cannot be moved away (`MANIFESTATION_IS_POINTER_TARGET`). The pointer must change first, in the same transaction. | | `manifestations_pointer_target_delete` (BEFORE DELETE ON `manifestations`) | The same rule for deletion. A whole-Work delete is unaffected: the Work row is gone before its Manifestations cascade. | @@ -684,7 +684,7 @@ coordinate space is identical. Anything else is an explicit, reviewed action. | Replace with a **newer revision** (arXiv v2) | A **new Manifestation** (`revision_of`) with a new Asset. The old Asset and all its annotations remain, still readable. v2 may become primary. | The pagination changed. Moving coordinates would silently point highlights at the wrong text. | | Preprint → publisher version | The same: a new M (`published_version_of`) and a new A. The preprint annotations stay on the preprint. | The same reason. The PDFs are completely different. | | **Another scan** of the same edition | A new Asset in the **same** Manifestation. Annotations stay on the scan they were drawn on. | Same edition, but the scans differ in page offsets, cropping and rotation. | -| **Merging** duplicate Works | Assets move **whole** with their Manifestation, and their annotations go with them unchanged. | An annotation's Asset never changes, so its coordinates stay valid. | +| **Merging** duplicate Works | Assets move **whole** with their Manifestation. Each annotation keeps its ID, content, coordinates and `asset_id`. Only its denormalized `work_id` changes, rewritten by `ON UPDATE CASCADE` from the Manifestation move (§4.1). | An annotation's Asset never changes, so its coordinates stay valid, and the database keeps the owner columns consistent. | | Merging two Assets that are **exact duplicates** (same `ingest_sha256`) | The user chooses: keep one Asset and **union** the annotation sets (the coordinates are provably identical), or keep both. | Content identity proves the coordinates are compatible. It is still never automatic. | | Replacing a file **in place** (a "correct scan" in the same M) | A new Asset that `supersedes` the old one. The old Asset's annotations stay on it, and the UI offers "copy annotations to the new file" only as an assisted action (below). | | @@ -1011,7 +1011,7 @@ not a #60 concern. | `supersedes_asset_id` | "This file replaces that one" within one Manifestation. | | `state` | `active` \| `trashed` (reserved for #57). Deletion removes the row (§9.2). | | `thumb_page`, `thumb_url` | Presentation (from `works`). | -| `canonical_annotation_set_revision`, `materialized_annotation_revision` | From `works` (§6.1). | +| `canonical_annotation_set_revision`, `materialized_pdf_annotation_revision` | Copied from `works` **under the same column names** (§6.1), so there is no third spelling. | | `captured_at` | For snapshots (Web Works). | | `created_at`, `updated_at` | | @@ -1098,6 +1098,21 @@ Asset AS-… (one File in the UI; owns annotations, page state, thumbnails) already got this file?". - `content_sha256` identifies the current working bytes. It is used for integrity, backup verification, and client cache validation (#52, #58). +- **Hash updates are crash-safe.** Replacing bytes on disk and updating the + hash in SQLite cannot be one atomic step, so every in-place rewrite + (materialization, linearization, copy-on-write retarget) follows a fixed + order: + 1. commit `content_sha256 = NULL`, `byte_size = NULL` and + `fingerprinted_at = NULL` (the working bytes are **pending**); + 2. replace the bytes durably (`fs_durability`); + 3. hash what was written and commit the new values. + + A crash anywhere in between leaves NULL, meaning "unknown", never a stale + hash that clients or backup would trust. The fingerprint pass (§12.4) treats + NULL as work to do and fills it on the next start. Backup verification and + client cache validation skip, and never trust, an Asset whose hash is NULL. + `ingest_sha256` is written once, before the first working copy exists, and + it never changes. - **Legacy Assets** never had their original bytes preserved, so their `ingest_sha256` stays **NULL**. PRKS must not invent it from bytes that have since been materialized. Exact-duplicate detection against legacy files uses @@ -1188,15 +1203,15 @@ before mutation. None of them is silent. | Operation | Effect | IDs | | --- | --- | --- | -| `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | -| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. If the old Work is left with no Manifestation, it is merged into the target or deleted, and the user chooses. | -| `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." Annotations stay with the Asset. | `AS-…` unchanged. | +| `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `sync_work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | +| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work, the pointer must move first. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. If the old Work is left with no Manifestation, it is merged into the target or deleted, and the user chooses. | +| `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the pointer must move first. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | **`MERGE_WORKS` transaction order.** The §4.1 constraints depend on this order, and all steps are one transaction: -1. Insert `work_lifecycle(source, 'merged', target)`. +1. Insert `sync_work_lifecycle(source, 'merged', target)`. 2. Release the source Work's primary and citation pointers. The trigger allows this only because step 1 exists. 3. `UPDATE manifestations SET work_id = target` for the moved Manifestations. @@ -1476,7 +1491,8 @@ for the fixture library. The legacy dict gains only **additive** fields: | --- | --- | | `PATCH /api/works/:id` with a Work-owned field | the Work | | … with a Manifestation-owned field | the **primary** Manifestation. The editor shows the primary, so this is what the user sees. The response names the `manifestation_id` it wrote. | -| … with a source field | the primary Asset's source aggregate (`SET_WORK_SOURCE` semantics unchanged) | +| … with `source_url` on a **non-video** Work (provenance/citation URL, a `SET_WORK_METADATA_FIELD` value) | the primary **Manifestation**'s `url`. This works whether or not the Work has an Asset. The existing video guard stays (work-source-identity.md). | +| `SET_WORK_SOURCE` / video source identity (`source_kind`, `provider`, `provider_id`, `source_url` on a video Work) | the primary Asset's `external_stream` aggregate (`SET_WORK_SOURCE` semantics unchanged) | | `POST /api/works` | creates Work + Manifestation + Asset in one transaction | | `POST /api/works/:id/pdf` (materialization) | the **working slot** of the primary Asset (§9.4). The client must send `asset_id` once more than one exists. Without it the request is refused (`ASSET_AMBIGUOUS`) rather than guessed. | | Annotation endpoints | the annotation's own Asset. A new annotation needs the displayed `asset_id`, which defaults to the primary only while exactly one PDF Asset exists. | @@ -1628,7 +1644,7 @@ its own identity or asset model. | --- | --- | | **#41 Citation V2** | Cite a **Manifestation**. Default to `citation_target(work)` = `COALESCE(citation_manifestation_id, primary_manifestation_id)`, and own the UI for choosing the citation Version (D4). Batch export resolves each Work to exactly one Manifestation. Build CSL-JSON from `citation_record(manifestation_id)` (§8.6), not from `works` columns. Decide cite-key persistence (§8.4) and the `urldate` quirk. Do not add citation fields to `works`. | | **#42 Ingestion / Web Works** | Every ingest creates, or attaches to, Work → Manifestation → Asset through one canonical command. Web Work = Manifestation(`web_page`) + snapshot Asset(s) under a new managed area, classified as canonical in the backup inventory. Duplicate warnings use §10 (hash, then identifiers, then optional RapidFuzz ranking). Provenance attaches to Manifestation fields and Asset origin. BibTeX/RIS import creates Manifestations and uses identifier normalization. | -| **#57 History / Trash** | Trash granularity is Work, Manifestation, or Asset. Asset `state = trashed` keeps bytes until purge, and purge uses the `pending_pdf_cleanup` rules. Merge and move are history events with previews. `work_lifecycle` redirects are the durable record of merges. | +| **#57 History / Trash** | Trash granularity is Work, Manifestation, or Asset. Asset `state = trashed` keeps bytes until purge, and purge uses the `pending_pdf_cleanup` rules. Merge and move are history events with previews. `sync_work_lifecycle` redirects are the durable record of merges. | | **#58 Offline UX** | "Available offline" is a property of **Assets** (bytes) plus the Work's metadata. Default to the primary Asset. Validate cached bytes with `content_sha256`. Pending edits stay keyed to the entity they were made on (§14.2). | | **#59 Capture** | Capture sends intent (URL, selection, snapshot) to the same ingestion command as #42. Dedup by normalized canonical URL, then the snapshot hash. No capture-specific Work shape. | | **#61 Reading Workflow** | The Secondary Reading View opens **another Asset**, of the same Manifestation or of another Manifestation of the same Work, and reads `manifestation_relations` for labels. Reading location is per Asset. Each pane shows its own Asset's annotations. "Open corresponding passage" and annotation transfer are assisted, reviewed actions (§6.2). Never assume page correspondence. | @@ -1738,7 +1754,7 @@ exactly right. | **H. Versions UI** | "Add another version / file", set primary, open a specific Version or File, and a version picker for Copy citation, all behind §13.4's progressive disclosure. Follow `DESIGN.md` for Work detail composition. #61's secondary view builds on this. | **Yes** | G | | **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256` (and `content_sha256` for legacy files). Offers the four choices in §10.2. | Yes | B, H | | **J. Identifier candidates + decline** | Normalized DOI/ISBN/arXiv candidates, a review list, and `duplicate_decisions`. | Yes | E, I | -| **K. Merge / move workflow** | `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | +| **K. Merge / move workflow** | `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `sync_work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | | **L. RapidFuzz evaluation** | A separate research/evaluation issue (dependency review, ranking quality on synthetic data). It only ranks; it never decides. | — | J | A, B and C can proceed in parallel after A's migration merges. D and E are From 787217e0534639e7a8513a48d92f8894f6665b03 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:42:11 +0000 Subject: [PATCH 05/24] docs(#60): quarantine legacy FK orphans, copy role tombstones, widen Asset predicate Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 101 ++++++++++++++++++++++++------------ 1 file changed, 67 insertions(+), 34 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index ac8b0e6b..57b1ef64 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -585,11 +585,12 @@ upgraded DBs: - each invariant's contradictory insert or update is refused; - legal moves cascade their owner columns; - a pointer target cannot be deleted or moved away while referenced; -- after the backfill, `PRAGMA foreign_key_check` reports nothing beyond - the legacy violations counted by the preflight (on clean fixtures, nothing at - all), and the integrity query returns no rows; -- the backfill preflight (§12.3) reports rows it cannot map rather than - inserting them. +- after the backfill, `PRAGMA foreign_key_check` reports nothing for the + rebuilt tables, and the integrity query returns no rows; +- a fixture with legacy orphaned annotations, roles and argument sources + upgrades successfully. Its orphans end up in `migration_quarantine`, + byte-for-byte equal to the originals, and are absent from the rebuilt + tables. --- @@ -1346,12 +1347,19 @@ The backfill rules: - A **video** Work gets an `external_stream` Asset carrying `provider`, `provider_id`, `source_url` and `thumb_url`. -- A Work with **no file, no video and no annotations** gets a Manifestation and - **no** Asset. -- A Work that has **annotations but no usable file reference** (a legacy - inconsistency) still gets a `managed_file` Asset, with a NULL - `storage_locator`, which reads as missing. Its annotations are never dropped - or left without an owner. +- **Asset-creation predicate.** A Work gets an Asset when it has **any** + Asset-owned value: a `file_path`, video identity, annotations, a non-empty + `source_mime`, `thumb_url` or `thumb_page`, or a non-zero materialization + revision. `POST /api/works` / `add_work()` accept `source_mime`, + `thumb_url` and `thumb_page` without a file, so metadata-only rows can + carry them. Those values need a canonical home when Asset authority lands, + or JSON parity would break. +- A Work that meets the predicate but has **no usable file or stream + reference** gets a `managed_file` Asset with a NULL `storage_locator`. It + is a placeholder that owns those values. The projection shows it exactly as + today (`file_path` empty: "No file attached"), and none of its values or + annotations are dropped or left without an owner. +- A Work with **none** of these values gets a Manifestation and **no** Asset. - A non-video `source_url` goes to `manifestations.url`. - **MIME type:** if `works.source_mime` is non-empty, `assets.media_type` gets **that exact value**. `application/pdf` is inferred only when it is absent @@ -1370,16 +1378,29 @@ The versions below are illustrative. Each is one migration in one PR (see 1. **vN (Slice A): create the entities and the integrity layer.** In one transaction: - 1. **Preflight.** Classify rows the backfill cannot map. Some legacy DBs - contain rows that already violate their existing `work_id` FK, such as - annotations or roles whose Work no longer exists; backup already warns - about these as `fk_violations`. Such rows are **carried over unchanged - with NULL new columns**: no owner is invented for them, and they do not - block the upgrade. A NULL child column exempts them from the composite - FKs, so the existing violation is neither repaired nor worsened. Only - their count is logged, which is privacy-safe. Anything else that cannot - be mapped **aborts** the migration with a precise reason code, and the - transaction rolls back. + 1. **Preflight and quarantine.** Some legacy DBs contain rows in the three + leaf tables that already violate one of their FKs: for example an + annotation or role whose Work no longer exists, or a role whose Person + does not. Backup already warns about these as `fk_violations`. They + **cannot be copied** into the rebuilt tables, because with + `foreign_keys = ON` the retained single-column FKs (`work_id → works`, + `person_id → persons`, …) are immediate constraints. Making only the new + columns NULL does not exempt a row, and deferring the check would only + move the failure to commit. So these rows are **quarantined**, never + dropped: + - They are identified with `PRAGMA foreign_key_check()` before + the rebuild. + - Each one is written verbatim, as `json_object(...)` of all its columns, + into a new FK-free canonical table `migration_quarantine(id, + source_table, source_rowid, row_json, reason, quarantined_at)`. + - The rebuild copy then skips them. + - Only per-table counts are logged, which is privacy-safe. + + The table lives in `prks_data.db`, so it is backed up with the library. + No owner is invented and the upgrade is not blocked. A later repair or + "reattach" tool can restore a row if its owner reappears. Any other row + that cannot be mapped (none are expected) **aborts** the migration with + a precise reason code, and the transaction rolls back. 2. Create `manifestations`, `assets`, `manifestation_relations` and `manifestation_identifiers`, with the parent pairs and composite FKs of §4.1. @@ -1396,9 +1417,10 @@ The versions below are illustrative. Each is one migration in one PR (see `annotations.asset_id` and pin `argument_sources` rows with pages when legacy code writes them without the new columns, reproducing the backfill rules. The child FKs cascade on DELETE. - 7. Run the integrity query, and compare `PRAGMA foreign_key_check` with the - preflight: **no new** violation may appear. Either check failing aborts - the migration. + 7. Run the integrity query and `PRAGMA foreign_key_check`. The rebuilt + tables must report **no** violations, because their orphans were + quarantined. The other tables must report no violation that the + preflight did not see. Either check failing aborts the migration. **No reader changes.** No filesystem work. D2 applies: no automatic full-library backup is made or required. @@ -1412,10 +1434,14 @@ The versions below are illustrative. Each is one migration in one PR (see 3. **vN+2: Manifestation authority.** The same move for bibliographic fields. The migration **copies revision counters** from `work-field/[W, f]` to `manifestation-field/[origin_MF(W), f]` (§14.2). -4. **vN+3: Role scope.** Set role scope by role type (§7.3), swap the - uniqueness index to include `manifestation_id`, and copy role revisions to - `manifestation-person-role` scopes for the rows that became - Manifestation-scoped. +4. **vN+3: Role scope.** Set role scope by role type (§7.3), and swap the + uniqueness index to include `manifestation_id`. Then copy **every** + `work-person-role/[W, P, r]` revision scope whose role type is + edition-scoped (Editor, Translator, Introduction, Foreword, Afterword) to + `manifestation-person-role/[origin_MF(W), P, r]`. This includes + **tombstones**, meaning scopes with a revision but no live row, which + `work_role_sync.get_roles_state()` deliberately keeps. Without them, an + offline add based on revision 0 could resurrect a deleted role. 5. **Later: retire legacy columns.** This is decided later (D10). Until then the columns stay as the maintained projection. @@ -1558,7 +1584,13 @@ The architecture must not assume a permanent local-first target (#179), but the existing durable-operation guarantees (I7, I12) are compatibility constraints and are kept. -**Scopes follow ownership, and revisions are carried, not reset.** +**Scopes follow ownership, and revisions are carried, not reset.** Every +copy iterates over the **`sync_entity_revisions` scope rows**, never over live +entity rows. Tombstones (a revision with no live row: a deleted annotation, a +removed role, a cleared field) are copied like any other scope, so a stale +device cannot resurrect what was deleted. The old scope row is kept, and legacy +operations keyed by it map through `origin_MF(W)`/`origin_AS(W)` as described +below. | Current scope | Future scope | Migration | | --- | --- | --- | @@ -1566,7 +1598,7 @@ and are kept. | `work-field/[W, f]` for W-owned `f` (status, title, abstract, author_text) | unchanged | — | | `work-source/W` | `asset-source/AS` | Copy to `origin_AS(W)`. | | `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `origin_AS(W)`. Deletion tombstones are copied too, so resurrection protection survives. | -| `work-person-role/[W, P, r]` | unchanged for W-scope; `manifestation-person-role/[MF, P, r]` for M-scope | Copy for backfilled M-scoped roles. | +| `work-person-role/[W, P, r]` | unchanged for W-scope; `manifestation-person-role/[MF, P, r]` for M-scope | Copy every scope row of an edition-scoped role type to `origin_MF(W)`, **tombstones included**. | | notes, tags, folder, playlist, open | unchanged | — | **Legacy operations after the move.** A pending `SET_WORK_METADATA_FIELD(W, @@ -1744,7 +1776,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests; (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins pages rows; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename and legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests; (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins pages rows; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | @@ -1775,9 +1807,10 @@ review. why citation identity includes `pages`. The alternative, "pages always pins a Version", is simpler, but it would force classical locators onto one edition. -2. **Legacy FK-orphan rows are carried, not blocking** (§12.3 step 1.1). Rows - that already violate their `work_id` FK keep NULL new columns and are - counted. The alternative is to make the upgrade refuse to run on such a DB. +2. **Legacy FK-orphan rows are quarantined, not blocking** (§12.3 step 1.1). + Rows that already violate an FK cannot be copied into a rebuilt table, so + they are kept verbatim in `migration_quarantine`. The alternative is to + make the upgrade refuse to run on such a DB. 3. **Slice A rebuilds `roles` and `annotations`** as well as `argument_sources`, so the composite FKs exist from the first migration. The alternative is to add the columns without FKs in Slice A and rebuild later, From d6b278e834ad7f42fea1f08e04964f10527dedd8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:42:45 +0000 Subject: [PATCH 06/24] docs(#60): refuse ambiguous legacy argument-source matches Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 57b1ef64..8998c5d6 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -938,8 +938,16 @@ primary key in place, so the migration rebuilds the table: **Legacy wire compatibility.** Clients that send `[{work_id, pages}]` keep working: -- An entry that matches an existing row's `(work_id, pages)` keeps that row's - `manifestation_id`. +- An entry that matches **exactly one** existing row's `(work_id, pages)` + keeps that row's `manifestation_id`. +- **Ambiguous match is refused.** If a `(work_id, pages)` key matches **more + than one** existing row (a Work-level locator and a pinned row with the same + pages, or pins to two Versions), the whole legacy list is refused with + `ARGUMENT_SOURCE_VERSION_REQUIRED`. The legacy shape cannot say which row it + means, so PRKS does not prefer either one. Only a Version-aware client can + create such rows, so only a Version-aware client can edit that list. A legacy + list containing the same `(work_id, pages)` twice is refused as a duplicate, + as today. - A new entry with empty `pages` is Work-level. - A new entry with `pages` is pinned to the Work's only Manifestation when it has exactly one. When it has several, the entry is **refused** with From 3898c831499eb9b8733c7199b4a728569d89789c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:45:00 +0000 Subject: [PATCH 07/24] docs(#60): bump argument-sources revision when quarantine shrinks a list Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 8998c5d6..c008fbeb 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -928,6 +928,13 @@ primary key in place, so the migration rebuilds the table: renumbers `order_index` as `0..n-1` for each Argument. This fixes the legacy `0, 0, 0` rows **without changing the order any reader sees**, so the `argument-sources` list value, and therefore its revision, is unchanged. + **Exception:** an Argument that lost a row to quarantine (§12.3 step 1.1, + for example a citation of a Work that no longer exists). `current_sources()` + reads rows without joining `works`, so such orphans are part of today's + synced list. That Argument's surviving rows are renumbered, and its + `argument-sources/` revision is **advanced** in the same transaction, so + an offline replace based on the old list conflicts instead of silently + succeeding. - It **pins** each row with non-empty `pages` to the Work's backfilled Manifestation, because that pinpoint was written against that pagination. It leaves rows with empty `pages` at Work level. A user can later unpin a @@ -1401,7 +1408,11 @@ The versions below are illustrative. Each is one migration in one PR (see - Each one is written verbatim, as `json_object(...)` of all its columns, into a new FK-free canonical table `migration_quarantine(id, source_table, source_rowid, row_json, reason, quarantined_at)`. - - The rebuild copy then skips them. + - The rebuild copy then skips them. Quarantining an `argument_sources` + row changes that Argument's synced list, so its `argument-sources/` + revision is advanced (§8.5). Quarantined annotations and roles + already belong to a Work that no longer exists, so no live scope + reports them. - Only per-table counts are logged, which is privacy-safe. The table lives in `prks_data.db`, so it is backed up with the library. @@ -1784,7 +1795,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests; (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins pages rows; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests; (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | From 37cd9f465841c4f1b891530d6321010f80451247 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:48:01 +0000 Subject: [PATCH 08/24] docs(#60): make only-child moves transactional (deferred NO ACTION, work retirement marker) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 102 +++++++++++++++++++++++++++++------- 1 file changed, 82 insertions(+), 20 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index c008fbeb..803f14b0 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -504,7 +504,9 @@ assets: UNIQUE (id, manifestation_id), UNIQUE (id, work_id) assets FOREIGN KEY (manifestation_id, work_id) REFERENCES manifestations(id, work_id) ON UPDATE CASCADE ON DELETE CASCADE manifestations FOREIGN KEY (primary_asset_id, id) REFERENCES assets(id, manifestation_id) - ON UPDATE RESTRICT ON DELETE RESTRICT DEFERRABLE INITIALLY DEFERRED + ON UPDATE NO ACTION ON DELETE NO ACTION DEFERRABLE INITIALLY DEFERRED + -- NO ACTION, not RESTRICT: SQLite applies RESTRICT immediately even + -- on a deferred FK, which would deadlock moving an only child -- leaf tables, rebuilt in the same migration (SQLite cannot add a table constraint in place) annotations FOREIGN KEY (asset_id, work_id) REFERENCES assets(id, work_id) ON UPDATE CASCADE ON DELETE CASCADE @@ -517,6 +519,14 @@ argument_sources -- works pointers: ALTER TABLE ADD COLUMN + triggers works.primary_manifestation_id TEXT works.citation_manifestation_id TEXT + +-- transaction-local Work retirement marker (see "Transitions" below) +work_retirement_guard (id INTEGER PRIMARY KEY) -- always empty +work_retirement ( + work_id TEXT PRIMARY KEY, + must_clear INTEGER NOT NULL DEFAULT 1 + REFERENCES work_retirement_guard(id) DEFERRABLE INITIALLY DEFERRED +) -- any row still present at COMMIT violates the FK, so the commit fails ``` The existing single-column FKs (`roles.work_id → works`, @@ -524,12 +534,12 @@ The existing single-column FKs (`roles.work_id → works`, | Trigger | Rule | | --- | --- | -| `works_manifestation_pointers_owned` (BEFORE UPDATE OF `primary_manifestation_id`, `citation_manifestation_id` ON `works`) | A non-NULL pointer must name a Manifestation whose `work_id` is this Work (`MANIFESTATION_OWNER_MISMATCH`). The primary may not be set back to NULL (`WORK_PRIMARY_MANIFESTATION_REQUIRED`). The one exception is a Work already recorded in `sync_work_lifecycle` as `merged`, whose row the same merge transaction is about to delete (§10.4). | +| `works_manifestation_pointers_owned` (BEFORE UPDATE OF `primary_manifestation_id`, `citation_manifestation_id` ON `works`) | A non-NULL pointer must name a Manifestation whose `work_id` is this Work (`MANIFESTATION_OWNER_MISMATCH`). The primary may not be set back to NULL (`WORK_PRIMARY_MANIFESTATION_REQUIRED`), **unless the Work is marked in `work_retirement`**, meaning the same transaction deletes or merges it (see "Transitions" below). | +| `works_retirement_clear` (AFTER DELETE ON `works`) | Removes the Work's `work_retirement` marker when its row is deleted, which is the only way the marker can leave before COMMIT succeeds. | | `works_manifestation_pointers_insert` (BEFORE INSERT ON `works`) | A Work is inserted with NULL pointers. Its Manifestation cannot exist before the Work, because `manifestations.work_id` references it. | | `manifestations_pointer_target_move` (BEFORE UPDATE OF `work_id` ON `manifestations`) | A Manifestation that its Work names as primary or citation cannot be moved away (`MANIFESTATION_IS_POINTER_TARGET`). The pointer must change first, in the same transaction. | | `manifestations_pointer_target_delete` (BEFORE DELETE ON `manifestations`) | The same rule for deletion. A whole-Work delete is unaffected: the Work row is gone before its Manifestations cascade. | | `manifestation_origin_immutable`, `asset_origin_immutable` (BEFORE UPDATE OF `origin_work_id`) | The legacy-operation mapping (§11.2) can never be rewritten. | -| `manifestation_primary_asset_kept` (BEFORE UPDATE OF `primary_asset_id` ON `manifestations`, BEFORE UPDATE OF `state` ON `assets`) | `primary_asset_id` may not be cleared while the Manifestation still has an `active` Asset, and the primary Asset may not be trashed until the pointer has moved. Deleting it is already refused by the deferred FK. | What each required invariant maps to: @@ -552,23 +562,63 @@ What each required invariant maps to: re-copy owner columns. - **A pointer target cannot leave silently.** A Manifestation that is its Work's primary or citation Version cannot be moved away or deleted until the - pointer changes. The same holds for a primary Asset (deferred FK, checked at - commit). + pointer changes (triggers, immediate). A primary Asset may be moved or + deleted inside a transaction, but that transaction cannot commit unless the + pointer has also changed (the deferred `NO ACTION` FK). - **A pinned citation blocks deleting a Version.** Deleting a Manifestation that an Argument pins is refused (`RESTRICT`). The canonical command reports it as in use, like `ARGUMENT_IN_USE` today. Deleting a whole Work still cascades cleanly: every child row goes with it. +**Transitions: every legal final state has a legal path.** Pointer rules must +not deadlock a move whose *result* is valid, and they must not need transient +placeholder Versions or Files. The two "only child" cases: + +- **Moving the only (primary) Asset out of a Manifestation.** The final state, + a Manifestation with zero Assets and a NULL `primary_asset_id`, is valid. In + one transaction: `UPDATE assets SET manifestation_id, work_id`, then + `UPDATE manifestations SET primary_asset_id = NULL` (either order works). The + deferred `NO ACTION` FK is satisfied at COMMIT. There is no "may not clear + while an Asset is active" trigger, because that rule deadlocked this case. + The final-state rule it expressed now lives in the canonical command's + final-state check and in the integrity query (below). +- **Moving the only Manifestation out of a Work.** A Work with zero Versions is + **not** a valid final state, so the emptied source Work has to be retired in + the same transaction. There are exactly two supported outcomes: + - **delete** the empty Work (`MOVE_MANIFESTATION` with + `empty_source = delete`); + - **merge** it into the target (`MERGE_WORKS`, §10.4). + + Both use the same explicit transition, which the triggers recognize: + 1. `INSERT INTO work_retirement (work_id)`. For a merge, the durable + `sync_work_lifecycle(merged)` row is also inserted. + 2. Clear the Work's pointers. The trigger allows this only because of step 1. + 3. Move the Manifestation(s). The owner columns follow by cascade. + 4. `DELETE FROM works` for the retired Work. `works_retirement_clear` removes + the marker. + + If step 4 never happens, the marker's guard FK makes COMMIT fail and + everything rolls back. So a retirement marker can never outlive its + transaction, and a Work can never persist without a primary. + +Moving a **primary** Asset or Manifestation while siblings remain simply +re-points the pointer to a sibling first (or, for the Asset, in either order). +Moving a non-primary child needs no pointer change at all. + **What SQLite cannot enforce at commit.** SQLite has no deferred triggers or -CHECKs, so "every Work *has* a primary Manifestation" and "a Manifestation with -an active Asset *has* a primary Asset" cannot be required at the moment a row -is inserted. Both are set by the single statement or canonical command that +CHECKs. So "every Work *has* a primary Manifestation" (at creation) and "a +Manifestation with an active Asset *has* a primary Asset, and it is active" +cannot be required as database constraints at commit. Both are set by the single statement or canonical command that creates the rows. In Slice A the mirror trigger does it: it inserts the Manifestation (and Asset) and sets the pointers inside the Work's own INSERT. -The triggers above then make sure a pointer, once set, can never be cleared, -moved or pointed at a foreign row. An **integrity query** (Works with a NULL -primary, Manifestations with active Assets but no primary Asset) runs in the -schema tests, after the backfill, and in backup verification. +The triggers above then make sure a Work's primary, once set, can only be +cleared by retiring the Work, and that no pointer ever names a foreign row. +Every canonical command that touches Versions or Files runs a **final-state +check** on the rows it touched before committing, and refuses the command if +the check fails. The same check exists as an **integrity query** (Works with a +NULL primary; Manifestations with active Assets but no primary Asset, or whose +primary Asset is not active), which runs in the schema tests, after the +backfill, and in backup verification. All of the above was prototyped against SQLite 3.45 (the version used in development). Contradictory rows for each invariant were refused, and legal @@ -584,7 +634,17 @@ upgraded DBs: - each invariant's contradictory insert or update is refused; - legal moves cascade their owner columns; -- a pointer target cannot be deleted or moved away while referenced; +- a pointer target cannot be deleted or moved away while referenced, and a + transaction that moves a primary Asset without changing the pointer fails at + COMMIT; +- **transition tests** (Slice A for the schema, Slice K for the commands): + moving the only primary Asset out leaves the source Manifestation with zero + Assets and a NULL primary; moving a non-primary Asset needs no pointer + change; moving the only Manifestation out succeeds with each supported + outcome for the emptied Work (delete, merge); clearing a primary without + retirement is refused; a retirement marker left behind makes COMMIT fail; + and every failed transition rolls back with all pointers and owner columns + unchanged; - after the backfill, `PRAGMA foreign_key_check` reports nothing for the rebuilt tables, and the integrity query returns no rows; - a fixture with legacy orphaned annotations, roles and argument sources @@ -1220,23 +1280,25 @@ before mutation. None of them is silent. | Operation | Effect | IDs | | --- | --- | --- | | `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `sync_work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | -| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work, the pointer must move first. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. If the old Work is left with no Manifestation, it is merged into the target or deleted, and the user chooses. | -| `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the pointer must move first. | `AS-…` unchanged. | +| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | +| `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the same transaction re-points `primary_asset_id` to a remaining active sibling, or clears it when none remains. A Manifestation with zero Assets is a valid final state. The deferred `NO ACTION` FK checks the pointer at COMMIT. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | **`MERGE_WORKS` transaction order.** The §4.1 constraints depend on this order, and all steps are one transaction: -1. Insert `sync_work_lifecycle(source, 'merged', target)`. +1. Insert `sync_work_lifecycle(source, 'merged', target)` and + `work_retirement(source)` (§4.1 "Transitions"). 2. Release the source Work's primary and citation pointers. The trigger allows - this only because step 1 exists. + this only because of the retirement marker. 3. `UPDATE manifestations SET work_id = target` for the moved Manifestations. The owner columns of their Assets, annotations, scoped roles and pinned argument sources follow by cascade. 4. Re-point Work-level rows: tags, Work-scoped roles, and Work-level argument sources. Only exactly identical citation rows are collapsed (§8.5), and the user's choices from the preview are applied. -5. Delete the source Work row, and tombstone its sync scopes. +5. Delete the source Work row, which also removes its retirement marker, and + tombstone its sync scopes. 6. Advance the revisions the merge changed, then commit. The deferred FKs are checked at this point. @@ -1795,7 +1857,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests; (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | @@ -1805,7 +1867,7 @@ exactly right. | **H. Versions UI** | "Add another version / file", set primary, open a specific Version or File, and a version picker for Copy citation, all behind §13.4's progressive disclosure. Follow `DESIGN.md` for Work detail composition. #61's secondary view builds on this. | **Yes** | G | | **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256` (and `content_sha256` for legacy files). Offers the four choices in §10.2. | Yes | B, H | | **J. Identifier candidates + decline** | Normalized DOI/ISBN/arXiv candidates, a review list, and `duplicate_decisions`. | Yes | E, I | -| **K. Merge / move workflow** | `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `sync_work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | +| **K. Merge / move workflow** | Command-level versions of the §4.1 transition tests, including each `empty_source` outcome. `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `sync_work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | | **L. RapidFuzz evaluation** | A separate research/evaluation issue (dependency review, ranking quality on synthetic data). It only ranks; it never decides. | — | J | A, B and C can proceed in parallel after A's migration merges. D and E are From 86b0b2380229d8cd7e8162525a9f5a2e633c951a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:48:37 +0000 Subject: [PATCH 09/24] docs(#60): advance live role scopes when quarantine removes a role row Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 803f14b0..2f609bd7 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1470,11 +1470,22 @@ The versions below are illustrative. Each is one migration in one PR (see - Each one is written verbatim, as `json_object(...)` of all its columns, into a new FK-free canonical table `migration_quarantine(id, source_table, source_rowid, row_json, reason, quarantined_at)`. - - The rebuild copy then skips them. Quarantining an `argument_sources` - row changes that Argument's synced list, so its `argument-sources/` - revision is advanced (§8.5). Quarantined annotations and roles - already belong to a Work that no longer exists, so no live scope - reports them. + - The rebuild copy then skips them. + - **Revisions follow the live aggregate that changes, not the table + type.** For each quarantined row, PRKS advances the revision of every + scope whose state endpoint reported that row as present: + - an `argument_sources` row whose Argument still exists: advance + `argument-sources/` (§8.5); + - a `roles` row whose **Work still exists** (for example only its + Person is missing): `work_role_sync.get_roles_state()` builds + `present` from `roles` without joining `persons`, so that row was + reported as `present: true`. Insert or advance the + `work-person-role/[W, P, role]` revision, leaving a tombstone. After + the migration the scope reports absence at a strictly newer + revision; + - a row whose owning Work or Argument no longer exists (annotations + can only be orphaned this way): no live scope reported it, so no + scope is created. - Only per-table counts are logged, which is privacy-safe. The table lives in `prks_data.db`, so it is backed up with the library. @@ -1857,7 +1868,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | From e6348e0417359062fe5a559e98620a8f5538a8fc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:49:24 +0000 Subject: [PATCH 10/24] docs(#60): enforce same-Work manifestation_relations with composite FKs Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 2f609bd7..c3aed405 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -400,7 +400,7 @@ enough (D8). | Entity | Purpose | | --- | --- | -| `manifestation_relations(from_id, to_id, relation)` | Typed links between Manifestations of the **same** Work: `revision_of`, `published_version_of`, `new_edition_of`, `translation_of`, `reprint_of`. These carry "arXiv v2 revises v1", "journal version of the accepted manuscript", and "Serbian translation of the Greek text". They carry no page correspondence (#61 owns assisted correspondence). | +| `manifestation_relations(work_id, from_id, to_id, relation)` | Typed links between Manifestations of the **same** Work. The rule is enforced by composite FKs on both endpoints and `CHECK (from_id <> to_id)` (§4.1). Relation types: `revision_of`, `published_version_of`, `new_edition_of`, `translation_of`, `reprint_of`. These carry "arXiv v2 revises v1", "journal version of the accepted manuscript", and "Serbian translation of the Greek text". They carry no page correspondence (#61 owns assisted correspondence). | | `manifestation_identifiers(manifestation_id, scheme, value, normalized)` | DOI, ISBN, arXiv ID (versioned and versionless), PMID, and similar. `works.doi`/`isbn` backfill into these later. This is the entity duplicate detection keys on. | | `roles` gaining a nullable `manifestation_id` | Edition-scoped credit (§7). The owner is enforced by a composite FK (§4.1). | | `argument_sources` rebuilt with `PRIMARY KEY (argument_id, order_index)`, a nullable `manifestation_id`, and citation identity `UNIQUE (argument_id, work_id, COALESCE(manifestation_id, ''), pages)` | One Argument can cite the Work in general **and** several Versions of it, each with its own pinpoint (§8.5). | @@ -515,6 +515,14 @@ roles FOREIGN KEY (manifestation_id, work_id) REFERENCES manifestations argument_sources FOREIGN KEY (manifestation_id, work_id) REFERENCES manifestations(id, work_id) ON UPDATE CASCADE ON DELETE RESTRICT +-- new table: both endpoints must be Manifestations of the row's own work_id +manifestation_relations (work_id, from_id, to_id, relation, + PRIMARY KEY (from_id, to_id, relation), CHECK (from_id <> to_id)) + FOREIGN KEY (from_id, work_id) REFERENCES manifestations(id, work_id) + ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED + FOREIGN KEY (to_id, work_id) REFERENCES manifestations(id, work_id) + ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED + -- deferred so a merge can move both endpoints one statement at a time -- works pointers: ALTER TABLE ADD COLUMN + triggers works.primary_manifestation_id TEXT @@ -552,6 +560,7 @@ What each required invariant maps to: | `argument_sources(work_id, manifestation_id)` has a matching owner | composite FK. A NULL `manifestation_id` is a Work-level citation. | | `annotations(work_id, asset_id)`: the Asset's Manifestation belongs to that Work | `annotations → assets(id, work_id)`, and `assets(manifestation_id, work_id) → manifestations(id, work_id)`. The chain carries ownership from the Asset through its Manifestation. | | An Asset's denormalized `work_id` matches its Manifestation's Work | composite FK on `assets` | +| `manifestation_relations` never crosses Works and never relates a Manifestation to itself | two deferred composite FKs, `(from_id, work_id)` and `(to_id, work_id)`, plus `CHECK (from_id <> to_id)`. If one endpoint moves to another Work while the relation remains, the commit fails. When a merge moves **both** endpoints, the relation's `work_id` follows by cascade. `MOVE_MANIFESTATION` deletes the moved Version's relations first (§10.4). | **How the cascade and restrict choices behave:** @@ -1868,7 +1877,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | From 127746a4192b1753c9f842e422aabb6fa9d104ac Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:50:12 +0000 Subject: [PATCH 11/24] docs(#60): make work_retirement machinery unbypassable Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index c3aed405..054b1217 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -529,7 +529,7 @@ works.primary_manifestation_id TEXT works.citation_manifestation_id TEXT -- transaction-local Work retirement marker (see "Transitions" below) -work_retirement_guard (id INTEGER PRIMARY KEY) -- always empty +work_retirement_guard (id INTEGER PRIMARY KEY CHECK (0)) -- can never hold a row work_retirement ( work_id TEXT PRIMARY KEY, must_clear INTEGER NOT NULL DEFAULT 1 @@ -543,7 +543,9 @@ The existing single-column FKs (`roles.work_id → works`, | Trigger | Rule | | --- | --- | | `works_manifestation_pointers_owned` (BEFORE UPDATE OF `primary_manifestation_id`, `citation_manifestation_id` ON `works`) | A non-NULL pointer must name a Manifestation whose `work_id` is this Work (`MANIFESTATION_OWNER_MISMATCH`). The primary may not be set back to NULL (`WORK_PRIMARY_MANIFESTATION_REQUIRED`), **unless the Work is marked in `work_retirement`**, meaning the same transaction deletes or merges it (see "Transitions" below). | -| `works_retirement_clear` (AFTER DELETE ON `works`) | Removes the Work's `work_retirement` marker when its row is deleted, which is the only way the marker can leave before COMMIT succeeds. | +| `works_retirement_clear` (AFTER DELETE ON `works`) | Removes the Work's `work_retirement` marker when its row is deleted. | +| `work_retirement_delete_only_after_work` (BEFORE DELETE ON `work_retirement`) | A marker cannot be deleted while its Work row still exists (`WORK_RETIREMENT_WORK_STILL_EXISTS`). The only path out is `works_retirement_clear`, which runs after the Work row is gone. | +| `work_retirement_no_update` (BEFORE UPDATE ON `work_retirement`) and `work_retirement_guard_no_update` | Markers and the guard are immutable, so a marker cannot be renamed to a missing Work and then deleted. | | `works_manifestation_pointers_insert` (BEFORE INSERT ON `works`) | A Work is inserted with NULL pointers. Its Manifestation cannot exist before the Work, because `manifestations.work_id` references it. | | `manifestations_pointer_target_move` (BEFORE UPDATE OF `work_id` ON `manifestations`) | A Manifestation that its Work names as primary or citation cannot be moved away (`MANIFESTATION_IS_POINTER_TARGET`). The pointer must change first, in the same transaction. | | `manifestations_pointer_target_delete` (BEFORE DELETE ON `manifestations`) | The same rule for deletion. A whole-Work delete is unaffected: the Work row is gone before its Manifestations cascade. | @@ -607,8 +609,16 @@ placeholder Versions or Files. The two "only child" cases: the marker. If step 4 never happens, the marker's guard FK makes COMMIT fail and - everything rolls back. So a retirement marker can never outlive its - transaction, and a Work can never persist without a primary. + everything rolls back. The machinery cannot be bypassed: + - the guard table has `CHECK (0)`, so no row can be inserted to satisfy the + FK; + - `must_clear` is `NOT NULL`; + - a marker cannot be deleted while its Work exists; + - a marker cannot be updated. + + **No transaction can commit a live Work with a NULL + `primary_manifestation_id`.** A retirement marker never outlives its + transaction. Moving a **primary** Asset or Manifestation while siblings remain simply re-points the pointer to a sibling first (or, for the Asset, in either order). @@ -652,6 +662,9 @@ upgraded DBs: change; moving the only Manifestation out succeeds with each supported outcome for the emptied Work (delete, merge); clearing a primary without retirement is refused; a retirement marker left behind makes COMMIT fail; + **the direct bypasses are refused**: inserting a guard row (`CHECK`), + deleting a marker while its Work exists, updating or renaming a marker, and + a `NULL` `must_clear`; and every failed transition rolls back with all pointers and owner columns unchanged; - after the backfill, `PRAGMA foreign_key_check` reports nothing for the @@ -1877,7 +1890,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; the direct guard and marker bypasses are refused; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | From 6a8b7c25f34cbb4e4d16be6f81264848759a4a9f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:55:04 +0000 Subject: [PATCH 12/24] docs(#60): lossless role-credit merge, projected title/abstract routing, CAS fingerprinting, destination primary on move, re-point processing imports Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 45 +++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 7 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 054b1217..2f4f8925 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1201,9 +1201,12 @@ Asset AS-… (one File in the UI; owns annotations, page state, thumbnails) (materialization, linearization, copy-on-write retarget) follows a fixed order: 1. commit `content_sha256 = NULL`, `byte_size = NULL` and - `fingerprinted_at = NULL` (the working bytes are **pending**); + `fingerprinted_at = NULL`, and increment `content_generation` (the + working bytes are **pending**); 2. replace the bytes durably (`fs_durability`); - 3. hash what was written and commit the new values. + 3. hash what was written and commit the new values, with the same + compare-and-set on `(storage_locator, content_generation)` that the + fingerprint pass uses (§12.4). A crash anywhere in between leaves NULL, meaning "unknown", never a stale hash that clients or backup would trust. The fingerprint pass (§12.4) treats @@ -1303,7 +1306,7 @@ before mutation. None of them is silent. | --- | --- | --- | | `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `sync_work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | | `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | -| `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the same transaction re-points `primary_asset_id` to a remaining active sibling, or clears it when none remains. A Manifestation with zero Assets is a valid final state. The deferred `NO ACTION` FK checks the pointer at COMMIT. | `AS-…` unchanged. | +| `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the same transaction re-points `primary_asset_id` to a remaining active sibling, or clears it when none remains. A Manifestation with zero Assets is a valid final state. **On the destination side**, if the destination Manifestation had no active Asset, which means its `primary_asset_id` is NULL, the moved active Asset becomes its primary in the same transaction. Otherwise the destination's primary is unchanged. The deferred `NO ACTION` FK and the command's final-state check verify both sides at COMMIT. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | **`MERGE_WORKS` transaction order.** The §4.1 constraints depend on this @@ -1316,9 +1319,23 @@ order, and all steps are one transaction: 3. `UPDATE manifestations SET work_id = target` for the moved Manifestations. The owner columns of their Assets, annotations, scoped roles and pinned argument sources follow by cascade. -4. Re-point Work-level rows: tags, Work-scoped roles, and Work-level argument - sources. Only exactly identical citation rows are collapsed (§8.5), and the - user's choices from the preview are applied. +4. Re-point Work-level rows to the target: + - tags; + - Work-scoped roles; + - Work-level argument sources; + - `processing_files.imported_work_id`, so that re-importing an + already-imported file stays idempotent and is not orphaned by the source + row's `ON DELETE SET NULL`. + + Only exactly identical citation rows are collapsed (§8.5). **Role + collisions never lose a credit.** If the source and the target both have a + Work-scoped `(person, role_type)` with *different* `credit_name` values, the + target's row is kept and the source spelling is preserved as a + `manifestation_credit_overrides` row on each moved Manifestation. Those + Versions keep citing the name they were credited with. The preview shows + every such collision and lets the user pick a single spelling instead. + Identical rows are collapsed. The user's choices from the preview are + applied. 5. Delete the source Work row, which also removes its retirement marker, and tombstone its sync scopes. 6. Advance the revisions the merge changed, then commit. The deferred FKs are @@ -1566,6 +1583,19 @@ fixed number of Assets per run, at startup or on demand. It streams each managed file through SHA-256, records `content_sha256`, `byte_size` and a `fingerprinted_at`, and skips missing files, which are reported, never treated as empty (the same rule as the text index). It never modifies bytes. + +The pass must never record a stale digest while PRKS rewrites the same file +(materialization, linearization, copy-on-write retarget): +- Each in-place rewrite increments `assets.content_generation` in the same + commit that NULLs the hash (§9.4). +- The pass reads `(storage_locator, content_generation)` and hashes the bytes + under the basename's `managed_pdf_path_lock()`, the lock the rewrite and + copy-on-write paths already take. +- It commits with a compare-and-set: + `UPDATE assets SET content_sha256 = ?, … WHERE id = ? AND storage_locator = ? + AND content_generation = ?`. +- If the locator or generation changed in between, the update matches no row, + the digest is discarded, and the Asset stays pending for the next pass. It stores hashes in the canonical DB because they describe canonical bytes, and they are re-verifiable, so a restore can recompute them. @@ -1629,7 +1659,8 @@ for the fixture library. The legacy dict gains only **additive** fields: | Legacy write | Routed to | | --- | --- | -| `PATCH /api/works/:id` with a Work-owned field | the Work | +| `PATCH /api/works/:id` with a Work-owned field | the Work, except `title` and `abstract`, below | +| … with `title` or `abstract` | **the value the legacy projection is showing.** If the primary Manifestation has a non-NULL override for that field, the edit writes the override; otherwise it writes the Work's canonical value. Either way the response shows the edit, and no hidden fallback changes silently. Changing the *canonical* value while an override exists needs the Version-aware API (§13.3). | | … with a Manifestation-owned field | the **primary** Manifestation. The editor shows the primary, so this is what the user sees. The response names the `manifestation_id` it wrote. | | … with `source_url` on a **non-video** Work (provenance/citation URL, a `SET_WORK_METADATA_FIELD` value) | the primary **Manifestation**'s `url`. This works whether or not the Work has an Asset. The existing video guard stays (work-source-identity.md). | | `SET_WORK_SOURCE` / video source identity (`source_kind`, `provider`, `provider_id`, `source_url` on a video Work) | the primary Asset's `external_stream` aggregate (`SET_WORK_SOURCE` semantics unchanged) | From 261b1b34ccdd2e5b785459519d51d22d9fc30b45 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:55:14 +0000 Subject: [PATCH 13/24] docs(#60): list content_generation in the Asset record Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 2f4f8925..8110f3a4 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1103,6 +1103,7 @@ not a #60 concern. | `byte_size` | Size of the current bytes. Filled at write time, or by the fingerprint pass. | | `ingest_sha256` | SHA-256 of the source bytes **as first ingested**, before linearization or materialization. Immutable once set. It is the content identity of the lineage. NULL for legacy Assets (§9.4). | | `content_sha256` | SHA-256 of the **working** bytes at `storage_locator`. Updated whenever PRKS rewrites them. Serves integrity checks and client cache validation. | +| `content_generation` | An integer incremented by every in-place rewrite of the working bytes. Hash writers commit only if it is unchanged since they read it (§9.4, §12.4). | | `origin` | `upload` \| `processing_import` \| `adopted` \| `web_capture` \| `legacy`. | | `origin_url`, `origin_ref` | Where the bytes came from (a download URL, or a Processing File ID). This is provenance, not a citation. | | `derived_from_asset_id` | A user-visible relation between **separate** Assets, for example a copy annotated in another tool that the user brought in. It is never used for PRKS's own materialization. | From 0f5c78ef80f977f2d68bf0b3f48d978a20749ac1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 14:57:54 +0000 Subject: [PATCH 14/24] docs(#60): credit overrides required for merge; rewrite holds path lock across hash protocol Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 8110f3a4..e491a67d 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -861,8 +861,11 @@ stored per row, so a user can override a default. - **Credit-name override:** the credit on a Work-scoped Author row is the default. A Manifestation may override how that person is credited on it through `manifestation_credit_overrides(manifestation_id, person_id, - role_type, credit_name)`. This is optional and ships only when a UI needs it. - Backfill writes none. + role_type, credit_name)`. `credits(M)` (§7.5) applies these overrides. The + backfill writes none, and no UI for editing them is needed early. **The + table is nevertheless required by the time `MERGE_WORKS` ships (Slice K)**, + because a merge preserves colliding credit spellings there (§10.4, step 4). + Without it, a merge would have to drop a credit. ### 7.3 Backfill rule @@ -1197,10 +1200,13 @@ Asset AS-… (one File in the UI; owns annotations, page state, thumbnails) already got this file?". - `content_sha256` identifies the current working bytes. It is used for integrity, backup verification, and client cache validation (#52, #58). -- **Hash updates are crash-safe.** Replacing bytes on disk and updating the - hash in SQLite cannot be one atomic step, so every in-place rewrite - (materialization, linearization, copy-on-write retarget) follows a fixed - order: +- **Hash updates are crash-safe and race-free.** Replacing bytes on disk and + updating the hash in SQLite cannot be one atomic step, so every in-place + rewrite (materialization, linearization, copy-on-write retarget) follows a + fixed order, **holding the basename's `managed_pdf_path_lock()` for all + three steps**. The current COW and replace paths already take that lock. The + fingerprint pass takes the same lock, so it can never hash between step 1 + and step 3: 1. commit `content_sha256 = NULL`, `byte_size = NULL` and `fingerprinted_at = NULL`, and increment `content_generation` (the working bytes are **pending**); @@ -1932,7 +1938,7 @@ exactly right. | **H. Versions UI** | "Add another version / file", set primary, open a specific Version or File, and a version picker for Copy citation, all behind §13.4's progressive disclosure. Follow `DESIGN.md` for Work detail composition. #61's secondary view builds on this. | **Yes** | G | | **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256` (and `content_sha256` for legacy files). Offers the four choices in §10.2. | Yes | B, H | | **J. Identifier candidates + decline** | Normalized DOI/ISBN/arXiv candidates, a review list, and `duplicate_decisions`. | Yes | E, I | -| **K. Merge / move workflow** | Command-level versions of the §4.1 transition tests, including each `empty_source` outcome. `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `sync_work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | +| **K. Merge / move workflow** | Command-level versions of the §4.1 transition tests, including each `empty_source` outcome. Ships `manifestation_credit_overrides` (§7.2) if it does not exist yet, and tests a credit-spelling collision. `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `sync_work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | | **L. RapidFuzz evaluation** | A separate research/evaluation issue (dependency review, ranking quality on synthetic data). It only ranks; it never decides. | — | J | A, B and C can proceed in parallel after A's migration merges. D and E are From 7c7e39c98f3a60ce343477c9cc9f6a2347530a8a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:02:29 +0000 Subject: [PATCH 15/24] docs(#60): content_generation NOT NULL DEFAULT 0; no empty overrides; override conflict unit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index e491a67d..b2697878 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -685,7 +685,7 @@ describes where today's value goes; §12 covers how. | Concept (current column/table) | Owner | Backfill of current value | Rationale / tradeoff | | --- | --- | --- | --- | | Canonical title (`title`) | **W** | stays on `works.title` | The name the user files the work under. | -| Publication title | **M** (`manifestations.title`, NULL = inherit Work title) | NULL (inherit) | For one version, editing the title changes both, as today. A translation sets its own. Inheritance avoids copying every title. | +| Publication title | **M** (`manifestations.title`, NULL = inherit Work title; never `''`, enforced by `CHECK (title IS NULL OR title <> '')`, and the same for the `abstract` override) | NULL (inherit) | For one version, editing the title changes both, as today. A translation sets its own. Inheritance avoids copying every title. | | Subtitle | **M** (new `subtitle`) | NULL | Differs by edition. There is no current column. | | Document type (`doc_type`) | **M** | copied | This is a BibTeX/CSL entry type: a preprint is `misc`/`online`, a journal version is `article`. The Types page derives from the primary M. **Tradeoff:** the Types page may later want a Work-level "kind of intellectual work" (monograph, paper, lecture). It is not needed now. | | Year / date (`year`, `published_date`) | **M** | copied | Publication date of this form. An optional Work-level `original_date` (for example "c. 375 BC") is a future addition. | @@ -1106,7 +1106,7 @@ not a #60 concern. | `byte_size` | Size of the current bytes. Filled at write time, or by the fingerprint pass. | | `ingest_sha256` | SHA-256 of the source bytes **as first ingested**, before linearization or materialization. Immutable once set. It is the content identity of the lineage. NULL for legacy Assets (§9.4). | | `content_sha256` | SHA-256 of the **working** bytes at `storage_locator`. Updated whenever PRKS rewrites them. Serves integrity checks and client cache validation. | -| `content_generation` | An integer incremented by every in-place rewrite of the working bytes. Hash writers commit only if it is unchanged since they read it (§9.4, §12.4). | +| `content_generation` | `INTEGER NOT NULL DEFAULT 0`, set to 0 on creation and by the backfill. It is incremented (`content_generation + 1`) by every in-place rewrite of the working bytes. Because it is never NULL, the compare-and-set can always match. Otherwise `NULL = ?` never matches and `NULL + 1` stays NULL. Hash writers commit only if it is unchanged since they read it (§9.4, §12.4). | | `origin` | `upload` \| `processing_import` \| `adopted` \| `web_capture` \| `legacy`. | | `origin_url`, `origin_ref` | Where the bytes came from (a download URL, or a Processing File ID). This is provenance, not a citation. | | `derived_from_asset_id` | A user-visible relation between **separate** Assets, for example a copy annotated in another tool that the user brought in. It is never used for PRKS's own materialization. | @@ -1459,6 +1459,7 @@ argument_sources(A, W-1, '45') doi, publisher, year, … (copied) media_type=COALESCE(source_mime, 'application/pdf'), origin='legacy', ingest_sha256=NULL, content_sha256=NULL (fingerprint pass fills), + content_generation=0, materialization revisions (copied), thumb_page (copied) annotations(…, asset_id = AS-uuid5(W-1)) rebuilt roles(…, manifestation_id = NULL) rebuilt; @@ -1667,7 +1668,7 @@ for the fixture library. The legacy dict gains only **additive** fields: | Legacy write | Routed to | | --- | --- | | `PATCH /api/works/:id` with a Work-owned field | the Work, except `title` and `abstract`, below | -| … with `title` or `abstract` | **the value the legacy projection is showing.** If the primary Manifestation has a non-NULL override for that field, the edit writes the override; otherwise it writes the Work's canonical value. Either way the response shows the edit, and no hidden fallback changes silently. Changing the *canonical* value while an override exists needs the Version-aware API (§13.3). | +| … with `title` or `abstract` | **the value the legacy projection is showing.** If the primary Manifestation has an override for that field (never `''`, by CHECK), the edit writes the override and advances `manifestation-field/[MF, f]`; otherwise it writes the Work's canonical value and advances `work-field/[W, f]`. The response shows the edit, and no hidden fallback changes silently. **An empty value on the override path clears the override back to inherit (NULL).** The legacy shape has no other way to say "inherit", and storing `''` would make a blank Version title. Changing the *canonical* value while an override exists, or choosing inherit versus value explicitly, needs the Version-aware API (§13.3). | | … with a Manifestation-owned field | the **primary** Manifestation. The editor shows the primary, so this is what the user sees. The response names the `manifestation_id` it wrote. | | … with `source_url` on a **non-video** Work (provenance/citation URL, a `SET_WORK_METADATA_FIELD` value) | the primary **Manifestation**'s `url`. This works whether or not the Work has an Asset. The existing video guard stays (work-source-identity.md). | | `SET_WORK_SOURCE` / video source identity (`source_kind`, `provider`, `provider_id`, `source_url` on a video Work) | the primary Asset's `external_stream` aggregate (`SET_WORK_SOURCE` semantics unchanged) | @@ -1748,6 +1749,7 @@ below. | --- | --- | --- | | `work-field/[W, f]` for M-owned `f` | `manifestation-field/[MF, f]` | Copy the revision to `origin_MF(W)` in the authority migration. | | `work-field/[W, f]` for W-owned `f` (status, title, abstract, author_text) | unchanged | — | +| (new) `manifestation-field/[MF, title\|abstract]` for overrides | new scope, revision 0 until first written | Overrides are created only by the Version-aware API. A **durable** legacy `SET_WORK_METADATA_FIELD(W, title\|abstract)` carries a base revision for `work-field/[W, f]`, which says nothing about the override. So while the primary Manifestation has an override for that field, the operation is **refused** with `FIELD_OVERRIDDEN_BY_VERSION` rather than routed. Online legacy `PATCH` follows §13.2 and advances the override's scope. | | `work-source/W` | `asset-source/AS` | Copy to `origin_AS(W)`. | | `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `origin_AS(W)`. Deletion tombstones are copied too, so resurrection protection survives. | | `work-person-role/[W, P, r]` | unchanged for W-scope; `manifestation-person-role/[MF, P, r]` for M-scope | Copy every scope row of an edition-scoped role type to `origin_MF(W)`, **tombstones included**. | From 303e1ec3a64d3b11dc96e164236cd2bb27957534 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:02:58 +0000 Subject: [PATCH 16/24] docs(#60): carry thumb_page revisions to asset-field scope; keep existing Version credit overrides on merge Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index b2697878..7e4ffc9c 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1337,9 +1337,12 @@ order, and all steps are one transaction: Only exactly identical citation rows are collapsed (§8.5). **Role collisions never lose a credit.** If the source and the target both have a Work-scoped `(person, role_type)` with *different* `credit_name` values, the - target's row is kept and the source spelling is preserved as a - `manifestation_credit_overrides` row on each moved Manifestation. Those - Versions keep citing the name they were credited with. The preview shows + target's row is kept, and each moved Manifestation keeps its **effective + pre-merge credit**. A moved Manifestation that already has its own + override for that `(person, role_type)`, for example from an earlier + merge, keeps it unchanged. Only moved Manifestations *without* one get a + new override carrying the source Work's spelling. No Version's displayed or + cited credit changes. The preview shows every such collision and lets the user pick a single spelling instead. Identical rows are collapsed. The user's choices from the preview are applied. @@ -1751,6 +1754,7 @@ below. | `work-field/[W, f]` for W-owned `f` (status, title, abstract, author_text) | unchanged | — | | (new) `manifestation-field/[MF, title\|abstract]` for overrides | new scope, revision 0 until first written | Overrides are created only by the Version-aware API. A **durable** legacy `SET_WORK_METADATA_FIELD(W, title\|abstract)` carries a base revision for `work-field/[W, f]`, which says nothing about the override. So while the primary Manifestation has an override for that field, the operation is **refused** with `FIELD_OVERRIDDEN_BY_VERSION` rather than routed. Online legacy `PATCH` follows §13.2 and advances the override's scope. | | `work-source/W` | `asset-source/AS` | Copy to `origin_AS(W)`. | +| `work-field/[W, thumb_page]` | `asset-field/[AS, thumb_page]` | Copy every scope row, tombstones included, to `origin_AS(W)`. A legacy `SET_WORK_METADATA_FIELD(W, thumb_page)` maps through `origin_AS(W)` like the other legacy operations. The same rule applies to any other Work field that becomes Asset-owned. | | `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `origin_AS(W)`. Deletion tombstones are copied too, so resurrection protection survives. | | `work-person-role/[W, P, r]` | unchanged for W-scope; `manifestation-person-role/[MF, P, r]` for M-scope | Copy every scope row of an edition-scoped role type to `origin_MF(W)`, **tombstones included**. | | notes, tags, folder, playlist, open | unchanged | — | @@ -1933,7 +1937,7 @@ exactly right. | **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; the direct guard and marker bypasses are refused; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | -| **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`). Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | +| **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`, `asset-field/[AS, thumb_page]`), tombstones included. Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | | **E. Manifestation authority** | vN+2: bibliographic fields owned by `manifestations`. `manifestation-field` scopes with carried revisions. `SET_WORK_METADATA_FIELD` mapped to `origin_MF(W)`. The `argument_sources` writer and sync accept `manifestation_id`, replacing the mirror trigger, and return `ARGUMENT_SOURCE_VERSION_REQUIRED` (§8.5). Identifier table populated from DOI/ISBN. | No | C | | **F. Role scope** | vN+3: role-type scope defaults, the uniqueness swap, `credits(M)`, and the role-sync scope for M-scoped roles. | No | E | | **G. Typed Version/File/Source API** | §13.3 read endpoints first, then mutations as canonical domain commands (#182/#199 pattern) with OpenAPI and openapi-core tests. New durable families declared per the sync guide. | API only | D, E, F | From e404705368c61acf68fd2f5748cb2d2f4e999393 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:05:39 +0000 Subject: [PATCH 17/24] docs(#60): reserve origin_AS(W) for Works with only Asset-bound revision tombstones Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 7e4ffc9c..f1e0dddd 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1477,13 +1477,29 @@ The backfill rules: - **Asset-creation predicate.** A Work gets an Asset when it has **any** Asset-owned value: a `file_path`, video identity, annotations, a non-empty `source_mime`, `thumb_url` or `thumb_page`, or a non-zero materialization - revision. `POST /api/works` / `add_work()` accept `source_mime`, + revision. It also gets one when it has **any durable revision state in a + scope that will become Asset-owned**, even if every current Asset-owned + value is empty: + - a `sync_entity_revisions` row for `work-field/[W, thumb_page]` (and any + other Work field scheduled to become Asset-owned); + - a `pdf-annotation/[W, …]` row, including tombstones of deleted + annotations; + - a `work-source/W` row. + + Tombstones and queued legacy operations must always be able to map to a + stable `origin_AS(W)`, and minting that Asset late in Slice D would make the + mapping appear late and lose deterministic identity. `POST /api/works` / `add_work()` accept `source_mime`, `thumb_url` and `thumb_page` without a file, so metadata-only rows can carry them. Those values need a canonical home when Asset authority lands, or JSON parity would break. - A Work that meets the predicate but has **no usable file or stream reference** gets a `managed_file` Asset with a NULL `storage_locator`. It - is a placeholder that owns those values. The projection shows it exactly as + is a placeholder that owns those values **and those revision scopes**. +- **After Slice A the rule keeps holding.** The mirror triggers create + `origin_AS(W)` the first time a Work without one gains any Asset-owned value + or annotation. That is necessarily before any Asset-bound revision or + tombstone can exist for it, because a tombstone can only follow a value. + Clearing values later never deletes the Asset. The projection shows it exactly as today (`file_path` empty: "No file attached"), and none of its values or annotations are dropped or left without an owner. - A Work with **none** of these values gets a Manifestation and **no** Asset. @@ -1934,7 +1950,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; the direct guard and marker bypasses are refused; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; the direct guard and marker bypasses are refused; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6a) Works whose only Asset-bound state is durable revision state get exactly one deterministic placeholder `origin_AS(W)`: `thumb_page` set and then cleared (only the revision tombstone remains), and an annotation deleted with no live annotation left. A Slice D test then copies those tombstones to the Asset scopes, and a queued legacy operation maps to that same Asset ID; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`, `asset-field/[AS, thumb_page]`), tombstones included. Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | From 781a36bbb905d827af46ffd0cd3d1196b7a9734d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:09:32 +0000 Subject: [PATCH 18/24] docs(#60): add work-source tombstone fixture to Slice A (6a) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index f1e0dddd..4edef3e3 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1950,7 +1950,7 @@ exactly right. | Slice | Content | User-visible? | Depends on | | --- | --- | --- | --- | -| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; the direct guard and marker bypasses are refused; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6a) Works whose only Asset-bound state is durable revision state get exactly one deterministic placeholder `origin_AS(W)`: `thumb_page` set and then cleared (only the revision tombstone remains), and an annotation deleted with no live annotation left. A Slice D test then copies those tombstones to the Asset scopes, and a queued legacy operation maps to that same Asset ID; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | +| **A. Entities + integrity layer + deterministic backfill + mirror triggers** | Migration vN (§12.3 step 1), with `db_schema.sql` updated to match. **Acceptance criteria:** (1) fresh and upgraded DBs have the same schema, including the composite FKs and triggers of §4.1, and `validate_current_schema` checks them; (2) a negative test for each §4.1 invariant (`works` primary and citation pointers; Manifestation primary Asset; roles, argument sources and annotations owners; cross-Work, mismatched-`work_id` and self `manifestation_relations`; moving one related endpoint without dropping the relation fails at COMMIT, while moving both in a merge keeps it); (3) legal-move cascade and pointer-target restrict tests, plus the §4.1 transition tests (move the only primary Asset out; move a non-primary Asset; move the only Manifestation out with the emptied Work deleted or merged; a leftover retirement marker fails COMMIT; the direct guard and marker bypasses are refused; rollback leaves all pointers and owner columns unchanged); (4) the integrity query is empty, and `PRAGMA foreign_key_check` adds no new violations after the backfill; (5) `source_mime` preserved; (6a) Works whose only Asset-bound state is durable revision state get exactly one deterministic placeholder `origin_AS(W)`: `thumb_page` set and then cleared (only the revision tombstone remains), an annotation deleted with no live annotation left, and a Work with a `work-source/W` revision whose video identity has since been cleared. `SET_WORK_SOURCE` refuses clearing a video, but the ordinary `PATCH` still accepts the source columns unvalidated (work-source-identity.md §2), so this state is reachable. A Slice D test then copies those tombstones to the Asset scopes, and queued legacy operations, including a `SET_WORK_SOURCE`, map to that same Asset ID; (6) for Arguments with no quarantined row, the `argument_sources` rebuild renumbers without changing the observed list or its revision, and pins rows that have pages. A fixture that loses an orphan citation shows the survivors renumbered and the `argument-sources` revision advanced. Fixtures for a live Work, a missing Person and a role row, **with and without** an existing role revision, show `get_roles_state()` reporting that scope as absent at a strictly newer revision; (7) a parity test (legacy columns = the new rows) over fixture libraries covering video, inferred-video, no-file, annotations-without-file, shared-basename rows, metadata-only rows with `source_mime`/`thumb_*`, and quarantined legacy FK-orphan rows; (8) no filesystem access and no backup creation. No readers change. | No | this design | | **B. Asset fingerprint pass + ingest hashing** | Bounded, resumable hashing pass (§12.4). Compute `ingest_sha256` on the upload/import stream **before** linearization, for new Assets. No duplicate UI yet. | No | A | | **C. Projection module** | `work_projection.legacy_work` / `legacy_work_summary`. Move readers family by family (detail, browse/recent, folder/person/playlist summaries, BibTeX via `citation_record`), each with JSON parity tests and a byte-identical BibTeX test. | No | A | | **D. Asset authority** | vN+1: locator, source aggregate, materialization revisions and `thumb_page` owned by `assets`. `annotations.asset_id` authoritative and NOT NULL. Text index keyed by Asset. Cleanup and backup audit count Asset locators. Scope and revision copies (`asset-source`, `asset-annotation`, `asset-field/[AS, thumb_page]`), tombstones included. Legacy operations mapped to `origin_AS(W)`. Browser last-page key migration. | No | C | From 747126a77feac51dd50ab0541eba0585101cc6ae Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:15:27 +0000 Subject: [PATCH 19/24] docs(#60): freeze inherited title/abstract on merge; conditional Asset on create Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 4edef3e3..21360495 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1323,9 +1323,17 @@ order, and all steps are one transaction: `work_retirement(source)` (§4.1 "Transitions"). 2. Release the source Work's primary and citation pointers. The trigger allows this only because of the retirement marker. -3. `UPDATE manifestations SET work_id = target` for the moved Manifestations. - The owner columns of their Assets, annotations, scoped roles and pinned - argument sources follow by cascade. +3. **Freeze inherited values, then move.** Before the move, each moved + Manifestation whose `title` or `abstract` override is NULL, and so + inherits from the source Work, gets the **source Work's current value** + written as an explicit override. This happens only where that value + differs from the target Work's, and never as `''`. The Version's + displayed and cited title and abstract therefore don't silently change to + the target's. This is the same rule as credit names in step 4. The preview + lists these, and the user may instead let a Version inherit the target's + values. Then `UPDATE manifestations SET work_id = target` for the moved + Manifestations. The owner columns of their Assets, annotations, scoped + roles and pinned argument sources follow by cascade. 4. Re-point Work-level rows to the target: - tags; - Work-scoped roles; @@ -1691,7 +1699,7 @@ for the fixture library. The legacy dict gains only **additive** fields: | … with a Manifestation-owned field | the **primary** Manifestation. The editor shows the primary, so this is what the user sees. The response names the `manifestation_id` it wrote. | | … with `source_url` on a **non-video** Work (provenance/citation URL, a `SET_WORK_METADATA_FIELD` value) | the primary **Manifestation**'s `url`. This works whether or not the Work has an Asset. The existing video guard stays (work-source-identity.md). | | `SET_WORK_SOURCE` / video source identity (`source_kind`, `provider`, `provider_id`, `source_url` on a video Work) | the primary Asset's `external_stream` aggregate (`SET_WORK_SOURCE` semantics unchanged) | -| `POST /api/works` | creates Work + Manifestation + Asset in one transaction | +| `POST /api/works` | creates Work + Manifestation in one transaction, plus an Asset **only when the §12.2 Asset-creation predicate holds** (a file, video identity, `source_mime`, `thumb_*`, …). A notes-only or metadata-only Work gets zero Assets, matching §4 and the backfill, so `asset_count` never reports a File or Source that doesn't exist. | | `POST /api/works/:id/pdf` (materialization) | the **working slot** of the primary Asset (§9.4). The client must send `asset_id` once more than one exists. Without it the request is refused (`ASSET_AMBIGUOUS`) rather than guessed. | | Annotation endpoints | the annotation's own Asset. A new annotation needs the displayed `asset_id`, which defaults to the primary only while exactly one PDF Asset exists. | | `/api/bibtex/:id` | `citation_record(citation_target(work))`, which is the primary while `citation_manifestation_id` is NULL (§8.3, §8.6) | From 4b749ea3a73b8bd3e16a4fa2020680ce8048dc4b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:18:44 +0000 Subject: [PATCH 20/24] docs(#60): merge freeze advances override scope; document empty-source case Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 21360495..98fbcdb3 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1331,7 +1331,15 @@ order, and all steps are one transaction: displayed and cited title and abstract therefore don't silently change to the target's. This is the same rule as credit names in step 4. The preview lists these, and the user may instead let a Version inherit the target's - values. Then `UPDATE manifestations SET work_id = target` for the moved + values. The freeze advances `manifestation-field/[MF, title|abstract]`. + **Accepted limitation, shown in the preview:** when the source Work's + value is *empty* (a NULL abstract, or an empty title if one exists) and + the target's is not, nothing can be frozen. `''` is forbidden and NULL + means "inherit", so the moved Version will show the target's value after + the merge. The preview shows this per Version as "will now show the + target's abstract", so the change is never silent. The user can accept it + or cancel the merge. No "explicitly empty" override state is added for + this rare case. Then `UPDATE manifestations SET work_id = target` for the moved Manifestations. The owner columns of their Assets, annotations, scoped roles and pinned argument sources follow by cascade. 4. Re-point Work-level rows to the target: @@ -1776,7 +1784,7 @@ below. | --- | --- | --- | | `work-field/[W, f]` for M-owned `f` | `manifestation-field/[MF, f]` | Copy the revision to `origin_MF(W)` in the authority migration. | | `work-field/[W, f]` for W-owned `f` (status, title, abstract, author_text) | unchanged | — | -| (new) `manifestation-field/[MF, title\|abstract]` for overrides | new scope, revision 0 until first written | Overrides are created only by the Version-aware API. A **durable** legacy `SET_WORK_METADATA_FIELD(W, title\|abstract)` carries a base revision for `work-field/[W, f]`, which says nothing about the override. So while the primary Manifestation has an override for that field, the operation is **refused** with `FIELD_OVERRIDDEN_BY_VERSION` rather than routed. Online legacy `PATCH` follows §13.2 and advances the override's scope. | +| (new) `manifestation-field/[MF, title\|abstract]` for overrides | new scope, revision 0 until first written | Overrides are created only by the Version-aware API and by `MERGE_WORKS` step 3 (freezing inherited values, §10.4). Both write paths advance `manifestation-field/[MF, title\|abstract]`. A **durable** legacy `SET_WORK_METADATA_FIELD(W, title\|abstract)` carries a base revision for `work-field/[W, f]`, which says nothing about the override. So while the primary Manifestation has an override for that field, the operation is **refused** with `FIELD_OVERRIDDEN_BY_VERSION` rather than routed. Online legacy `PATCH` follows §13.2 and advances the override's scope. | | `work-source/W` | `asset-source/AS` | Copy to `origin_AS(W)`. | | `work-field/[W, thumb_page]` | `asset-field/[AS, thumb_page]` | Copy every scope row, tombstones included, to `origin_AS(W)`. A legacy `SET_WORK_METADATA_FIELD(W, thumb_page)` maps through `origin_AS(W)` like the other legacy operations. The same rule applies to any other Work field that becomes Asset-owned. | | `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `origin_AS(W)`. Deletion tombstones are copied too, so resurrection protection survives. | From 0925f6557eb4779c22f8e1e344c70399c33de682 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:24:11 +0000 Subject: [PATCH 21/24] docs(#60): freeze inherited metadata on MOVE_MANIFESTATION; compare like-for-like legacy hashes Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 98fbcdb3..2ec5ead3 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1223,9 +1223,18 @@ Asset AS-… (one File in the UI; owns annotations, page state, thumbnails) it never changes. - **Legacy Assets** never had their original bytes preserved, so their `ingest_sha256` stays **NULL**. PRKS must not invent it from bytes that have - since been materialized. Exact-duplicate detection against legacy files uses - `content_sha256` and is documented as weaker: it matches an unannotated legacy - file, and misses one whose bytes PRKS has rewritten. + since been materialized. **Legacy fallback compares like with like.** + Legacy `content_sha256` describes *working* bytes, which ingestion has + already linearized (`store_new_managed_pdf_*` → + `_finalize_exclusive_managed_create`). Comparing it with a new file's + pre-linearization `ingest_sha256` would miss even an identical upload. So + the fallback compares a legacy `content_sha256` with the new upload's + **post-linearization, pre-annotation** hash, which is the new Asset's + initial `content_sha256`, computed in the same ingestion step. This is + documented as weaker. It matches an unannotated legacy file only if the + linearizer produces the same bytes for the same input (same qpdf version + and deterministic IDs). It misses any file whose bytes PRKS has since + rewritten. - **When a separate Asset is correct:** a different file the user brings in (another scan, another download, a copy annotated elsewhere, a re-captured snapshot). Such Assets have their own annotations and page state, and the @@ -1275,7 +1284,7 @@ of them merges anything by itself**. | Level | Evidence | Confidence | Allowed automation | | --- | --- | --- | --- | -| Exact Asset duplicate | equal `ingest_sha256` (or `content_sha256` for legacy files) | Certain for the content | **Warn before import**, showing the existing Work. Never auto-link or auto-delete. | +| Exact Asset duplicate | equal `ingest_sha256`. For legacy files, the legacy `content_sha256` is compared with the new upload's post-linearization hash (§9.4) | Certain for the content | **Warn before import**, showing the existing Work. Never auto-link or auto-delete. | | Identifier match | equal normalized DOI, ISBN-13, or arXiv ID (versionless → same work, versioned → same revision) | Strong for "same publication", but **not** proof of "same Work" (a DOI identifies one form) | Suggest. The user decides. | | Metadata similarity | normalized title, creator family names, year, publisher/journal | Weak | Suggest in a review list only. RapidFuzz may rank candidates (#42/#179). **Never decides.** | @@ -1312,7 +1321,7 @@ before mutation. None of them is silent. | Operation | Effect | IDs | | --- | --- | --- | | `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `sync_work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | -| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | +| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. **Inherited metadata is frozen before the move**, by the same rule as `MERGE_WORKS` step 3. A NULL `title` or `abstract` override whose old-Work value differs from the destination's is written as an explicit override. The empty-source case is shown in the preview. The Version's displayed and cited metadata therefore never changes silently, and the preview lists these along with the dropped relations. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | | `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the same transaction re-points `primary_asset_id` to a remaining active sibling, or clears it when none remains. A Manifestation with zero Assets is a valid final state. **On the destination side**, if the destination Manifestation had no active Asset, which means its `primary_asset_id` is NULL, the moved active Asset becomes its primary in the same transaction. Otherwise the destination's primary is unchanged. The deferred `NO ACTION` FK and the command's final-state check verify both sides at COMMIT. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | @@ -1974,7 +1983,7 @@ exactly right. | **F. Role scope** | vN+3: role-type scope defaults, the uniqueness swap, `credits(M)`, and the role-sync scope for M-scoped roles. | No | E | | **G. Typed Version/File/Source API** | §13.3 read endpoints first, then mutations as canonical domain commands (#182/#199 pattern) with OpenAPI and openapi-core tests. New durable families declared per the sync guide. | API only | D, E, F | | **H. Versions UI** | "Add another version / file", set primary, open a specific Version or File, and a version picker for Copy citation, all behind §13.4's progressive disclosure. Follow `DESIGN.md` for Work detail composition. #61's secondary view builds on this. | **Yes** | G | -| **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256` (and `content_sha256` for legacy files). Offers the four choices in §10.2. | Yes | B, H | +| **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256`. For legacy files it compares `content_sha256` with the upload's post-linearization hash (§9.4). Offers the four choices in §10.2. | Yes | B, H | | **J. Identifier candidates + decline** | Normalized DOI/ISBN/arXiv candidates, a review list, and `duplicate_decisions`. | Yes | E, I | | **K. Merge / move workflow** | Command-level versions of the §4.1 transition tests, including each `empty_source` outcome. Ships `manifestation_credit_overrides` (§7.2) if it does not exist yet, and tests a credit-spelling collision. `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `sync_work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | | **L. RapidFuzz evaluation** | A separate research/evaluation issue (dependency review, ranking quality on synthetic data). It only ranks; it never decides. | — | J | From 8cf7454cf01930668dd0ca163d83c36913e405ad Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:27:51 +0000 Subject: [PATCH 22/24] docs(#60): MOVE_MANIFESTATION credit freeze + credit-delta preview; complete override writer list Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 2ec5ead3..f85ad767 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -863,8 +863,9 @@ stored per row, so a user can override a default. through `manifestation_credit_overrides(manifestation_id, person_id, role_type, credit_name)`. `credits(M)` (§7.5) applies these overrides. The backfill writes none, and no UI for editing them is needed early. **The - table is nevertheless required by the time `MERGE_WORKS` ships (Slice K)**, - because a merge preserves colliding credit spellings there (§10.4, step 4). + table is nevertheless required by the time `MERGE_WORKS` and + `MOVE_MANIFESTATION` ship (Slice K)**, because both preserve colliding credit + spellings there (§10.4). Without it, a merge would have to drop a credit. ### 7.3 Backfill rule @@ -1321,7 +1322,7 @@ before mutation. None of them is silent. | Operation | Effect | IDs | | --- | --- | --- | | `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `sync_work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | -| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. **Inherited metadata is frozen before the move**, by the same rule as `MERGE_WORKS` step 3. A NULL `title` or `abstract` override whose old-Work value differs from the destination's is written as an explicit override. The empty-source case is shown in the preview. The Version's displayed and cited metadata therefore never changes silently, and the preview lists these along with the dropped relations. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | +| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. **Inherited metadata is frozen before the move**, by the same rule as `MERGE_WORKS` step 3. A NULL `title` or `abstract` override whose old-Work value differs from the destination's is written as an explicit override. The empty-source case is shown in the preview. **Credits:** for a person holding the same Work-scoped `(person, role_type)` on both Works under a different `credit_name`, the old spelling is frozen as a `manifestation_credit_overrides` row, by the same rule as `MERGE_WORKS` step 4 (an existing override is kept). The destination Work's *set* of Work-scoped creators does apply after the move, because moving a Version asserts that it belongs to that intellectual work. So Authors present on only one side cannot be frozen. The preview shows the Version's **credit delta** (added and removed creators). The preview lists the frozen values, the credit delta and the dropped relations. Title and abstract never change silently, and credit changes are always shown before the user confirms. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | | `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the same transaction re-points `primary_asset_id` to a remaining active sibling, or clears it when none remains. A Manifestation with zero Assets is a valid final state. **On the destination side**, if the destination Manifestation had no active Asset, which means its `primary_asset_id` is NULL, the moved active Asset becomes its primary in the same transaction. Otherwise the destination's primary is unchanged. The deferred `NO ACTION` FK and the command's final-state check verify both sides at COMMIT. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | @@ -1793,7 +1794,7 @@ below. | --- | --- | --- | | `work-field/[W, f]` for M-owned `f` | `manifestation-field/[MF, f]` | Copy the revision to `origin_MF(W)` in the authority migration. | | `work-field/[W, f]` for W-owned `f` (status, title, abstract, author_text) | unchanged | — | -| (new) `manifestation-field/[MF, title\|abstract]` for overrides | new scope, revision 0 until first written | Overrides are created only by the Version-aware API and by `MERGE_WORKS` step 3 (freezing inherited values, §10.4). Both write paths advance `manifestation-field/[MF, title\|abstract]`. A **durable** legacy `SET_WORK_METADATA_FIELD(W, title\|abstract)` carries a base revision for `work-field/[W, f]`, which says nothing about the override. So while the primary Manifestation has an override for that field, the operation is **refused** with `FIELD_OVERRIDDEN_BY_VERSION` rather than routed. Online legacy `PATCH` follows §13.2 and advances the override's scope. | +| (new) `manifestation-field/[MF, title\|abstract]` for overrides | new scope, revision 0 until first written | Overrides are created by three paths: the Version-aware API, `MERGE_WORKS` step 3, and `MOVE_MANIFESTATION`, where both commands freeze inherited values (§10.4). All three advance `manifestation-field/[MF, title\|abstract]`. A **durable** legacy `SET_WORK_METADATA_FIELD(W, title\|abstract)` carries a base revision for `work-field/[W, f]`, which says nothing about the override. So while the primary Manifestation has an override for that field, the operation is **refused** with `FIELD_OVERRIDDEN_BY_VERSION` rather than routed. Online legacy `PATCH` follows §13.2 and advances the override's scope. | | `work-source/W` | `asset-source/AS` | Copy to `origin_AS(W)`. | | `work-field/[W, thumb_page]` | `asset-field/[AS, thumb_page]` | Copy every scope row, tombstones included, to `origin_AS(W)`. A legacy `SET_WORK_METADATA_FIELD(W, thumb_page)` maps through `origin_AS(W)` like the other legacy operations. The same rule applies to any other Work field that becomes Asset-owned. | | `pdf-annotation/[W, ann]` | `asset-annotation/[AS, ann]` | Copy to `origin_AS(W)`. Deletion tombstones are copied too, so resurrection protection survives. | From 7b375cecafc7beba4022c99fabe1df8d422d49a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:29:52 +0000 Subject: [PATCH 23/24] docs(#60): lazily create origin_AS for first queued Asset edit; bump argument-sources on MOVE_MANIFESTATION Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index f85ad767..4ea6e30e 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1322,7 +1322,7 @@ before mutation. None of them is silent. | Operation | Effect | IDs | | --- | --- | --- | | `MERGE_WORKS(source → target)` | Moves all source Manifestations (with their Assets and annotations) under the target. Unions tags. Folder, playlist and status: target wins unless the user picks. Research/Private Notes: **the user chooses** (keep target, keep source, or concatenate with a visible separator); never silently concatenated. Roles are unioned, with duplicates collapsed. Argument sources and research mentions are re-pointed, and any citation-identity collision is shown in the preview (§8.5). `last_opened_at` is max. | The source `W-…` gets `sync_work_lifecycle(state = merged, target)`. Old links, tabs and `[[W-…]]` **reads** follow the redirect. **Every** pending operation naming the source is refused with `WORK_MERGED` + `target_work_id` and is never applied to the target. The client may re-apply the user's intent explicitly (§14.2). | -| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. **Inherited metadata is frozen before the move**, by the same rule as `MERGE_WORKS` step 3. A NULL `title` or `abstract` override whose old-Work value differs from the destination's is written as an explicit override. The empty-source case is shown in the preview. **Credits:** for a person holding the same Work-scoped `(person, role_type)` on both Works under a different `credit_name`, the old spelling is frozen as a `manifestation_credit_overrides` row, by the same rule as `MERGE_WORKS` step 4 (an existing override is kept). The destination Work's *set* of Work-scoped creators does apply after the move, because moving a Version asserts that it belongs to that intellectual work. So Authors present on only one side cannot be frozen. The preview shows the Version's **credit delta** (added and removed creators). The preview lists the frozen values, the credit delta and the dropped relations. Title and abstract never change silently, and credit changes are always shown before the user confirms. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | +| `MOVE_MANIFESTATION(M → Work)` | "This is really a Version of that Work." One `UPDATE manifestations SET work_id` re-keys everything the Manifestation owns by cascade (§4.1): its Assets' and annotations' `work_id`, its scoped roles, and its pinned argument sources. Every Argument that has a citation pinned to the moved Version gets its `argument-sources/` revision **advanced in the same transaction**, exactly as `MERGE_WORKS` does, so a client holding the pre-move list conflicts instead of overwriting or re-pinning the moved citation. None of these can collide, because each identity includes the unchanged `MF-…` ID. If the Manifestation is a pointer target of the old Work and siblings remain, the pointer is re-pointed to a sibling first. If it is the old Work's **only** Manifestation, the command requires an explicit `empty_source` outcome, `delete` or `merge`, and retires the old Work in the same transaction (§4.1 "Transitions"). No placeholder Version is ever created. **Inherited metadata is frozen before the move**, by the same rule as `MERGE_WORKS` step 3. A NULL `title` or `abstract` override whose old-Work value differs from the destination's is written as an explicit override. The empty-source case is shown in the preview. **Credits:** for a person holding the same Work-scoped `(person, role_type)` on both Works under a different `credit_name`, the old spelling is frozen as a `manifestation_credit_overrides` row, by the same rule as `MERGE_WORKS` step 4 (an existing override is kept). The destination Work's *set* of Work-scoped creators does apply after the move, because moving a Version asserts that it belongs to that intellectual work. So Authors present on only one side cannot be frozen. The preview shows the Version's **credit delta** (added and removed creators). The preview lists the frozen values, the credit delta and the dropped relations. Title and abstract never change silently, and credit changes are always shown before the user confirms. Relations to Manifestations of the old Work are dropped, with a preview. | `MF-…` unchanged. | | `MOVE_ASSET(A → Manifestation)` | "This file is another scan of that edition." It sets the Asset's `manifestation_id` and `work_id` together (the composite FK requires the pair to match). Its annotations keep their `asset_id` and follow the new `work_id` by cascade. If the Asset is its Manifestation's primary, the same transaction re-points `primary_asset_id` to a remaining active sibling, or clears it when none remains. A Manifestation with zero Assets is a valid final state. **On the destination side**, if the destination Manifestation had no active Asset, which means its `primary_asset_id` is NULL, the moved active Asset becomes its primary in the same transaction. Otherwise the destination's primary is unchanged. The deferred `NO ACTION` FK and the command's final-state check verify both sides at COMMIT. | `AS-…` unchanged. | | `DECLINE_DUPLICATE(a, b)` | Records "not a duplicate". | `duplicate_decisions(entity_type, low_id, high_id, decision, decided_at)`. | @@ -1818,6 +1818,18 @@ retargeted: The same rule applies to annotation operations (→ `origin_AS(W)`) and to `SET_WORK_SOURCE`. +**First Asset-bound edit with no origin Asset yet.** The server cannot see +operations still queued in a browser. So a live Work can reach Slice D with +no Asset-owned value, no Asset-bound revision and therefore no +`origin_AS(W)`, while a client still holds a queued first edit (for example a +`thumb_page` update at base revision 0). Such an operation creates the +placeholder `origin_AS(W)` **lazily, in the same transaction, just before +routing**. It uses the deterministic ID `AS-uuid5(…, "asset:" + W)` that the +backfill would have used, so the result is the same as if Slice A had +reserved it. The fresh scope starts at revision 0, so the base-0 edit applies +normally. Only a live, unmerged `W` gets this. A merged or deleted Work returns +its usual result. + **New durable families** (Manifestation fields, M-scoped roles, set-primary, relations) each declare their shape per "Adding a family" in `docs/local-first-sync.md`. Asset creation stays online-only (I13). Durable From 6046c5a82ad4de5cd2c196f864f96e494e872469 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 15:30:39 +0000 Subject: [PATCH 24/24] docs(#60): drop non-reproducible post-linearization legacy fallback; keep only no-false-positive legacy match Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016PBTe3CeL5KTUqEj9zJ6Yz --- docs/work-identity-model.md | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/docs/work-identity-model.md b/docs/work-identity-model.md index 4ea6e30e..b56eec45 100644 --- a/docs/work-identity-model.md +++ b/docs/work-identity-model.md @@ -1224,18 +1224,26 @@ Asset AS-… (one File in the UI; owns annotations, page state, thumbnails) it never changes. - **Legacy Assets** never had their original bytes preserved, so their `ingest_sha256` stays **NULL**. PRKS must not invent it from bytes that have - since been materialized. **Legacy fallback compares like with like.** - Legacy `content_sha256` describes *working* bytes, which ingestion has - already linearized (`store_new_managed_pdf_*` → - `_finalize_exclusive_managed_create`). Comparing it with a new file's - pre-linearization `ingest_sha256` would miss even an identical upload. So - the fallback compares a legacy `content_sha256` with the new upload's - **post-linearization, pre-annotation** hash, which is the new Asset's - initial `content_sha256`, computed in the same ingestion step. This is - documented as weaker. It matches an unannotated legacy file only if the - linearizer produces the same bytes for the same input (same qpdf version - and deterministic IDs). It misses any file whose bytes PRKS has since - rewritten. + since been materialized. **Legacy Assets do not take part reliably in + exact-byte duplicate detection.** Their only hash, `content_sha256`, + describes working bytes that ingestion has already linearized. + `backend/pdf_linearize.py` runs `qpdf --linearize ` **without** + `--deterministic-id`, and qpdf documents that such output is not + byte-for-byte reproducible (its document IDs have a random element). So the + same original PDF can produce different working bytes on every ingestion. A + "post-linearization hash" of a new upload therefore cannot be matched + against a legacy file. Adding `--deterministic-id` for future uploads would + not help either, because historical files were linearized under the old + invocation. The design therefore makes only one, no-false-positive, check: + - **Opportunistic legacy match.** If a legacy `content_sha256` equals a new + upload's raw `ingest_sha256`, the stored legacy bytes *are* the uploaded + bytes. This happens when linearization was skipped or disabled and the + file was never annotated. It is reported as an exact duplicate, because + equal hashes cannot be wrong. + - **Otherwise** a legacy file is found only through identifier and + metadata candidates (§10.3), which are suggestions, not exact matches. + - New Assets always carry `ingest_sha256`, computed on the raw incoming + stream, which is deterministic by definition. They take part fully. - **When a separate Asset is correct:** a different file the user brings in (another scan, another download, a copy annotated elsewhere, a re-captured snapshot). Such Assets have their own annotations and page state, and the @@ -1285,7 +1293,7 @@ of them merges anything by itself**. | Level | Evidence | Confidence | Allowed automation | | --- | --- | --- | --- | -| Exact Asset duplicate | equal `ingest_sha256`. For legacy files, the legacy `content_sha256` is compared with the new upload's post-linearization hash (§9.4) | Certain for the content | **Warn before import**, showing the existing Work. Never auto-link or auto-delete. | +| Exact Asset duplicate | equal `ingest_sha256`. For legacy files there is only an opportunistic match: legacy `content_sha256` equal to the upload's raw `ingest_sha256`, which has no false positives. Legacy files otherwise have no reliable exact-match coverage (§9.4) | Certain for the content | **Warn before import**, showing the existing Work. Never auto-link or auto-delete. | | Identifier match | equal normalized DOI, ISBN-13, or arXiv ID (versionless → same work, versioned → same revision) | Strong for "same publication", but **not** proof of "same Work" (a DOI identifies one form) | Suggest. The user decides. | | Metadata similarity | normalized title, creator family names, year, publisher/journal | Weak | Suggest in a review list only. RapidFuzz may rank candidates (#42/#179). **Never decides.** | @@ -1996,7 +2004,7 @@ exactly right. | **F. Role scope** | vN+3: role-type scope defaults, the uniqueness swap, `credits(M)`, and the role-sync scope for M-scoped roles. | No | E | | **G. Typed Version/File/Source API** | §13.3 read endpoints first, then mutations as canonical domain commands (#182/#199 pattern) with OpenAPI and openapi-core tests. New durable families declared per the sync guide. | API only | D, E, F | | **H. Versions UI** | "Add another version / file", set primary, open a specific Version or File, and a version picker for Copy citation, all behind §13.4's progressive disclosure. Follow `DESIGN.md` for Work detail composition. #61's secondary view builds on this. | **Yes** | G | -| **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256`. For legacy files it compares `content_sha256` with the upload's post-linearization hash (§9.4). Offers the four choices in §10.2. | Yes | B, H | +| **I. Exact-duplicate warning at ingestion** | Uses `ingest_sha256`, plus the opportunistic legacy match (legacy `content_sha256` = upload `ingest_sha256`, §9.4). **Stability test:** the same source PDF ingested several times, with linearization on, yields the same `ingest_sha256` every time while the working bytes and `content_sha256` differ. That proves the comparison key does not depend on qpdf's nondeterminism. A test also checks that an unlinearized, unannotated legacy file matches opportunistically, and that a linearized one is not claimed as an exact match. Offers the four choices in §10.2. | Yes | B, H | | **J. Identifier candidates + decline** | Normalized DOI/ISBN/arXiv candidates, a review list, and `duplicate_decisions`. | Yes | E, I | | **K. Merge / move workflow** | Command-level versions of the §4.1 transition tests, including each `empty_source` outcome. Ships `manifestation_credit_overrides` (§7.2) if it does not exist yet, and tests a credit-spelling collision. `MERGE_WORKS` (in the §10.4 transaction order), `MOVE_MANIFESTATION` and `MOVE_ASSET` with previews, `sync_work_lifecycle` read redirects, and `WORK_MERGED` refusals plus client reconciliation (§14.2). Coordinate with #57. | Yes | J | | **L. RapidFuzz evaluation** | A separate research/evaluation issue (dependency review, ranking quality on synthetic data). It only ranks; it never decides. | — | J |