diff --git a/documents/obstacles/session-residue.md b/documents/obstacles/session-residue.md new file mode 100644 index 0000000..4b0b23f --- /dev/null +++ b/documents/obstacles/session-residue.md @@ -0,0 +1,29 @@ +--- +authors: [mathias_soeholm] +--- + +# Session Residue + +## Description + +AI writes artifacts relative to the session that produced them, not as standalone descriptions of the current state. Reversed decisions, superseded approaches and user corrections stay in the generated text. The content is correct, but it describes the current state by comparing it with earlier states that the reader never saw. + +For example, a session first puts a `/compatibility` prefix on all endpoint paths, then removes it. The final plan file lists the endpoint paths and adds "(does not include the `/compatibility` prefix)". A reader of the plan has never seen that prefix, so the note only raises the question of why it is there. + +## Root Causes + +LLMs generate text from the current request and the conversation history. A correction adds to that context; it does not erase earlier approaches or responses. Those details can continue to influence the artifact. + +The model also writes for the wrong reader. It treats the artifact as a reply to the latest correction, so the text answers the user in the session instead of a future reader who has only the result. + +In [_LLMs Get Lost in Multi-Turn Conversation_](https://arxiv.org/html/2505.06120v1), Appendix F.2, researchers found that even entirely correct Python programs and SQL queries were longer when requirements arrived across multiple turns. They attribute this "answer bloat" to models retaining earlier assumptions and relying too heavily on previous answer attempts. + +This is consistent with Session Residue, though the study did not directly test unnecessary historical references in documentation or code comments. + +## Impact + +- Misleads the reader, who looks for a reason the earlier state matters +- Goes stale fast: "now", "instead of" and "no longer" become wrong after the next change +- Comments describe the change, not the code. This is the "comment as changelog" smell ("Journal Comments" in _Clean Code_); the commit message and pull request already record the change +- Each change in the session can leave another trace, so the noise builds up +- When an agent later reads the artifact, the residue puts the rejected option back into its context, causing Negative Bleedthrough diff --git a/documents/relationships.mmd b/documents/relationships.mmd index a1567b5..c804cf0 100644 --- a/documents/relationships.mmd +++ b/documents/relationships.mmd @@ -204,4 +204,9 @@ graph LR obstacles/compliance-bias -->|causes| anti-patterns/silent-misalignment obstacles/black-box-ai -->|causes| anti-patterns/silent-misalignment obstacles/keeping-up -->|causes| anti-patterns/flying-blind - obstacles/keeping-up -->|causes| anti-patterns/big-chunk-working \ No newline at end of file + obstacles/keeping-up -->|causes| anti-patterns/big-chunk-working + patterns/memory-grooming -->|solves| obstacles/session-residue + patterns/noise-cancellation -->|solves| obstacles/session-residue + patterns/decision-guards -->|solves| obstacles/session-residue + obstacles/session-residue -->|related| obstacles/excess-verbosity + obstacles/session-residue -->|causes| obstacles/negative-bleedthrough diff --git a/website/config/authors.yaml b/website/config/authors.yaml index 5b791a1..bbbdb6c 100644 --- a/website/config/authors.yaml +++ b/website/config/authors.yaml @@ -29,3 +29,6 @@ svetkis: aino_vonge_corry: name: Aino Vonge Corry github: apaipi +mathias_soeholm: + name: Mathias Juhl Søholm + github: mathiassoeholm diff --git a/website/tests/unit/completeCatalog.test.tsx b/website/tests/unit/completeCatalog.test.tsx index 0749f8b..20553fb 100644 --- a/website/tests/unit/completeCatalog.test.tsx +++ b/website/tests/unit/completeCatalog.test.tsx @@ -4,6 +4,7 @@ import PatternCatalogPage from '@/app/pattern-catalog/page' import { COMPLETE_CATALOG_TEST_IDS } from '@/app/pattern-catalog/test-ids' import { PATTERN_CATALOG_GROUPS } from '@/app/pattern-catalog/constants' import { getAllPatterns } from '@/lib/markdown' +import { getAuthorById } from '@/lib/authors' import { useRouter } from 'next/navigation' function setMobileViewport() { @@ -173,8 +174,9 @@ describe('PatternCatalogPage', () => { ).toBeInTheDocument() expect(within(detailPane).getByText(/Description/i)).toBeInTheDocument() expect(within(detailPane).getByText(/Documented by/i)).toBeInTheDocument() + const authorName = getAuthorById(target.authors![0])!.name expect( - within(detailPane).getByRole('link', { name: /Lada Kesseler's avatar Lada Kesseler/i }) + within(detailPane).getByRole('link', { name: `${authorName}'s avatar ${authorName}` }) ).toBeInTheDocument() expect(within(detailPane).queryByText(/Open full entry/i)).not.toBeInTheDocument() })