Repository navigation
feat: make a task answer whether its facts come from outside the repo - #108
Merged
Merged
Conversation
`hooks/task approve` now reads a `## External Facts` section in `prd.md`
and refuses four shapes: no answer, an answer that is neither `no` nor
`yes`, `no` without a reason, and `yes` without a sourced conclusion.
The rule is not "search more". Measured 2026-10-31 over 94 archived tasks:
`## Reference Pointers` holds 382 bullets and not one is a URL, 3 of 94
tasks have a `reference/` directory, and the archive's only four
third-party URLs sit in a single task. None of that can show whether
TaskFlow suppresses search, because most work here is internal and should
cite nothing. So the field does not count searches -- it records the
judgement, which is the one thing that can be checked. A `yes` is a
commitment to go and look: it is refused until a source and the
conclusion that source changed are both named. A `no` is refused until it
says what puts the conclusion inside the repository, because an
unexplained `no` is the default the field exists to displace.
`promote` writes the section unanswered on purpose, so a new task cannot
be approved until someone answers it. That is the half that makes the
field mandatory rather than merely available.
`capability-evidence` still counts only `Skill`/`Agent`/`mcp__*`, so
`WebSearch`/`WebFetch` neither satisfy nor fail the capability gate. That
separation is 2026-09-27's R12 and it is unchanged -- letting a built-in
tool satisfy the gate would make it passable by searching once.
Three things the measurement settled that the plan had to absorb:
- the check belongs in `prd.md` alone. Whether evidence is gathered at
all is a requirements question; the Plan's job is how, not whether.
- `SKILL.md:54`'s `inspect` enumeration is not extended. Its eight
entries are "open the repository and look" -- adding a judgement
question there would dress a decision up as an inspection.
- already-archived tasks are not backfilled. `achieved/` is read-only
in place, as it was for the `done`-only rule.
Six mutations, each reddening its own named assertion and each restored
byte-identically: dropping the `no`-reason check, allowing zero sources,
not validating the URL, not requiring "what it settled", accepting an
answer outside `no`/`yes`, and passing an empty answer. One line the
first version carried turned out to be dead -- it refused an empty
section before a `case` arm that refuses it with the same message -- and
was removed rather than left as decoration.
… record
v1 shipped a `## External Facts` field that recorded a judgement. It was
an answer to the wrong question. The observation behind this task was
never "the field is missing" -- it was that an Agent working inside
TaskFlow tends not to look outside the repository at all. A field a model
can satisfy by writing `no` does not touch that, and the v1 PRD said so
in its own risk section before going ahead anyway.
What actually pulls attention inward, each at a line:
- `SKILL.md`'s `inspect` enumeration lists seven sources, all inside the
repository or the Agent's own state.
- `summarize-state` opens a session with internal conclusions.
- `prompt-reminder` speaks every turn, about task state only.
- `capability-evidence` counts `Skill`/`Agent`/`mcp__*` and nothing
else, so a web lookup is invisible to the whole workflow.
So the question now gets asked at the three points where its answer can
change: before Phase 1 converges, at the Phase 3 spec decision -- where
defining an interface or a compatibility boundary is exactly when "only
our own code" stops being true -- and once more before approval, against
Steps that did not exist when the first answer was given. The `inspect`
enumeration is deliberately not extended: its entries are "open the repo
and look", and a judgement question among them would dress a decision up
as an inspection. A smoke assertion counts the entries so a ninth cannot
be added quietly.
Lookups are now visible, and the isolation is the point of the design.
`hooks/external-evidence` records `WebSearch`/`WebFetch` into
`taskflow/external`; `capability-evidence` is untouched and still refuses
to count built-in tools, so a lookup can never release a stage (R12,
2026-09-27). The two predicates live in two files with two consumers
because one `case` with a shared fall-through would put the two meanings
back together. The smoke suite proves the separation rather than
asserting it in prose: a task holding two real lookups and no invocation
must still be refused, and refused for the missing stage.
`approve` reads the external store for the opposite purpose -- a `yes`
must have every `- Source:` backed by a recorded lookup, or the section
is a form filled in from memory. `achieved/2026-09-08-runtime-hooks` is
the case in point: it names Anthropic's hooks reference and Codex's
hooks guide in prose, with no URL and no date, in a repository with no
way to tell whether either was opened. A `no` must now point at where
the work is settled -- a backticked path or `path:line` -- because "this
is an internal change" is the sentence an unexamined `no` produces.
Four mutations, each reddening its own named assertion: dropping the
lookup-backing check, never failing on an unbacked source, dropping the
`no` pointer requirement, and adding a ninth `inspect` entry. Each was
restored byte-identically.
Two defects the smoke suite caught while this was being written, both
worth the fixture that found them:
- `a|b` recorded as `search|a|b` -- one record with a forged field
boundary. The assertion compares the whole line, not the count. The
fix then ate the separator itself by running `tr` over the joined
string, which the same assertion caught.
- the isolation fixture shared a capability store the earlier cases had
already pushed out of step, so `approve` refused for the wrong
reason. It gets its own worktree now.
Still open, and stated rather than papered over: a model that answers
`no` and writes a plausible pointer gets through. Enforcing otherwise
would mean demanding external evidence from tasks that genuinely need
none, which would make most of this repository's work lie. The field
records a decision and the record makes it checkable; it does not make
the decision honest.
The archive was being run after the merge, which forces it onto no branch: the moved directory and the rewritten Todo entry have nowhere to land except a push to the base outside any pull request, or a second pull request carrying only bookkeeping. `CONTRIBUTING.md` forbids both, in the same paragraph that requires a task's documents to ride in its own pull request -- so the practice was contradicting the rule it was meant to follow, and the rule already covered the archive by name. The mistake came from a message, not a document. `hooks/task complete` prints a teardown line that waits for the merge, and that condition is about `git worktree remove` -- deleting a directory that may still hold the only copy of something. Nothing about the archive transaction needs the merge: `complete` reads the working tree, moves the directory and rewrites the Todo entry, and never consults the base branch. Reading one as the other is what produced a second pull request for this repository twice. Both statements now say which is which. `CONTRIBUTING.md` states the order -- finish, `complete`, commit code and documents and archive together, push, open the pull request -- and names the two ways of getting it backwards and why each is forbidden. `SKILL.md` states it where the command is defined, and says the teardown line is about the worktree rather than about when to archive. This commit is the rule applied to itself: the archive for `2026-10-31-external-facts` is in it, alongside the code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An Agent working inside TaskFlow tends not to look outside the repository. This change asks it where the answer comes from at the three points where that answer can change, records the lookups it makes, and refuses a claim to have looked that nothing backs.
v1 of this PR (still in the branch history, as
96044df) shipped only the field — a## External Factssectionapprovevalidates. That recorded a judgement without touching the tendency to writenoreflexively, and the v1 PRD said so in its own risk section. The user's correction on 2026-11-01 is what this PR now answers. v1's documents are archived underTaskFlowDocs/2026-10-31-external-facts/old/v1/.TaskFlow traceability
TaskFlowDocs/2026-10-31-external-facts/(v2)hooks/external-evidence(new),hooks/hooks.json,hooks/task,hooks/smoke-test,hooks/README.md,skills/taskflow/SKILL.md,skills/taskflow/references/{artifacts,runtime}.md,CHANGELOG.md,CONTRIBUTING.md(archive ordering), and the archive of this task itselfmainhkwuks/TaskFlowWhy a field was not enough
Every source this workflow points an Agent at is inside the repository:
SKILL.md:54inspectlists seven sources — all internalsummarize-stateprompt-remindercapability-evidenceSkill/Agent/mcp__*and nothing else, so a web lookup is invisibleNothing in that list prompts the question, so a
nowritten by default is indistinguishable from one reached by thinking.What the three pause points do
Before Phase 1 converges, at the Phase 3 spec decision — defining an interface or a compatibility boundary is exactly when "only our own code" stops being true — and once more before approval, against Steps that did not exist when the first answer was given.
The
inspectenumeration is deliberately not extended. Its entries are "open the repo and look"; a judgement question among them would dress a decision up as an inspection. A smoke assertion counts the entries so a ninth cannot be added quietly.The isolation, which is the design
capability-evidenceis untouched, so a built-in tool still cannot satisfy the gate (R12, 2026-09-27). The two predicates live in two files with two consumers: onecasewith a shared fall-through would put the meanings back together. The suite proves the separation rather than asserting it — a task holding two real lookups and no invocation must still be refused, and refused for the missing stage.approvereadsexternalfor the opposite purpose: ayesneeds every- Source:backed by a recorded lookup.achieved/2026-09-08-runtime-hooksis the case in point — it names Anthropic's hooks reference and Codex's hooks guide in prose, with no URL and no date, in a repository with no way to tell whether either was opened.A
nomust now point at where the work is settled (a backticked path orpath:line); "this is an internal change" is refused, because it is the sentence an unexaminednoproduces.Verification
Hooksworkflow reports pass for this branch — linked below once it runsbash hooks/smoke-test—ALL SMOKE PASSED, ~20 new assertionsbash hooks/release-check .—STATUS: passgit diff --check— cleanFAIL the unbacked source was not namedFAIL the unbacked source was not namednopointer requirementFAIL external facts message for no reason names nothinginspectentryFAIL the inspect enumeration changed size## Verification / ReviewThe archive is in this pull request
hooks/task completehas been run on2026-10-31-external-facts, and the move toachieved/plus the Todo rewrite are the third commit here. Archiving after the merge would have left them on no branch — landable only by pushing tomainoutside a pull request, or by a second pull request carrying bookkeeping alone, both of whichCONTRIBUTING.mdforbids.That document already covered the archive by name; the practice had drifted from it because
hooks/task completeprints a teardown line that waits for the merge, and that condition is aboutgit worktree remove, not about when to archive.CONTRIBUTING.mdandSKILL.mdnow say which is which, and a smoke assertion requires both to keep saying it.Two defects the fixture caught
a|brecorded assearch|a|b— one record with a forged field boundary. The assertion compares the whole line, not the count. The first fix then ate the separator by runningtrover the joined string, which the same assertion caught.approverefused for the wrong reason. It has its own worktree now.Review boundaries
nowith a plausible pointer still gets through. Enforcing otherwise would demand external evidence from tasks that genuinely need none, which would make most of this repository's work lie. The field records a decision and the record makes it checkable; it does not make the decision honest.PostToolUseforWebSearch/WebFetchis documented by Claude Code's hooks reference (read 2026-11-01) but not yet observed in a running session —runtime.mdrecords it as documented, not measured, and says what moves it to measured.