Skip to content

feat: make a task answer whether its facts come from outside the repo - #108

Merged
hkwuks merged 3 commits into
mainfrom
feat/external-facts
Nov 1, 2026
Merged

hkwuks merged 3 commits into
mainfrom
feat/external-facts

Conversation

@hkwuks

@hkwuks hkwuks commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

An Agent working inside TaskFlow tends not to look outside the repository. This change asks it where the answer comes from at the three points where that answer can change, records the lookups it makes, and refuses a claim to have looked that nothing backs.

v1 of this PR (still in the branch history, as 96044df) shipped only the field — a ## External Facts section approve validates. That recorded a judgement without touching the tendency to write no reflexively, and the v1 PRD said so in its own risk section. The user's correction on 2026-11-01 is what this PR now answers. v1's documents are archived under TaskFlowDocs/2026-10-31-external-facts/old/v1/.

TaskFlow traceability

  • Task: TaskFlowDocs/2026-10-31-external-facts/ (v2)
  • Scope: hooks/external-evidence (new), hooks/hooks.json, hooks/task, hooks/smoke-test, hooks/README.md, skills/taskflow/SKILL.md, skills/taskflow/references/{artifacts,runtime}.md, CHANGELOG.md, CONTRIBUTING.md (archive ordering), and the archive of this task itself
  • Base branch: main
  • Target repository: hkwuks/TaskFlow

Why a field was not enough

Every source this workflow points an Agent at is inside the repository:

where what it says
SKILL.md:54 inspect lists seven sources — all internal
summarize-state opens the session with internal conclusions
prompt-reminder every turn, about task state only
capability-evidence counts Skill/Agent/mcp__* and nothing else, so a web lookup is invisible

Nothing in that list prompts the question, so a no written by default is indistinguishable from one reached by thinking.

What the three pause points do

Before Phase 1 converges, at the Phase 3 spec decision — defining an interface or a compatibility boundary is exactly when "only our own code" stops being true — and once more before approval, against Steps that did not exist when the first answer was given.

The inspect enumeration is deliberately not extended. Its entries are "open the repo and look"; a judgement question among them would dress a decision up as an inspection. A smoke assertion counts the entries so a ninth cannot be added quietly.

The isolation, which is the design

taskflow/evidence   ← Skill / Agent|Task / mcp__*   → releases a stage
taskflow/external   ← WebSearch / WebFetch          → releases nothing

capability-evidence is untouched, so a built-in tool still cannot satisfy the gate (R12, 2026-09-27). The two predicates live in two files with two consumers: one case with a shared fall-through would put the meanings back together. The suite proves the separation rather than asserting it — a task holding two real lookups and no invocation must still be refused, and refused for the missing stage.

approve reads external for the opposite purpose: a yes needs every - Source: backed by a recorded lookup. achieved/2026-09-08-runtime-hooks is the case in point — it names Anthropic's hooks reference and Codex's hooks guide in prose, with no URL and no date, in a repository with no way to tell whether either was opened.

A no must now point at where the work is settled (a backticked path or path:line); "this is an internal change" is refused, because it is the sentence an unexamined no produces.

Verification

  • The Hooks workflow reports pass for this branch — linked below once it runs
  • bash hooks/smoke-test — ALL SMOKE PASSED, ~20 new assertions
  • bash hooks/release-check . — STATUS: pass
  • git diff --check — clean
  • Four mutations, each reddening its own named assertion and restored byte-identically:
mutation assertion that fired
drop the lookup-backing check FAIL the unbacked source was not named
never fail on an unbacked source FAIL the unbacked source was not named
drop the no pointer requirement FAIL external facts message for no reason names nothing
add a ninth inspect entry FAIL the inspect enumeration changed size
  • Results recorded in the TaskFlow Plan — ## Verification / Review

The archive is in this pull request

hooks/task complete has been run on 2026-10-31-external-facts, and the move to achieved/ plus the Todo rewrite are the third commit here. Archiving after the merge would have left them on no branch — landable only by pushing to main outside a pull request, or by a second pull request carrying bookkeeping alone, both of which CONTRIBUTING.md forbids.

That document already covered the archive by name; the practice had drifted from it because hooks/task complete prints a teardown line that waits for the merge, and that condition is about git worktree remove, not about when to archive. CONTRIBUTING.md and SKILL.md now say which is which, and a smoke assertion requires both to keep saying it.

Two defects the fixture caught

  • a|b recorded as search|a|b — one record with a forged field boundary. The assertion compares the whole line, not the count. The first fix then ate the separator by running tr over the joined string, which the same assertion caught.
  • the isolation fixture shared a capability store the earlier cases had pushed out of step, so approve refused for the wrong reason. It has its own worktree now.

Review boundaries

  • No secrets or opaque remote payloads included
  • No unrelated task or user files changed
  • Remote/base assumptions stated
  • Known limitation, stated in the Plan rather than papered over: a model that answers no with a plausible pointer still gets through. Enforcing otherwise would demand external evidence from tasks that genuinely need none, which would make most of this repository's work lie. The field records a decision and the record makes it checkable; it does not make the decision honest.
  • PostToolUse for WebSearch/WebFetch is documented by Claude Code's hooks reference (read 2026-11-01) but not yet observed in a running session — runtime.md records it as documented, not measured, and says what moves it to measured.

`hooks/task approve` now reads a `## External Facts` section in `prd.md`
and refuses four shapes: no answer, an answer that is neither `no` nor
`yes`, `no` without a reason, and `yes` without a sourced conclusion.

The rule is not "search more". Measured 2026-10-31 over 94 archived tasks:
`## Reference Pointers` holds 382 bullets and not one is a URL, 3 of 94
tasks have a `reference/` directory, and the archive's only four
third-party URLs sit in a single task. None of that can show whether
TaskFlow suppresses search, because most work here is internal and should
cite nothing. So the field does not count searches -- it records the
judgement, which is the one thing that can be checked. A `yes` is a
commitment to go and look: it is refused until a source and the
conclusion that source changed are both named. A `no` is refused until it
says what puts the conclusion inside the repository, because an
unexplained `no` is the default the field exists to displace.

`promote` writes the section unanswered on purpose, so a new task cannot
be approved until someone answers it. That is the half that makes the
field mandatory rather than merely available.

`capability-evidence` still counts only `Skill`/`Agent`/`mcp__*`, so
`WebSearch`/`WebFetch` neither satisfy nor fail the capability gate. That
separation is 2026-09-27's R12 and it is unchanged -- letting a built-in
tool satisfy the gate would make it passable by searching once.

Three things the measurement settled that the plan had to absorb:

  - the check belongs in `prd.md` alone. Whether evidence is gathered at
    all is a requirements question; the Plan's job is how, not whether.
  - `SKILL.md:54`'s `inspect` enumeration is not extended. Its eight
    entries are "open the repository and look" -- adding a judgement
    question there would dress a decision up as an inspection.
  - already-archived tasks are not backfilled. `achieved/` is read-only
    in place, as it was for the `done`-only rule.

Six mutations, each reddening its own named assertion and each restored
byte-identically: dropping the `no`-reason check, allowing zero sources,
not validating the URL, not requiring "what it settled", accepting an
answer outside `no`/`yes`, and passing an empty answer. One line the
first version carried turned out to be dead -- it refused an empty
section before a `case` arm that refuses it with the same message -- and
was removed rather than left as decoration.
… record

v1 shipped a `## External Facts` field that recorded a judgement. It was
an answer to the wrong question. The observation behind this task was
never "the field is missing" -- it was that an Agent working inside
TaskFlow tends not to look outside the repository at all. A field a model
can satisfy by writing `no` does not touch that, and the v1 PRD said so
in its own risk section before going ahead anyway.

What actually pulls attention inward, each at a line:

  - `SKILL.md`'s `inspect` enumeration lists seven sources, all inside the
    repository or the Agent's own state.
  - `summarize-state` opens a session with internal conclusions.
  - `prompt-reminder` speaks every turn, about task state only.
  - `capability-evidence` counts `Skill`/`Agent`/`mcp__*` and nothing
    else, so a web lookup is invisible to the whole workflow.

So the question now gets asked at the three points where its answer can
change: before Phase 1 converges, at the Phase 3 spec decision -- where
defining an interface or a compatibility boundary is exactly when "only
our own code" stops being true -- and once more before approval, against
Steps that did not exist when the first answer was given. The `inspect`
enumeration is deliberately not extended: its entries are "open the repo
and look", and a judgement question among them would dress a decision up
as an inspection. A smoke assertion counts the entries so a ninth cannot
be added quietly.

Lookups are now visible, and the isolation is the point of the design.
`hooks/external-evidence` records `WebSearch`/`WebFetch` into
`taskflow/external`; `capability-evidence` is untouched and still refuses
to count built-in tools, so a lookup can never release a stage (R12,
2026-09-27). The two predicates live in two files with two consumers
because one `case` with a shared fall-through would put the two meanings
back together. The smoke suite proves the separation rather than
asserting it in prose: a task holding two real lookups and no invocation
must still be refused, and refused for the missing stage.

`approve` reads the external store for the opposite purpose -- a `yes`
must have every `- Source:` backed by a recorded lookup, or the section
is a form filled in from memory. `achieved/2026-09-08-runtime-hooks` is
the case in point: it names Anthropic's hooks reference and Codex's
hooks guide in prose, with no URL and no date, in a repository with no
way to tell whether either was opened. A `no` must now point at where
the work is settled -- a backticked path or `path:line` -- because "this
is an internal change" is the sentence an unexamined `no` produces.

Four mutations, each reddening its own named assertion: dropping the
lookup-backing check, never failing on an unbacked source, dropping the
`no` pointer requirement, and adding a ninth `inspect` entry. Each was
restored byte-identically.

Two defects the smoke suite caught while this was being written, both
worth the fixture that found them:

  - `a|b` recorded as `search|a|b` -- one record with a forged field
    boundary. The assertion compares the whole line, not the count. The
    fix then ate the separator itself by running `tr` over the joined
    string, which the same assertion caught.
  - the isolation fixture shared a capability store the earlier cases had
    already pushed out of step, so `approve` refused for the wrong
    reason. It gets its own worktree now.

Still open, and stated rather than papered over: a model that answers
`no` and writes a plausible pointer gets through. Enforcing otherwise
would mean demanding external evidence from tasks that genuinely need
none, which would make most of this repository's work lie. The field
records a decision and the record makes it checkable; it does not make
the decision honest.
The archive was being run after the merge, which forces it onto no branch:
the moved directory and the rewritten Todo entry have nowhere to land
except a push to the base outside any pull request, or a second pull
request carrying only bookkeeping. `CONTRIBUTING.md` forbids both, in the
same paragraph that requires a task's documents to ride in its own pull
request -- so the practice was contradicting the rule it was meant to
follow, and the rule already covered the archive by name.

The mistake came from a message, not a document. `hooks/task complete`
prints a teardown line that waits for the merge, and that condition is
about `git worktree remove` -- deleting a directory that may still hold
the only copy of something. Nothing about the archive transaction needs
the merge: `complete` reads the working tree, moves the directory and
rewrites the Todo entry, and never consults the base branch. Reading one
as the other is what produced a second pull request for this repository
twice.

Both statements now say which is which. `CONTRIBUTING.md` states the
order -- finish, `complete`, commit code and documents and archive
together, push, open the pull request -- and names the two ways of
getting it backwards and why each is forbidden. `SKILL.md` states it
where the command is defined, and says the teardown line is about the
worktree rather than about when to archive.

This commit is the rule applied to itself: the archive for
`2026-10-31-external-facts` is in it, alongside the code.
@hkwuks
hkwuks merged commit 9c27ceb into main Nov 1, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant