Skip to content

feat(agent-service): add docs search and read tools - #9424

Draft
eugenegujing wants to merge 1 commit into
apache:mainfrom
eugenegujing:feat/agent-docs-tools
Draft

eugenegujing wants to merge 1 commit into
apache:mainfrom
eugenegujing:feat/agent-docs-tools

Conversation

@eugenegujing

@eugenegujing eugenegujing commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The agent could not answer a question about using Texera from the documentation. It had no way to read docs/, so it answered from the model's memory and could not give a link. This PR adds two read-only tools over the repository's own docs/ folder and a place in the agent's context for what they return.

The design in one picture

flowchart TB
  md["docs/*.md"] -- "COPY docs ./docs<br/>at image build" --> image[agent-service image]
  image -- "read once, when the<br/>first agent is created" --> index["In-memory tf-idf index<br/>one entry per section"]
  index --> search["searchDocs(query)<br/>best matching sections"]
  index --> read["readDoc(path, section)<br/>page or section text"]
  search --> section["Documentation Read<br/>section of the agent's context"]
  read --> section
  section --> answer["Answer with a link<br/>to the published page"]
Loading

In the UI. The same question, "What are the output port modes of an operator?", on the same workflow. The answer is on the docs page reference/operators/output-modes.

main This PR
B2-main-output-modes B2-this-pr-output-modes-1-top
The agent answers that output port modes are not a concept in the system. The agent searches the docs, reads the page, and lists the three modes from it, with a link to the page at the end.

Size and reading order

1,002 lines of code in 10 files, 898 of them in two new files, and 2,669 lines of tests and evaluation data in 8 files.

Order Files Lines What they hold
1 tools/docs-index.ts +678 Loading, splitting into sections, ranking, text cleaning, path and section lookup. It imports only node:fs and node:path
2 tools/docs-tools.ts +220 The two tools and their output format
3 util/context-utils.ts +66 The Documentation Read section
4 texera-agent.ts, tools/index.ts +20 Loading the index and registering the tools
5 prompts.ts +9 The prompt rule and the Context Format entry
6 agent-service.dockerfile, config/env.ts, .env.example, bin/local-dev/main.sh +9 Where the docs folder comes from

The tools

  • searchDocs(query, limit?) returns the best matching sections with their heading chain, page path, page link and a snippet. When nothing matches it returns a short outline of the documentation areas (about 1,400 characters today).
  • readDoc(path, section?, offset?) returns the whole page when it fits in 4,000 characters (212 of the 219 pages do), otherwise its section list and introduction, or the named section. A directory path lists the pages under it, and an unknown path returns the closest page names and the outline. Longer output is clipped, and its last line names the offset to continue from.
  • Arguments are matched loosely. A path may differ in case, be a directory, carry .md or a /docs/... prefix, or be the page link. A section may differ in case, punctuation or emoji, or be given as the Page > Heading chain that searchDocs prints. null for an optional argument counts as absent. A limit out of range is clamped and the output says so, and a bad offset comes back as an error the model can read.

How the output reaches the model

The agent rebuilds its context on every step, and its history lists a past tool call by name only. That is enough for the operator tools, whose effect shows in the dataflow, but text returned by a docs tool had nowhere to go. assembleContext now adds a # Documentation Read section between the ongoing task and the dataflow. It lists the docs calls of the current task, oldest first, each under its turn number, tool and input, with the output. A call that failed is listed with its error, and one that never ran, because the tool is not registered or the SDK refused its arguments, with a note saying so. Its entries stay within 20,000 characters, and the intro line and the dropped-outputs line add about 150 on top. Over that, the newest calls are kept whole and one line says how many older ones were left out. serializeTask is untouched because #9244 edits it.

One question, step by step

sequenceDiagram
  actor U as User
  participant A as Agent service
  participant D as Docs index
  participant M as LLM

  U->>A: "How do I upload a dataset?"
  A->>M: Context (step 1)
  M-->>A: Call searchDocs
  A->>D: searchDocs("upload dataset")
  D-->>A: Matching sections
  rect rgba(46, 160, 67, 0.15)
    Note over A: Documentation Read now holds the search output
    A->>M: Context (step 2)
  end
  M-->>A: Call readDoc
  A->>D: readDoc("tutorials/create-dataset-upload-data")
  D-->>A: Page text
  rect rgba(46, 160, 67, 0.15)
    Note over A: Documentation Read now holds both outputs
    A->>M: Context (step 3)
  end
  M-->>A: Answer with the page link
  A-->>U: Answer
Loading

The index and the prompt

Part What it does
Loading Reads the pages under TEXERA_DOCS_DIR once, when the first agent is created, skipping contribution-guidelines/. A missing, empty or unreadable folder is logged once, each with its own line, and no docs tool is registered until the service restarts. The prompt text is the same either way.
Sections Splits a page at heading levels 1 to 3 and keeps fenced code whole. Heading names lose their link markup, and a repeated name gets a number. A heading with only sub-sections reads as their list. A page with nothing before its first heading is introduced by its description.
Ranking tf-idf over the source text without HTML comments and <img> tags, with heading words at three times the weight, camelCase splitting, stopwords and a light suffix stemmer.
Text shown Breadcrumbs, HTML comments and <img> tags are removed outside code, and relative link targets become absolute, so every link in the text can be cited as it stands.
Deployment COPY docs ./docs in the Dockerfile, where TEXERA_DOCS_DIR defaults to ./docs. bin/local-dev/main.sh exports the repo's docs, and .env.example sets ../docs.
Prompt One rule is added after "Read documentation first". For a question about how Texera works it says to search the docs, read the best match, answer with the page link, cite only links the tools returned, say so when the docs do not cover it, and not edit the workflow. The Context Format part gains the new section.

No embeddings, no model download, no new dependency. The index covers about 430 KB of markdown and loads in about 30 ms.

Measured, without a model

  • agent-service/scripts/docs-search-eval.ts runs 25 questions against the real docs/. The expected page is in the top 3 for 21 and in the top 5 for 23. The two misses are documentation gaps.
  • The context section carries about 2,500 to 5,600 characters for a question answered from one or two pages, about 1,700 for a search with no hit, and about 2,000 for a path that names no page.
  • Of the 221 distinct internal links in the text shown, 216 answer 200 on the published site. The other five are broken in the docs themselves (four pages linked from examples/_index.md and a sample CSV in tutorials/create-dataset-upload-data.md).

Known limits

  • Citing only returned links and saying when the docs do not cover a question are prompt rules, not guarantees.
  • Docs output is kept for the current task only. A follow-up question in the next message searches again.
  • Page links use https://texera.apache.org/docs/latest/<page>/. Today each of those is a small stub that forwards to the current release's page and drops the #anchor, so a 200 there does not prove the page exists in that release.
  • The history line of a docs call still reads (succeeded) when its output is an [ERROR], as for every tool today. After feat(agent-service): show past tool results in the agent's history #9244 a docs error will show both there and in the new section.
  • COPY docs also puts a 1.8 MB screenshot and the excluded contributor pages into the image.
  • Ranking is by keywords, so a question worded differently from the docs can miss.
  • The tools repeat what the docs say. tutorials/create-dataset-upload-data.md still tells the user to open an environment tab and click Add Dataset, which the editor no longer has, and the agent repeated that step.

Any related issues, documentation, discussions?

Closes #9421. Proposed in #9377 (change 2), following up on #9246. Related: #9244 (past tool results in the history, first line only, which complements the new section), #8773 (docs gaps and broken links found here), #9031, #4932.

Note for reviewers: this branch and the run-errors branch from #9377 both add to prompts.ts, texera-agent.ts and context-utils.ts (and their specs) in neighbouring places, so the two do not apply on top of each other as they are. Whichever lands second will be rebased.

How was this PR tested?

cd agent-service && bun test: 526 pass, 0 fail across 21 files (326 before this PR), plus bun run typecheck and bunx prettier --check.

Spec Covers
docs-index.spec.ts Parsing, heading chains, section lookup by name and chain, tokenizer and stemmer, heading weight, what is indexed, snippets, link resolution, text cleaning, path lookup, the outline, emoji-safe cuts, empty folders. Fixtures only.
docs-tools.spec.ts Result format, the outline fallback on a 300-page tree, limit clamping, input schemas with null arguments, whole-page reads at the size boundary, listings cut between entries, continuation by offset, bad arguments, a failing index, index loading and its log lines.
context-utils.spec.ts Position, order and labels of the section, failed and rejected calls, finished and abandoned tasks, the budget at its boundary and after 60 calls.
texera-agent.spec.ts With a scripted model: a docs call's output is in the next step's context, gone from the next request, left out over the budget, and a rejected call is reported. The default index path with a present, missing and unreadable folder.
prompts.spec.ts The kept rule, the added rule, and the Context Format entry.

Mutation checks on this logic: 163 mutants, all killed. Also a local docker build of the agent-service image to confirm the index loads inside it as the texera user, and a new check in bin/local-dev/tests/test_local_dev_sh.sh for the export, run on its own here.

Live test on a local stack running this branch, with claude-haiku-4.5

Request What the agent did Result
"How do I upload a dataset in Texera?" searchDocs, readDoc, answer The steps of the tutorial page, with its link. Before the context section existed the same question repeated searchDocs with the same query on every step and never answered
The same question in Chinese searchDocs with mixed keywords, readDoc, answer Correct answer in Chinese with the same link
Hash Join, output modes, first login one search and one read each Answers match the pages, every cited link opens
A search with no hit, and a page that does not exist outline, or the closest pages, then a correct follow-up About 1,500 and 1,800 characters of tool output instead of 28,000
A section longer than the clip readDoc with the section Read in one call after the text cleaning. An earlier build clipped it and the model continued with offset as the trailer told it
A call with a string offset rejected by the SDK The model saw the rejection note and repeated the call with a number
"What is the capital of France?" no tool call Declined as off topic
"Add a Sort operator ... check the docs first" in the real UI searchDocs, readDoc, addOperator Sort added with the documented properties and executed, visible on the canvas

No workflow was changed by a question. A question took 3 steps and about 33k input tokens, of which about 10.5k per step is the existing system prompt.

Was this PR authored or co-authored using generative AI tooling?

Yes. Generated-by: Claude Code (claude-fable-5-2). Co-authored with Claude; the author reviewed the change before submission.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Committers with relevant context: @parshimers
    You can request their reviews formally with /request-review @parshimers.

  • Contributors with relevant context: @bobbai00, @aglinxinyuan, @Yicong-Huang
    You can notify them by mentioning @bobbai00, @aglinxinyuan, @Yicong-Huang in a comment.

@codecov-commenter

codecov-commenter commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.88%. Comparing base (b2ff978) to head (985bc72).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9424      +/-   ##
============================================
+ Coverage     91.78%   91.88%   +0.09%     
  Complexity     5023     5023              
============================================
  Files          1283     1285       +2     
  Lines         56875    57563     +688     
  Branches       7442     7442              
============================================
+ Hits          52205    52893     +688     
  Misses         2800     2800              
  Partials       1870     1870              
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from b2ff978
agent-service 99.64% <100.00%> (+0.08%) ⬆️
amber 85.75% <ø> (ø) Carriedforward from b2ff978
computing-unit-managing-service 64.05% <ø> (ø) Carriedforward from b2ff978
config-service 87.73% <ø> (ø) Carriedforward from b2ff978
file-service 82.31% <ø> (ø) Carriedforward from b2ff978
frontend 96.42% <ø> (ø) Carriedforward from b2ff978
notebook-migration-service 83.77% <ø> (ø) Carriedforward from b2ff978
pyamber 98.50% <ø> (ø) Carriedforward from b2ff978
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from b2ff978

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add a keyword index over docs/ built when the first agent is created, the searchDocs and readDoc tools over it, a Documentation Read section in the agent's context that carries their output for the current task, a prompt rule that uses them for questions about Texera, a search evaluation script, a TEXERA_DOCS_DIR setting exported by bin/local-dev, and a COPY docs line in the agent-service Dockerfile.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent cannot answer questions about using Texera from the documentation

2 participants