Add interpreter-wide search to the Help pane

[kv9898]

Dependency

Requires posit-dev/ark#1401 to merge, followed by Positron adopting the resulting upstream Ark version through its normal update process. This PR intentionally does not change the Ark submodule pointer. Until that upstream update lands, the checked-in Ark does not support the new Help protocol. Local R validation used the companion Ark checkout at f645159d, kept outside the committed Positron gitlink.

Summary

Adds interpreter-wide search and autocomplete to the first row of the Help pane, retaining in-page search on the second row. Resolves #422.

• Query-based autocomplete returns at most 50 backend-ranked suggestions; the frontend debounces requests and discards obsolete responses.
• Python suggestions and broad searches share static filesystem/ZIP discovery plus already-loaded module metadata, without importing packages for discovery.
• Backend indexes check relevant environment and documentation metadata for changes.
• Busy sessions wait before dispatch; submitted searches show a waiting status. Help RPCs have explicit 30-second timeouts.
• Search IDs correlate replies and navigation so expired or superseded UI searches cannot navigate later. Console Help events remain untagged.
• Native R help.search() semantics and rendering, selected-topic resolution, and narrow-pane layout are retained.

Scope and limitations

Index construction/rebuild remains synchronous; this does not add background indexing or service R Help requests while user code is running. Python may omit runtime-only summaries and dynamically exposed paths from unloaded packages.

#4484 is acknowledged and unchanged; its refresh/history behavior is left to the Positron team. F1 lookup changes remain outside this PR.

Protocol: get_help_topics(query, limit), search_help(query, search_id), and optional search_id on show_help. Coordinate the frontend and both backend versions when landing.

Demo

search.mp4

Verification

• Frontend Help suites: 21 tests passed.
• Python Help/index/pydoc: 122 passed, 1 skipped; Ruff checks passed.
• npm run typecheck-client and client transpilation passed.
• R/Python extension compilation passed.
• Companion Ark: selected Help tests and debug build passed (details in its PR).
• Manually tried the revised development build successfully.
• git diff --check passed. Targeted frontend lint has no errors; warnings remain.

These are local results; upstream CI is not yet green.

[juliasilge]

Thank you for this contribution! We will be able to pick this up for review the week of September 21 or 28.

[kv9898]

@juliasilge that would be this or next week, right?

[juliasilge]

Thank you so much for this contribution, @kv9898! 🙌

Before we look at small details, I want to talk about three problems with the overall design. Each problem affects the shape of the comm methods, so we'll need to get the design right there, before implementing the Ark and Python sides of the comms.

1. Python autocomplete imports every installed package into the user's session

get_help_topics() calls pydoc.ModuleScanner().run(callback, "", ...). Internally, ModuleScanner uses pkgutil.walk_packages(), which must import each package to find its submodules. The docstring of walk_packages says this:

Note that this function must import all packages (NOT all modules!) on the given path, in order to access the __path__ attribute to find submodules.

The scan runs in the kernel process, so all of these imports go into the user's sys.modules. The __init__ of each package runs, with all of its side effects (for example, plugin registration, logging handlers, or GPU setup). Some packages also print to stdout during import. Also, because the key is "" and not None, ModuleScanner reads the source of every module and imports all built-in modules.

I measured the same scan in some local environments:

Environment	Topics	Time	New entries in sys.modules
Base Python	2,181	1.6s	640
Data science environment	27,183	33.3s	7,977
Older data science environment	17,051	37.7s	10,253

The pydoc search?key= page that search_help() opens does a similar scan with the same imports, on the pydoc server thread.

Some options that do not import user packages:

• Use pkgutil.iter_modules() for top-level names only. This function does not import anything, and it's fast.
• Find submodules from the file system. Get each package location from importlib.util.find_spec(), then call pkgutil.iter_modules([location], prefix) on that path.
• Do the full scan in a subprocess of the same interpreter, in the background, and cache the result.

2. Slow work runs on the interpreter's own thread, with the default 5-second timeout

Both backends do expensive work synchronously on the thread that runs user code. In Ark, the comm handlers run on the R thread, and help.search() builds the search database on its first call. In Python, the scan above runs when the kernel handles the comm message. As a result, help requests wait behind any user code that is running, and user code waits behind the help requests.

PositronHelpComm is created without RPC options, so search_help and get_help_topics use the default timeout of 5 seconds from PositronBaseComm. Thus:

• If the first search in a session takes more than 5 seconds, the user sees "An error occurred while searching help". Then the results page opens anyway when the backend finishes.
• If the topic list takes more than 5 seconds, the error is not shown and no suggestions appear.
• If R or Python is busy with a long computation, every help request times out.

The design needs to keep this work off the path of user code, or at least make it cheap and give the methods an explicit timeout that matches the real cost.

3. The frontend receives the full topic list and filters it on every keystroke

get_help_topics returns every topic in one response. For R, Ark returns one pkg::alias entry for each alias. With a few hundred packages installed, this is approximately 100k entries. The component then lowercases, filters, and sorts the full list on each keystroke, and only after that takes the first 50 results. This will cause input lag in large environments. The component also caches the full list in a useRef Map inside the React component, with no invalidation when a user installs a package.

I think a query-based method is a better design. For example, the frontend sends the current text, and the backend returns the best N matches. This is the same model that console completions use. With this design, the full list does not go over the comm, the frontend does not need a large cache, and the backend can use its own index. Newly installed packages are also found, because each query reads the current state.

I am happy to answer questions about any of these points. Thank you again for your work on this!

[eitsupi]

I found this thread while looking at the help experience in arf console.

The concerns here about doing help indexing/search on the R thread are very similar to problems we've been working on in arf. arf uses r-documentation-rs to read installed R documentation metadata directly from Rust, build a lightweight index, and only load the selected help page afterward.

For reference, in one benchmark over 130 installed packages / 6,693 entries, arf built its help index in about 25 ms (eitsupi/arf#375).

r-documentation-rs already reads Meta/hsearch.rds at a low level, and I'm planning to add a typed standalone API for it as well, similar to the existing HelpTopicIndex for Meta/Rd.rds.

Since Ark and arf need much of the same low-level handling of installed R documentation, I wonder if sharing this infrastructure through r-documentation-rs could be useful for both projects, rather than maintaining separate implementations.

Just wanted to mention the existing arf implementation in case it is useful while reconsidering the help-search design here.

[kv9898]

Thanks @eitsupi, this looks relevant to the R-side indexing work.

@juliasilge, would you be comfortable with me trying r-documentation-rs in the companion Ark PR? I’d start with a small prototype that reads installed documentation metadata and builds the search index in a background Rust worker, retaining Ark’s existing path for opening a selected help topic.

Before integrating it, I’d check search coverage against the current help.search() behavior, library-path precedence, and refresh behavior after package changes. I understand the typed standalone hsearch API is still planned and the crate’s APIs are currently unstable.

This would move expensive indexing off the R thread; it wouldn’t by itself make Help comm requests work while R is busy. For this revision, I propose handling busy sessions explicitly in the UI rather than broadening the work into independent request dispatch.

Does that sound like a reasonable direction to explore?

[kv9898]

@juliasilge @eitsupi, a follow-up on scope: I plan to keep the current Ark PR (#1401) free of a new r-documentation-rs dependency and explore that integration in a separate follow-up PR. Both implementations would use the same Help comm protocol, so the indexing implementation could change without another frontend/protocol rewrite. I’ll address query-based suggestions and the current implementation’s cost/busy-session behavior in this revision; the Rust-library integration won’t be a prerequisite for landing it.

On Python, I’m proceeding with import-free static discovery supplemented by a snapshot of already-loaded module metadata. This preserves discoverable module names and available summaries without importing packages to populate suggestions or run broad searches. I’ll document the remaining coverage limitation: runtime-only summaries and dynamically exposed paths of unloaded packages may be absent. Explicitly opening a selected topic will retain the existing help-resolution behavior.

I’ll bring the concrete shared protocol proposal here before wiring the revised backends and frontend together.

[kv9898]

Scope note: this work also encounters the existing same-URL Help refresh behavior tracked in #4484 (successive R searches such as ??cat followed by ??dog). I acknowledge that limitation, but am leaving its resolution to the Positron team rather than choosing a refresh/history policy in this PR. This PR retains the existing behavior and does not attempt to fix #4484.

[kv9898]

@juliasilge, I have revised this PR and the companion Ark PR to address your three points:

1. Python discovery importing installed packages

Both autocomplete and the pydoc search-results route now use a shared index built from static filesystem/ZIP discovery plus a snapshot of already-loaded module metadata. Discovery no longer uses ModuleScanner or imports packages to enumerate their submodules or obtain summaries. Explicitly opening a selected topic keeps the existing help-resolution behavior.

The tradeoff is that runtime-only summaries and dynamically exposed paths of unloaded packages may be absent. Tests cover discovery without executing package initializers, including the HTTP search route.

2. Synchronous work, busy interpreters, and timeouts

The frontend waits for an idle/ready interpreter before dispatching suggestions or a submitted search, and shows a waiting status for submitted searches. Both Help RPCs now have an explicit 30-second timeout. Submitted searches also have a 30-second waiting deadline, reset when dispatched.

search_help(query, search_id) still returns a boolean; the corresponding show_help event echoes the ID. The frontend handles either reply/event order and rejects late events after timeout, a newer search, session changes, or client teardown. Untagged console Help events remain supported. This does not cancel backend computation or HTTP page loading.

Indexing is cached, but first construction/rebuild still runs synchronously. This revision does not introduce background indexing or independent dispatch while R is busy. Ark retains native help.search() and rendering; the proposed r-documentation-rs integration remains a separate follow-up using the same protocol.

3. Full topic catalogue and cache invalidation

get_help_topics(query, limit) now returns at most 50 suggestions, filtered and ranked by the backend. The frontend debounces queries and discards obsolete responses rather than receiving and filtering the full catalogue.

The backend indexes check freshness: R tracks library paths and installed documentation metadata; Python tracks search paths, loaded-module metadata, and filesystem/ZIP metadata. Relevant changes rebuild the index, including additions/removals and documentation updates.

Validation: 21 frontend tests passed; Python Help/index/pydoc tests passed (122 passed, 1 skipped); the 23 selected Ark Help tests passed across the initial run and corrected rerun. Frontend typechecking, client transpilation, R/Python extension compilation, and the Ark debug build passed. I have also tried the revised development build manually and it works well locally.

As noted separately, #4484 remains unchanged and is left to the Positron team. The frontend and both backends need to land together because the Help request signatures changed.

[kv9898]

@juliasilge @eitsupi, I have an optional r-documentation-rs prototype available for review: kv9898/ark#6. Thanks again for suggesting the shared reader infrastructure.

It is a draft PR in my fork, targeting feature-help (the branch behind posit-dev/ark#1401), so the diff shows only the reader adoption. It uses rd-rds 0.4.0 to read installed Meta/hsearch.rds aliases, retains native help.search() matching/rendering and the existing Help protocol, and falls back to R for unsupported metadata. It can be considered separately without making it a prerequisite for #15923 or Ark #1401.

I also benchmarked the current R-only implementation (e9e36753) against the prototype (c8d64da6):

Backend operation	R-only	Rust-reader prototype
First-use suggestions	495 ms	350 ms
Cached suggestions, per call	3.13 ms	3.07 ms
Fresh search without prior suggestions	646 ms	833 ms
First search after suggestions	167 ms	553 ms
Repeated search	167 ms	167 ms

Method: Linux laptop (Core Ultra 9 285H), R 4.6.1, 340 installed package entries; optimized build, median of 15 samples per implementation/scenario. Suggestions used plot with limit 50; searches used linear model. Cached-suggestion samples averaged 30 calls. To reduce the timing drift observed in separate runs, the main comparison alternated the original and prototype R backend definitions in the same optimized, reader-capable Ark process. The original definition uses the R-only path; the prototype calls the compiled Rust reader. Each sample started with a fresh application cache, with the relevant caches primed outside timing for warm scenarios. The filesystem cache was warm. This measures backend work, excluding startup, frontend debounce, RPC transport, HTML rendering, and waiting behind user code.

The result is a tradeoff: about 29% faster first-use suggestions, but slower initial searches. The baseline prepares R’s native search database while constructing suggestions; the prototype reads aliases separately, so the first submitted search pays the deferred native-database build. A fresh submitted search pays for both paths. Cached suggestions and repeated searches were essentially unchanged. These are measurements of this prototype on one machine, not a general performance claim about the library.

The benchmark verified identical bounded suggestions and that the Rust reader succeeded without taking the fallback. The prototype also passed 25 targeted Help tests and Clippy. Indexing still runs synchronously; this draft does not yet implement the background worker discussed earlier. I would welcome feedback on the reader adapter and index lifecycle before taking that further. The existing upstream PRs remain independent of this experiment.

[juliasilge]

Thank you for this revision, @kv9898! The design is in a much better place now. Python no longer imports packages to find topics, suggestions are query-based and bounded, and the help RPCs have explicit timeouts. I ran the new Python index in my local environments, and the only new entries in sys.modules were codec modules. 🎉

Here are the next steps that I see.

1. Do not add r-documentation-rs to Ark

Thank you for the prototype in kv9898/ark#6, and thank you @eitsupi for the suggestion. We do not want to take on this dependency in Ark, now or as a follow-up in the near term. That helpful benchmark in the prototype (thank you!) shows a small gain for the first suggestions, but the first search becomes slower. The indexing also stays on the R thread. For this small change in performance, I'm not up for us taking on a new dependency. Let's keep Ark on native help.search(), and remove the follow-up plan from the PR description.

2. A new import rebuilds the full Python index

The cache key for HelpIndex includes the snapshot of loaded modules. Thus, each new import in the session discards the full filesystem index. I measured this in an example data science environment with approximately 28k topics:

Scenario	Time on the kernel thread
First keystroke (cold build)	2-4s, and the index builds two times (see below)
Later keystrokes	approximately 60 ms
First keystroke after import pandas	approximately 3.1s (full rebuild)

In a realistic data science session, users import new packages all the time. Thus, the first help search after most imports blocks user code for some seconds. I suggest we approach with two layers.

• Cache the filesystem index with only sys.path and the file fingerprints as its key.
• Then, for each request, apply the snapshot of loaded modules on top of that index. (This step is fast because it does not walk the filesystem.)

3. The first build always runs two times

When tokenize.detect_encoding and decode read source files, Python imports encodings.* modules (for example, encodings.latin_1 and encodings.big5). These imports change the snapshot of loaded modules. Thus, the first update_context() after the first build discards the new index. The fix for item 2 also removes this problem.

4. An environment that is too large gives no index

When discovery goes past max_locations, Discovery.entries raises a RuntimeError. No code catches this error, so the full build fails and the user gets no suggestions. I think a partial index with a diagnostic is better than no index.

5. R search with regex characters

.ps.help.searchHelp gives the raw query to utils::help.search(), which reads it as a regular expression. Do queries such as [.data.frame or c( cause an error?

I'll do a detailed review of the frontend code after these changes lands, and then when the design is ready, we will move this branch into the main repository to run the full CI and merge it. (Your commits and credit will stay the same.)

Thank you again for all of your work on this! 🙌

[kv9898]

Thanks, @juliasilge! I’ve revised both PRs according to your suggestions:

1. Kept Ark on native help.search(), closed the r-documentation-rs prototype, and removed the follow-up plan from both PR descriptions.
2. Split the Python cache into filesystem discovery and loaded-module metadata, so ordinary imports no longer rebuild the full filesystem index.
3. The same separation prevents codec imports during the first build from triggering another full build.
4. Discovery now preserves a partial index and logs a diagnostic when it reaches the location limit.
5. In the companion Ark PR Add interpreter-wide R help search support ark#1401, invalid regex queries such as [.data.frame and c( fall back to literal matching. Valid regexes and native fuzzy matching retain their behavior.

I added regression coverage and tested the revised behavior in Positron Dev, including reproducing the R errors before the fix and confirming the results afterward. Initial indexing remains synchronous, as documented.

These changes are ready for another review, including the frontend review you mentioned. Thank you!