input: Add atomic inline tokens to Input and Textarea

[suxiaoshao]

Closes #3110

Description

Add optional atomic inline tokens to Input and Textarea for mentions, commands, file and image references. A token can display “Alice” while the input value and the clipboard contain @alice. Users select, delete and undo the whole token; the application maps its ID back to the resource it names.

Model and API

The input stays a plain-text control: the value is the text, and each token is an annotation over a byte range that follows edits — the same model as Draft.js entity ranges and the VS Code chat input's parts over Monaco decorations, rather than a document tree with atom nodes. Base owns the behavior (atomic caret movement, selection and deletion, history with token deltas, mixed text/element layout and wrapping, activation); Component owns the default appearance.

// Insert a reference at the caret or over the selection; `?` reports why it was refused.
state.replace_with_token(InlineToken::new("person:alice", "@alice").with_label("Alice"), window, cx)?;

// Save and restore a draft with its tokens; `set_value` takes text or content.
let draft = state.content();
state.set_value(draft, window, cx);

// Build content from storage; every token is validated as it is attached.
let draft = InputContent::new("Ask @alice")
    .with_token(4..10, InlineToken::new("person:alice", "@alice").with_label("Alice"))?;

• InlineToken carries a resource ID, its text and a display label. The ID names the resource and may occur more than once in one input.
• InputContent is what content() returns and what set_value accepts; plain text converts into content without tokens. InputContent::with_token returns Result, so a content value is always consistent before it reaches the input.
• Input / Textarea (Base and Component) take a token slot and on_token_click. A click selects the whole token and then runs the listener; dragging or Shift-selecting does not open it. ActivateToken opens an exactly selected token from a key binding or assistive technology.
• Component renders tokens as InputToken by default, with icon and Styled; a selected token paints its own selected state, and the text selection highlight stops at its edges. The Base token element uses the arrow cursor, and tokens are measured whenever they render, so an element that grows reflows on the next frame.
• The JavaScript API mirrors this on InputState / TextareaState and on the registered components: content(), tokens(), set_value(string | InputContent), replace_with_token, replace_range_with_token, token, on_token_click. Ranges are UTF-16 string offsets, and validation errors carry error.code.

Tokens remain opt-in. Editor, NumberInput, formatted masks and password fields do not accept token insertion. Copy/paste uses ordinary text; resource lookup, delimiters around an inserted reference and submission remain application decisions.

Story and docs

The Story is a chat composer shared by Input and Textarea: an InputGroup with a toolbar that inserts a /commit-pr command, an [Image 1] attachment, a $gpui-kit skill and an @alice mention as tokens, a Send action, draft save/restore, read-only and disabled toggles and a readout of the text, references and last action. The JavaScript Story has the same example. English and Chinese documentation cover insertion, appearance, drafts, validation and the JavaScript API for Input, Textarea and InputGroup.

Public API

gpui-base (gpui_base::input)

Data:

• InlineToken — a reference rendered as one editing unit. new(id, text), with_label(label); readers id(), text(), label(). Derives Clone, Debug, PartialEq, Eq, Hash.
• InlineTokenSpan — a token at its current UTF-8 byte range: range() -> Range<usize>, token() -> &InlineToken.
• InputContent — text with its tokens, what content() returns and set_value accepts. new(text), with_token(range, token) -> Result<Self, InlineTokenError> (validates boundary, overlap and text match), text(), tokens(). From for every text type set_value accepted before (&str, String, SharedString, Cow<str>, Arc<str>, Box<str>, char, and their references).
• InlineTokenError — #[non_exhaustive]: InvalidRange, InvalidBoundary, InvalidToken, OverlappingTokens, TextMismatch, UnsupportedMode, ValidationRejected, CompositionActive. Implements Error and Display.
• InlineTokenContext — what a token renderer sees: token(), range(), is_selected(), is_disabled(), is_readonly(), line_height(), available_width().
• InlineTokenClickEvent — what a click listener receives: token(), range(), bounds() -> Bounds<Pixels>, click() -> &ClickEvent.
• ActivateToken — action that opens an exactly selected token (key bindings, assistive technology).

InputState / TextareaState (InputBaseState<InputMode | TextareaMode>):

• replace_with_token(token, window, cx) -> Result<(), InlineTokenError> — replace the selection, or insert at the caret, with a token.
• replace_range_with_token(range, token, window, cx) -> Result<(), InlineTokenError> — replace a byte range, expanding overlaps to whole tokens.
• tokens() -> &[InlineTokenSpan] — the current tokens in document order.
• content() -> InputContent — the text with its tokens.
• Changed: set_value(value: impl Into<InputContent>, window, cx) — was impl Into<SharedString>; plain text still converts, content also restores tokens.

Input / Textarea elements:

• token(render: impl Fn(&InlineTokenContext, &mut Window, &mut App) -> impl IntoElement) — the element each token renders as.
• on_token_click(listener: impl Fn(&InlineTokenClickEvent, &mut Window, &mut App)) — runs after a completed, unconsumed click on a token.

Hidden (#[doc(hidden)], for styled controls only): InlineTokenRenderer, InlineTokenClickListener, InputBaseState::install_token_presentation.

gpui-component (gpui_component::input)

• InputToken — the element a token renders as by default: new(&InlineTokenContext), icon(impl Into<Icon>), Styled. Selected, disabled and readonly states come from the context.
• Input / Textarea: token(..) and on_token_click(..) as in Base; InputGroupInput / InputGroupTextarea inherit them.
• Re-exports of the Base data types above and ActivateToken.

gpui-shell (host integration)

• StateMethodDescriptor — an opt-in operation on retained state exposed to scripts: new::<T>(name, signature, call), with_readonly(bool), name(), signature(), is_readonly().
• StateDescriptor::with_methods(Vec<StateMethodDescriptor>), methods().
• ComponentElementCallback::build_interactive_data_with(args, window, cx) -> Result<Option<AnyElement>> — build a frame-owned inline subtree whose child callbacks retire with that frame.
• ComponentCallback::invoke_data_with(args, window, cx) -> Result<()>.
• InlineTokenCallbacks — script callbacks adapted to an element's token / on_token_click builders: new(&Entity<InputBaseState<M>>, renderer, listener), apply(element, token, on_token_click).
• inline_token_context_data, inline_token_click_data, input_token_state_methods(), textarea_token_state_methods().

JavaScript (gpui-base / gpui-component)

• InputState / TextareaState: content(): InputContent, tokens(): InlineTokenSpan[], replace_with_token(token), replace_range_with_token(range, token), set_selected_range(range), replace(text); changed: set_value(next: string | InputContent). Ranges are UTF-16 offsets; validation errors carry error.code.
• Input / Textarea / InputGroupInput / InputGroupTextarea: token(render), on_token_click(listener).
• Types: InputRange, InlineToken, InlineTokenSpan, InputContent, InlineTokenContext, InlineTokenClickEvent.

Breaking Changes

None. Existing input APIs and InputEvent variants are unchanged; the token APIs are additive. set_value now takes impl Into<InputContent>, and every text type it accepted before converts into content, so existing calls compile and behave as they did.

How to Test

Validated on macOS on top of the latest main:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- --deny warnings
cargo test -p gpui-base --lib input::
cargo test -p gpui-shell --lib component_callback_value_tests
cargo test -p gpui-component-shell --test inline_tokens_host --test input_group_host --test layout_host
cargo test -p gpui-component-story --lib token_story
GPUI_COMPONENT_SHELL_BIN=target/debug/gpui-component-shell node crates/component-shell/tests/types/run.mjs
cargo run

The Base tests cover token insertion and rejection, history with token deltas, boundary and word movement, IME composition, mode gates, wrapping and re-measurement, click selection and re-entrant activation. The Story test drives the composer through insertion, partial-selection deletion, undo, save/restore, sending and the read-only/disabled gates for both controls. Host tests cover UTF-16 emoji boundaries, atomic rejection, re-render retention, token activation and custom child callbacks.

In the Story gallery, open Input or Textarea and scroll to “Atomic inline tokens”: click a token to select it and read the status line, insert the same reference twice, select part of a token and delete it, undo, and save and restore the draft.

Real IME candidate/cancellation behavior on Windows and Linux remains unverified.

AI Assistance

The implementation, tests and documentation were written with Codex assistance; the API review and rework (resource IDs, set_value(content), the presentation seam, click selection, naming and the composer Story) were done with Claude Code and reviewed by hand.

🤖 Generated with Claude Code

[huacnlee]

Thank you.

You can take a little time to read my changes, primary to tidy the APIs to follow the GPUI and GPUI Kit style. Then next time will better.

[Muhammad-Owais-Warsi]

I can't find the example in story , could you please link it

[huacnlee]

I can't find the example in story , could you please link it

At bottom of Input story.

[Muhammad-Owais-Warsi]

the last example in input story is for text color

[suxiaoshao]

the last example in input story is for text color

[Muhammad-Owais-Warsi]

It seems the website is still not updated for me, it's still showing old components only for me