Color F# and C# cells by what their names resolve to

README.md

@@ -66,7 +66,7 @@ The architecture is built on one principle: every feature is an extension, and e
 
 ![A C# cell with the completion list open, listing members of a user-defined list](docs/images/intellisense.png)
 
-Every language kernel answers completions and hover from a real language service rather than by text matching: Roslyn for C#, FSharp.Compiler.Service for F#, a live runspace for PowerShell, and your own interpreter for Python. Kernels also compute diagnostics ahead of execution, which hosts and tools reach through `ILanguageKernel.GetDiagnosticsAsync`; errors from a run are reported as cell output.
+Every language kernel answers completions and hover from a real language service rather than by text matching: Roslyn for C#, FSharp.Compiler.Service for F#, a live runspace for PowerShell, and your own interpreter for Python. Kernels also compute diagnostics ahead of execution, which hosts and tools reach through `ILanguageKernel.GetDiagnosticsAsync`; errors from a run are reported as cell output. The C# and F# kernels also classify the symbols in a cell, so the editor colors a type, a function and a value differently.
 
 NuGet packages are referenced inline with `#r "nuget: PackageName, Version"`, and custom package sources are supported with `#i "nuget: <url>"`. A plain version is an exact pin; floating versions such as `1.*` or `*-*` and ranges such as `[1.0.0,2.0.0)` follow NuGet's own rules, and leaving the version out picks the latest stable release. Python uses `#!pip` for package management, and JavaScript uses `#!npm` for npm packages. State persists across cells within each kernel, and variables are shared across kernels through a central variable store.
 

docs/architecture/front-ends.md

@@ -46,6 +46,14 @@ The refresh policy is set by the expensive kernels, where a request is a full co
 
 While any cell is running no request is issued (`NotebookCellContext.IsAnyExecuting`, passed to the editor as `DiagnosticsSuspended`). `Verso.Host` answers requests one at a time, so a request during a run would wait for the run, and some kernels answer empty while executing. A refresh asked for in that time runs once the run is over, in the same top-to-bottom order. Run All is one `execution/runAll` request, so in VS Code the cells are refreshed once it has finished, not one by one as it goes. One request is in flight per editor; a reply for text that has since changed is dropped; a failed request (null) keeps the markers that are showing, while an empty list clears them. Cells over 2,000 lines are not analyzed.
 
+#### Editor semantic tokens
+
+Semantic tokens are pulled, as hover is: `monaco-interop.js` registers a document semantic tokens provider per language, and Monaco asks for a model's tokens when it is attached, after an edit (debounced by Monaco, from 300 ms up as answers take longer), when the theme changes, and when the provider raises `onDidChange`. The provider calls `MonacoEditor.GetSemanticTokens`, which goes through `Cell.razor` to `INotebookService.GetSemanticTokensAsync`; `ServerNotebookService` calls the kernel in process if it implements `ISemanticTokenProvider`, and `RemoteNotebookService` sends `kernel/getSemanticTokens` to `Verso.Host`. A kernel that does not classify answers an empty list and is not started for it.
+
+The provider keeps each model's last answer and the version of the text it was for. Monaco asks for every model when `onDidChange` is raised, and a model whose text has not changed since its answer gets that answer back without a request. The provider raises it when a diagnostics request goes out for text or kernel state that no semantic request has been sent for: after a run, for the cells the run refreshes, and on focus after another cell ran. Monaco passes the id of the answer it shows; a model already showing its last answer keeps it rather than having it painted again. An edit sends both requests, and the diagnostics request does not cause a second semantic one. Requests from all editors are queued and sent one at a time, so in VS Code a run waits behind at most one of them.
+
+While any cell is running, a request is refused with Monaco's "busy" error, which keeps the tokens shown, and is asked again when the run is over. A failed request (null) is refused the same way; an empty list clears the tokens. Cells over 2,000 lines are not analyzed. Semantic highlighting is turned on per editor (`'semanticHighlighting.enabled': true`), since standalone themes leave it off. Monaco styles a token by the theme rule whose name matches its type with the modifiers appended (`variable.mutable` falls back to `variable`); `MonacoThemeBuilder` adds a rule for each standard token type under the theme's syntax color for it.
+
 Monaco's bundled TypeScript worker would otherwise mark JavaScript and TypeScript cells as if each were a standalone file, so its validation is turned off when Monaco loads; its completion and hover stay on.
 
 Isolated (iframe) layouts need more than the base notebook operations, so hosts also implement `IIsolatedLayoutHost` (an extension of `INotebookService` in `Verso.Blazor.Shared.Services`). It covers frame allocation, renderer-package fetch, mount and unmount lifecycle, the message pump between the frame and the kernel, and theme propagation. Blazor Server implements it in-process; Blazor WASM implements it over JSON-RPC.
@@ -247,7 +255,7 @@ Kernel calls context.RequestInputAsync(prompt, isPassword, ct)
 | `NotebookHandler` | open, close, save, getLanguages, getCellTypes, setFilePath |
 | `CellHandler` | add, insert, remove, move, updateSource, changeType, changeLanguage |
 | `ExecutionHandler` | run, runAll, cancel |
-| `KernelHandler` | restart, getCompletions, getDiagnostics, getHoverInfo |
+| `KernelHandler` | restart, getCompletions, getDiagnostics, getHoverInfo, getSemanticTokens |
 | `OutputHandler` | clearAll |
 | `InteractionHandler` | interact (cell interaction dispatch) |
 | `LayoutHandler` | getLayouts, switch, render, getCellContainer |

docs/extensions/extension-interfaces.md

@@ -674,7 +674,7 @@ Interaction types in the `verso/` namespace are reserved by the host and are int
 
 ## Augmentation Interfaces
 
-These interfaces are not standalone extension capabilities. They are implemented **alongside** a primary capability interface (e.g., `ILanguageKernel + IExtensionSettings` or `IDataFormatter + ICellInteractionHandler`). They do not extend `IExtension` and cannot be the sole interface on a `[VersoExtension]` class.
+These interfaces are not standalone extension capabilities. They are implemented **alongside** a primary capability interface (e.g., `ILanguageKernel + IExtensionSettings`, `ILanguageKernel + ISemanticTokenProvider` or `IDataFormatter + ICellInteractionHandler`). They do not extend `IExtension` and cannot be the sole interface on a `[VersoExtension]` class.
 
 ### IExtensionSettings
 
@@ -755,6 +755,24 @@ public sealed class MyKernel : ILanguageKernel, IExtensionSettings
 }
 ```
 
+### ISemanticTokenProvider
+
+Classifies the symbols in a cell, so the editor can color a type, a function and a value differently where its lexer sees the same identifier. Implemented alongside `ILanguageKernel`; the C# and F# kernels implement it.
+
+| Member | Type | Description |
+|---|---|---|
+| `GetSemanticTokensAsync(string)` | `Task<IReadOnlyList<SemanticToken>?>` | Classifies the cell's source. `null` when the code could not be analyzed. |
+
+A `SemanticToken` is `(Line, StartColumn, Length, Type, Modifiers)`. Positions are 0-based and relative to the cell's source, as for a `Diagnostic`; `Length` counts UTF-16 characters, as .NET strings and the editor do, and a token lies on one line. Tokens may come in any order; the editor sorts them and drops a token that overlaps the one before it, one past the end of its line, and one of a type it does not know.
+
+`Type` is one of the `SemanticTokenTypes` constants, which are the Language Server Protocol's standard names: `namespace`, `type`, `class`, `struct`, `enum`, `interface`, `typeParameter`, `parameter`, `variable`, `property`, `enumMember`, `event`, `function`, `method`, `keyword` and `operator`. `Modifiers` holds `SemanticTokenModifiers` (`mutable`, `static`) or is `null`. The theme colors each type by one of its syntax colors (see [Theme Authoring](theme-authoring.md#syntax-color-mapping)).
+
+Report what the lexer cannot know and leave out what it already colors. A keyword, a string or a number gains nothing from a token, and a token whose type the theme does not color is drawn in the editor's foreground color, which can be worse than the lexer's guess. A computation expression builder in F# (`async`, `seq`) is the exception: the lexer reads it as an identifier, and a `keyword` token gives it the keyword color.
+
+The editor asks when a cell is first shown, after the user pauses typing, and after a run, for the cell that ran and every cell below it that shows kernel markers; it asks again on focus when another cell has run since the last answer. It does not ask while any cell in the notebook is running, and the colors already shown stay until the next answer. A request goes out from one editor at a time. Like diagnostics, the request is advisory: `null`, an exception or a kernel that cannot start keeps the colors shown, and an empty list clears them. Leading `#!` magic lines are blanked before the code reaches the kernel.
+
+A classification usually falls out of the same type check as `GetDiagnosticsAsync`, and the editor asks for both after an edit. A kernel that implements both should answer them from one check of the same text, so a pause in typing compiles once; the C# and F# kernels do.
+
 ### ICellInteractionHandler
 
 Handles bidirectional interactions from rendered cell content. Implement alongside a formatter or renderer to support interactive outputs (e.g., paginated tables, inline forms, drill-down panels).

docs/extensions/theme-authoring.md

@@ -248,7 +248,7 @@ How much work the shadow has to do depends on the palette. Dark themes separate
 
 ## Syntax Color Mapping
 
-Themes provide syntax highlighting via `GetSyntaxColors()`, which returns a `SyntaxColorMap`. The platform recognizes 12 standard token types:
+Themes provide syntax highlighting via `GetSyntaxColors()`, which returns a `SyntaxColorMap`. The platform recognizes 13 standard token types:
 
 | Token Type | Description | Light Default | Dark Default |
 |------------|-------------|--------------|-------------|
@@ -263,9 +263,12 @@ Themes provide syntax highlighting via `GetSyntaxColors()`, which returns a `Syn
 | `punctuation` | Brackets, semicolons | `#000000` | `#D4D4D4` |
 | `preprocessor` | Preprocessor directives | `#808080` | `#C586C0` |
 | `attribute` | Attributes and decorators | `#267F99` | `#4EC9B0` |
-| `namespace` | Namespace references | `#267F99` | `#4EC9B0` |
+| `namespace` | Namespace and module references | `#267F99` | `#4EC9B0` |
+| `enumMember` | Enum members, union cases and named constants | `#0070C1` | `#4FC1FF` |
 
-Always provide all 12 tokens. Renderers may fall back to `EditorForeground` for missing tokens.
+Always provide all 13 tokens. Renderers may fall back to `EditorForeground` for missing tokens.
+
+The same colors apply to the semantic tokens a kernel reports (see [`ISemanticTokenProvider`](extension-interfaces.md#isemantictokenprovider)). A class, struct, enum, interface or type parameter takes the `type` color; a method takes `function`; a parameter, property or event takes `variable`. A theme without `enumMember` colors enum members and union cases by its `type` color.
 
 ## Overriding Defaults
 

src/Verso.Abstractions/Extensions/ISemanticTokenProvider.cs

@@ -0,0 +1,27 @@
+namespace Verso.Abstractions;
+
+/// <summary>
+/// Augmentation interface for language kernels that can classify the symbols in a cell, so the
+/// editor can color a type, a function and a value differently where a lexer alone sees the same
+/// identifier. A kernel implements this alongside <see cref="ILanguageKernel"/>. Implementing
+/// <see cref="ISemanticTokenProvider"/> alone is not a valid extension capability.
+/// </summary>
+/// <remarks>
+/// The editor asks after the user pauses typing and when the cell's kernel state may have changed
+/// (its run finished, the kernel restarted), and keeps the colors it already shows while a cell is
+/// running. A classification is usually a by-product of the type check that
+/// <see cref="ILanguageKernel.GetDiagnosticsAsync"/> runs on the same text, and the editor asks for
+/// both after an edit, so a kernel answers both from one check of that text.
+/// </remarks>
+public interface ISemanticTokenProvider
+{
+    /// <summary>
+    /// Classifies the symbols in <paramref name="code"/>.
+    /// </summary>
+    /// <param name="code">The full source text of the cell.</param>
+    /// <returns>
+    /// The cell's tokens, in any order, or <c>null</c> when the code could not be analyzed. The
+    /// editor keeps the colors it already shows on <c>null</c>; an empty list clears them.
+    /// </returns>
+    Task<IReadOnlyList<SemanticToken>?> GetSemanticTokensAsync(string code);
+}

src/Verso.Abstractions/Models/SemanticToken.cs

@@ -0,0 +1,74 @@
+namespace Verso.Abstractions;
+
+/// <summary>
+/// One classified span of a cell's source, returned by <see cref="ISemanticTokenProvider"/>.
+/// A token lies on a single line.
+/// </summary>
+/// <param name="Line">The zero-based line of the token, relative to the cell's source as for a <see cref="Diagnostic"/>.</param>
+/// <param name="StartColumn">The zero-based column where the token begins, in UTF-16 characters.</param>
+/// <param name="Length">The token's length in UTF-16 characters (as <see cref="string.Length"/> counts them).</param>
+/// <param name="Type">
+/// What the token names, one of <see cref="SemanticTokenTypes"/>. The editor ignores a token of a
+/// type it does not know.
+/// </param>
+/// <param name="Modifiers">
+/// Refinements of <paramref name="Type"/>, from <see cref="SemanticTokenModifiers"/>, or <c>null</c>
+/// for none. The editor ignores a modifier it does not know.
+/// </param>
+public sealed record SemanticToken(
+    int Line,
+    int StartColumn,
+    int Length,
+    string Type,
+    IReadOnlyList<string>? Modifiers = null);
+
+/// <summary>
+/// The token types a <see cref="SemanticToken"/> may carry. The names are the standard ones of the
+/// Language Server Protocol, which the editor's themes color by.
+/// </summary>
+public static class SemanticTokenTypes
+{
+    /// <summary>A namespace, or a module that groups declarations the way one does (an F# module).</summary>
+    public const string Namespace = "namespace";
+    /// <summary>A type with no more specific kind below, or a type abbreviation.</summary>
+    public const string Type = "type";
+    /// <summary>A class, record class, exception or other reference type.</summary>
+    public const string Class = "class";
+    /// <summary>A struct or other value type.</summary>
+    public const string Struct = "struct";
+    /// <summary>An enumeration.</summary>
+    public const string Enum = "enum";
+    /// <summary>An interface.</summary>
+    public const string Interface = "interface";
+    /// <summary>A generic type parameter.</summary>
+    public const string TypeParameter = "typeParameter";
+    /// <summary>A function or method parameter.</summary>
+    public const string Parameter = "parameter";
+    /// <summary>A variable or value.</summary>
+    public const string Variable = "variable";
+    /// <summary>A property or field, including a record field.</summary>
+    public const string Property = "property";
+    /// <summary>An enumeration member, a union case or a named constant.</summary>
+    public const string EnumMember = "enumMember";
+    /// <summary>An event.</summary>
+    public const string Event = "event";
+    /// <summary>A function.</summary>
+    public const string Function = "function";
+    /// <summary>A method.</summary>
+    public const string Method = "method";
+    /// <summary>A keyword that a lexer cannot recognize as one (an F# computation expression builder).</summary>
+    public const string Keyword = "keyword";
+    /// <summary>An operator.</summary>
+    public const string Operator = "operator";
+}
+
+/// <summary>
+/// The modifiers a <see cref="SemanticToken"/> may carry.
+/// </summary>
+public static class SemanticTokenModifiers
+{
+    /// <summary>A value or field that can be reassigned.</summary>
+    public const string Mutable = "mutable";
+    /// <summary>A member that belongs to its type rather than to an instance.</summary>
+    public const string Static = "static";
+}

src/Verso.Blazor.Shared/Components/Editor/MonacoEditor.razor

@@ -12,6 +12,7 @@
     [Parameter] public Func<string, int, Task<object?>>? OnGetHoverInfo { get; set; }
     [Parameter] public Func<string, int, Task<object?>>? OnGetCompletions { get; set; }
     [Parameter] public Func<string, Task<object?>>? OnGetDiagnostics { get; set; }
+    [Parameter] public Func<string, Task<object?>>? OnGetSemanticTokens { get; set; }
 
     /// <summary>
     /// When true, the editor issues no diagnostics requests. A refresh asked for while suspended
@@ -92,9 +93,9 @@
     }
 
     /// <summary>
-    /// The cell has finished running: ask again for this editor's diagnostics and for every
-    /// editor below it that is showing kernel markers, nearest first, since the run may have
-    /// changed what their code refers to.
+    /// The cell has finished running: ask again for this editor's diagnostics and semantic
+    /// tokens, and for every editor below it that is showing kernel markers, nearest first,
+    /// since the run may have changed what their code refers to.
     /// </summary>
     public async Task RefreshAfterRunAsync()
     {
@@ -171,6 +172,18 @@
         catch { return null; }
     }
 
+    [JSInvokable]
+    public async Task<object?> GetSemanticTokens(string code)
+    {
+        try
+        {
+            return OnGetSemanticTokens is not null
+                ? await OnGetSemanticTokens(code)
+                : null;
+        }
+        catch { return null; }
+    }
+
     public async ValueTask DisposeAsync()
     {
         if (_initialized)

src/Verso.Blazor.Shared/Components/Notebook/Cell.razor

@@ -193,6 +193,7 @@
                               OnGetHoverInfo="HandleGetHoverInfo"
                               OnGetCompletions="HandleGetCompletions"
                               OnGetDiagnostics="HandleGetDiagnostics"
+                              OnGetSemanticTokens="HandleGetSemanticTokens"
                               DiagnosticsSuspended="IsNotebookExecuting"
                               OnLineLimitChanged="HandleLineLimitChanged" />
                 @if (_lineLimit is { Clipped: true } limit)
@@ -736,6 +737,33 @@
         };
     }
 
+    private async Task<object?> HandleGetSemanticTokens(string code)
+    {
+        // Markdown has no kernel to ask; an empty set leaves the editor's own coloring.
+        if (string.Equals(CellData.Type, "markdown", StringComparison.OrdinalIgnoreCase))
+            return new { items = Array.Empty<object>() };
+
+        // Masked as for diagnostics, so a leading magic line is not classified as code and
+        // positions still line up.
+        var result = await Service.GetSemanticTokensAsync(
+            CellData.Id, DiagnosticsSource.MaskLeadingMagicCommands(code));
+
+        // Null means the request could not be answered; the editor keeps the colors it shows.
+        if (result is null) return null;
+
+        return new
+        {
+            items = result.Items.Select(t => new
+            {
+                line = t.Line,
+                startColumn = t.StartColumn,
+                length = t.Length,
+                type = t.Type,
+                modifiers = t.Modifiers
+            }).ToArray()
+        };
+    }
+
     private void ToggleTypeDropdown()
     {
         _showTypeDropdown = !_showTypeDropdown;