Conversation
Replace per-file GVFS.Common links with a project reference now that NativeAOT trimming keeps hook startup flat. Keep Windows shared source links for hook platform helpers. Assisted-by: GPT-5.6 Sol Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
A sparse index (git index.sparse) stores a collapsed subtree as one directory entry with mode 040000 and a trailing '/'. GVFS builds its projection from the index and cannot expand such an entry. Today the mount ingests it as an ordinary entry, reports the repository ready, then crashes about three seconds later inside a ProjFS enumeration callback, because the trailing slash produces an empty child name. That projection also silently omits every collapsed subtree. This replaces the late crash and silent data loss with an early, actionable failure. The earlier prediction that the parser would reject mode 040000 with "Invalid file type" is wrong on Windows. The whole mode-validation block runs only when SupportsFileMode is true, and WindowsFileSystem returns false, so the parser never reads the mode field. Detection must not depend on the mode. Increment 1 - recognise a sparse-directory entry: - Add FileType.Directory and map mode 040000 (0x4000) to it in FileTypeAndMode, so macOS (SupportsFileMode true) stops throwing "Invalid file type" and takes the same path as Windows. - Add the parse-mode switch case that requires mode bits to be zero. - Add GitIndexEntry.PathEndsInSlash, which detects the entry from the trailing '/' in the raw path buffer, with no per-entry allocation and no dependence on the mode field. Increment 2 - fail fast before the mount reports ready: - FailIfSparseDirectoryEntry throws InvalidDataException with a message that names the offending path and tells the user to run 'git sparse-checkout disable' to expand the index. It runs from both ValidateIndexEntry and the projected-entry branch of AddIndexEntryToProjection. The exception propagates through Initialize and TryStart to FailMountAndExit, which exits before "Virtual repo is ready". The two increments land together on purpose. Recognition alone is unsafe: it would make macOS reach the same enumeration crash instead of throwing early, so it must ship with the fail-fast. The fail-fast is a bug fix for a crash and silent data-loss risk, not a new feature, so it is not gated behind a config flag, matching the existing unconditional index-validation checks. Tests: a self-contained v4 index builder emits entries with arbitrary mode and trailing-slash paths. Coverage includes FileTypeAndMode mapping, parse acceptance of 040000 on both platforms, the fail-fast message, a malformed directory with non-zero mode bits, v4 prefix compression after a trailing-slash entry, and an unchanged full index. Full suite: 1008 tests, 997 passed, 0 failed, 11 skipped. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Introduce the pure, testable core for automatic cone management behind a new default-off git-config flag (gvfs.auto-sparse-index). No runtime behavior changes while the flag is off. - ConeBuilder maps a set of modified paths to a cone pattern set: parent-only patterns for modified files and their ancestors, recursive patterns for modified folders. Patterns covered by a recursive ancestor are removed. A single modified file never produces a recursive pattern. - ConeFileWriter serializes a cone pattern set to git's exact cone-mode info/sparse-checkout format, escaping glob-special characters as git does, and writes it atomically with a backup for rollback. Legacy /.gitattributes content is detected. - SparseCheckoutPathResolver resolves the per-worktree sparse-checkout file path, using the per-worktree git dir for a linked worktree instead of the shared git dir. This is pure logic with no live-enlistment mutation. Unit tests cover the design worked examples, edge cases, glob escaping, and primary-versus-worktree path resolution. Assisted-by: Claude Sonnet 4.5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Add the GVFS.Hooks side of automatic sparse-index support, gated behind the gvfs.auto-sparse-index config flag (default false). When the flag is off, only a single cheap parse runs and no new pipe traffic or config read occurs. Part 1 - table-driven Git command-line parser. GitPathspecParser and ParsedGitCommand (in GVFS.Common) extract the working-tree pathspecs a Git command names. That set is the input the mount needs to widen the sparse-index cone before Git runs. The parser is a per-command grammar table keyed by a positional kind: add, stage, rm, mv, restore and commit treat positionals as pathspecs; checkout and reset take a leading ref then pathspecs; switch takes none; stash push and save take pathspecs; diff, log, show, grep, blame and unknown commands only take paths after "--". It handles "--" separators, combined short flags, value options in both "--opt value" and "--opt=value" forms, --pathspec-from-file, --pathspec-file-nul, a leading run of Git globals (-C, -c, --git-dir, --work-tree, ...), and filters the --git-pid and --exit_code tokens Git injects into hook argument lists. It does not modify the existing GitCommandLineParser, UnstageCommandParser or WorktreeCommandParser, which have live callers. A table-driven parser was chosen over System.CommandLine after a measured comparison (standalone NativeAOT harness, n=200 warm and n=50 cold, run twice). The table cost +52 KB of binary and no measurable startup change; System.CommandLine cost +1.52 MB and +1.5 ms warm / +24 ms cold per invocation. GVFS.Hooks.exe runs on every Git command, so per-invocation startup is direct user-visible latency. Part 2 - hook-to-mount cone-management contract (client side). NamedPipeMessages.ConeManagement defines the widen and narrow requests and their result codes, with NUL-separated body encoding. The hook client sends a widen before a path-naming command and a narrow after it, keyed by the Git session id. Both are bounded and non-blocking on failure: a missed widen only makes Git transiently expand the in-memory index and re-collapse it, so every failure path proceeds silently and never blocks Git. The mount-side handler is owned by the cone-management work and is intentionally not implemented here. Tests: 46 parser cases and 8 pipe round-trip cases. Full unit suite 1040 passed, 0 failed, 11 skipped. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Add the CLI surface for the built-in sparse index, a feature distinct from gvfs sparse: gvfs sparse narrows the projection, while gvfs sparse-index keeps the projection full and lets the on-disk .git/index collapse to cone entries. New verb gvfs sparse-index (SparseIndexVerb): - --status (default): reports whether gvfs.auto-sparse-index is enabled, whether the on-disk index is sparse, the entry count, and the index size. These numbers are the feature's value proposition, so the verb surfaces them. - --enable: refuses when gvfs sparse has a projection sparse-set (mutual exclusion, with a message explaining the projection-vs-index distinction), otherwise sets gvfs.auto-sparse-index plus core.sparseCheckoutCone, index.sparse, and sparse.expectFilesOutsideOfPatterns. - --disable: the recovery path. It unmounts if needed, expands the index with update-index --force-write-index under neutralized hooks, clears the config, and reports the before and after entry count and size. A bare git sparse-checkout disable fails when the repo is unmounted, because the pre-command and virtual-filesystem hooks abort with no mount to answer the named pipe, and disable forces a worktree update that unmounted ProjFS rejects. GitProcess.ForceExpandSparseIndex expands the index in place without a worktree update, which is non-destructive. GitIndexInspector and GitIndexInfo read the DIRC header and scan entries for the first directory-mode entry to power --status. The scan is version 2, 3, and 4 aware and handles v4 prefix compression. Config plumbing is gated behind gvfs.auto-sparse-index, default false: - RequiredGitConfig.GetRequiredSettings gains a bool overload that adds the three sparse keys only when enabled; the no-argument overload delegates with false, so config is byte-for-byte unchanged for repos that never opt in. - InProcessMount reads the flag and passes it through. - gvfs clone --sparse-index writes the flag at clone time. Unit tests cover config parity (default equals disabled, enabled adds exactly three keys) and the index parser (full and sparse indexes across v2 and v4). Assisted-by: Claude Opus 4.7 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
…ndex Brings in detection of git sparse-directory index entries (040000, trailing slash) with an unconditional parse-time fail-fast before the mount reports ready. See decisions/0003-sparse-index-fail-fast.md. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Adds the gvfs.auto-sparse-index flag (default off), the cone builder, cone file writer, and sparse-checkout path resolver in GVFS.Common/Sparse. See decisions/0004-cone-membership-under-vfs.md. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Adds the table-driven GitPathspecParser and ParsedGitCommand in GVFS.Common, the cone-management named-pipe contract (client side), and the GVFS.Hooks pre/post-command cone-widen/narrow calls. See decisions/0006-hooks-parser-table-driven.md. Conflict resolved: GVFSConstants.cs gvfs.auto-sparse-index constant. W6 and W7 each added it in different regions of GitConfig, so the merge produced two identical definitions (CS0102 duplicate member). Kept W6's copy (introduced first, after PrefetchOffload) and removed W7's duplicate block. Key string and default value were identical, so no reference change was needed. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Adds the gvfs sparse-index verb (--status/--enable/--disable), GitIndexInspector,
GitProcess.ForceExpandSparseIndex, the flag-gated RequiredGitConfig overload,
mount re-assertion of sparse keys, and clone --sparse-index. See
decisions/0007-sparse-index-cli-and-recovery.md.
Conflicts resolved:
1. GVFSConstants.cs gvfs.auto-sparse-index constant (three-way with W6 and W7).
The constant name and default value were identical across all three; only
the comment differed. Kept W6's copy (introduced first). W8's separate
SparseIndex verb-name constant merged cleanly and is retained.
2. W5 recovery message (cross-workstream, per ADR 0007). W8 proved W5's original
instruction ("git sparse-checkout disable") fails twice over on an unmounted
GVFS enlistment (pre-command hook abort; then a working-tree update ProjFS
rejects). Replaced it with the ADR 0007 wording pointing at
'gvfs sparse-index --disable'. W5 could not apply this itself because
FailIfSparseDirectoryEntry lived on a sibling branch. Also updated W5's
SparseIndexParsingTests assertion, which checked for the old text.
Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Replaces the 29 linked GVFS.Common source files in GVFS.Hooks.csproj with a single ProjectReference to GVFS.Common, and drops the now-duplicate _GenerateConstantsFile target (GVFS.Common.csproj already owns it). See decisions/0002-hooks-common-reference.md. Conflict resolved (semantic, per the W10 task and ADR 0002): W7 had added three new linked files to GVFS.Hooks.csproj (GitPathspecParser.cs, ParsedGitCommand.cs, ConeManagementNamedPipeMessages.cs). W9 removes all links in favor of the ProjectReference, which already brings those types in from GVFS.Common. Keeping both would produce duplicate-type / ambiguous-reference errors, so the resolution takes W9's csproj in full, dropping W7's now-redundant links. GitPathspecParser and the cone-management messages resolve through the ProjectReference. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Add repeatable trace2 workloads for full-index and sparse-index comparisons. Capture timing spread, index state, projection size, and expansion reasons. Assisted-by: GPT-5.6 Sol Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
A sparse index (git index.sparse) collapses a folder to a single entry with mode 040000, a tree object id, and a trailing '/'. Earlier work made the projection recognise such an entry and fail the mount fast, because building a projection from it silently omitted the collapsed subtree and later crashed ProjFS enumeration. This change adds real expansion of the collapsed subtree, so the projection serves the full working directory from a sparse index. Expansion is gated by the existing gvfs.auto-sparse-index config flag, default false. When the flag is off the fail-fast is unchanged, so this is a no-op for every current enlistment. Increment 1 - local tree reader (GVFS.Common.Git): - TreeEnumeration.cs: TreeEnumerationResult (Success, MissingTree, CorruptTree, NotTree) and a TreeEntryVisitor delegate. The states are distinct so a caller can download-then-retry a missing tree but fail fast on corruption or a non-tree object. - LibGit2Repo.EnumerateTree: reads a local tree through the in-process libgit2 repo, calling the visitor once per entry with the raw UTF-8 name, the 20-byte object id, the git file mode, and whether the entry is a subtree. It maps a failed open to MissingTree or CorruptTree using ObjectExists, and reports a non-tree object as NotTree. Added the git_tree_entry_name interop. - GitRepo.TryEnumerateTree wraps it through the libgit2 invoker; an unavailable repo reports MissingTree so the caller can try a download. Increment 2 - local tree expansion (GVFS.Virtualization.Projection): - GitIndexProjection.SparseDirectoryExpander.cs: an iterative (explicit-stack) DFS expander. It reads each tree through IProjectionTreeReader and adds the children to the projection in the same FolderData / FileData shape the parser already produces for a full index. It reuses one visitor delegate and one 20-byte SHA buffer, so it adds no per-entry allocation. It preserves mode data (symlink, gitlink, executable) into nonDefaultFileTypesAndModes on platforms that parse the mode field. It carries the gvfs sparse-cone inclusion down the walk in O(1) per folder (FrameInclusion), mirroring SortedFolderEntries.GetOrAddFolder. - GitIndexEntry: IsSparseDirectory / ProjectionPathLength and a BuildingProjection_ParsePath(int) overload that parses the path without its trailing '/'. This is the fix for the original crash: the standard parse of "GVFS/" produces an empty final part that ProjFS rejects; the logical-length parse produces the collapsed folder name instead. The overload does not touch PathBuffer at or beyond the logical length and does not disturb the previous-separator state, so index-v4 prefix decompression of the next entry is unaffected. - SortedFolderEntries.AddFolder and FolderData.AddChildFolder insert a new child folder by binary search, so git tree order is handled. AddFile and AddFolder now reject an empty name, and LazyUTF8String gains an allocation-free IsEmpty. Increment 3 - missing tree download: - The expander downloads a missing tree once and retries the read. A permanent miss, a corrupt tree, or a non-tree object throws InvalidDataException and fails the projection build, so GVFS never serves a partial working directory, per the CONTRIBUTING.md rule for data-loss risk. The download is synchronous under the projection write lock; see the ADR for that decision. Cache replay (the persisted-projection reuse that avoids a full tree walk on every index change) is not in this change. Expansion stays behind the default- off flag so the slow cold-walk path is never enabled by default. Tests (GVFS.UnitTests): - SparseDirectoryExpanderTests: a fake IProjectionTreeReader serves synthetic trees. Cases: flat and nested expansion into FolderData / FileData; mode recording on and off by platform; sparse-cone inclusion; missing-then- download-then-retry; and fail-fast on a permanent miss, a corrupt tree, a non-tree object, and an empty entry name. - GitIndexEntryTests: the logical-length parse strips the trailing '/' for a top-level and a nested sparse-directory path, and a regular file is not detected as a sparse directory. - Full suite: 1137 tests, 1126 passed, 0 failed, 11 skipped (the 11 are the native PostIndexChangedHook tests that need the built hook exe). Baseline 1121/1110/0/11 preserved; 16 new tests, all pass. Build: 0 warnings, 0 errors. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Three fixes let the harness run against a collapsed sparse index. They are timing-neutral for the measured operations. - Sparse-directory guard: a tracked path inside a collapsed directory is not listed by `ls-files --sparse`; only its containing directory appears as a sparse-directory entry. Accept the path when it has a sparse-directory ancestor, instead of requiring an exact leaf entry that a collapsed index never holds. - Cleanup restore: on a collapsed index `checkout -- <path>` cannot match a path inside a sparse directory, and `git add` would force a full-index expansion. Restore the tracked fixture path from the base-commit blob with `cat-file blob`, which needs no pathspec match and no expansion. Full mode keeps the original checkout. - Fixture builder: the old path round-tripped the whole index (read-tree plus write-tree). On a multi-million-entry index that rebuilds the entire cache-tree and, in a partial clone, triggers a promisor fetch. Build the fixture commit by editing only the tree path that changes, with recursive ls-tree plus mktree. It is offline and produces an identical commit. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
…jection Adds SparseIndexProjectionTests, an end-to-end functional test that proves GVFS projects a collapsed sparse index correctly when gvfs.auto-sparse-index is on. Before this change the sparse-directory expansion path had only unit coverage. The failure it guards against is silent: a collapsed sparse-directory entry projected a truncated folder, so a file deep inside (GVFS/FastFetch/Program.cs) read as "does not exist" and the folder listed only a few of its files, with no crash. The test provisions its own clean enlistment, collapses the index on the live mount with `git -c index.sparse=true update-index --force-write-index` (which rewrites the index in place without touching the working tree, so it works on a ProjFS enlistment where `sparse-checkout reapply` cannot open the in-cone placeholders), drops the cached projection, and remounts with the flag on. It then asserts: - the on-disk index stays collapsed (read from the raw DIRC header, and via ls-files --sparse, to avoid forcing expansion with git ls-files), - the collapsed directory enumerates its full HEAD-tree child set, - the deep file reads with content that byte-matches the committed blob, - the index is still collapsed at read time, and - the mount stays Ready past the window in which the truncated projection used to crash at larger scale. The clean per-fixture enlistment also avoids the stale ProjFS folder placeholder that survives a remount once a collapsed folder is enumerated mid-window. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Strengthen the collapsed sparse-index functional test to prove tree-wide completeness by set equality instead of spot checks. The test now captures the expected file set from `git ls-tree -r HEAD`, then after collapsing the index on the live mount and remounting with gvfs.auto-sparse-index on, does a full recursive enumeration of the projected working tree and asserts the two sets are identical. On mismatch it reports the symmetric difference so any missing (silently lost) or extra (spurious) path is named. This subsumes the earlier FastFetch / Program.cs spot checks and guards against silent file loss anywhere in the tree. Also keep the direct content proof (Program.cs byte-matches the committed blob) and the on-disk index-stays-collapsed assertions at mount, at read, and after the read, so a projection that came from a re-expanded index is caught. Result on the test enlistment: index collapses 864 -> 31 entries; the flag-on remount projects 864 files, exactly equal to HEAD (864). No file is lost. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
During conflict resolution a user names the conflicted file directly, e.g. "git add out/f.txt" or "git checkout --ours out/f.txt". When that file is outside the sparse-index cone, plain cone mode refuses the operation and the index is left fully expanded, so "git rebase --continue" cannot proceed. The pre-command hook already sends a cone-widen request for any command that names a path, so the "git add"/"git rm"/"git restore"/"git checkout -- <path>"/"git reset -- <path>" cases are covered. The gap was "git checkout --ours/--theirs/-p <path>": those flags carry no ref operand, so the conflicted path is the first positional, but the checkout grammar (one leading ref) misread it as a branch and never widened for it. Teach GitPathspecParser a per-command set of ref-suppressing flags. When one is present before "--", the command has no ref operand and every positional is a pathspec. Set --ours/--theirs/--patch/-p for checkout. Detection scans the whole command so flag order does not matter; past "--" a matching token stays a literal path. Add unit tests covering each conflict-resolution command, the ref-suppressing flags, flag-after-path ordering, and the "-- --ours" literal-path edge. New behaviour stays gated behind gvfs.auto-sparse-index (default false). Empirical measurements and the git-layer end-to-end proof are recorded in the project's ADR 0012 and benchmarks/raw/w7-ooc. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
gvfs clone --sparse-index previously only wrote git config; the enlistment
still came up with a full index at its first mount, so a user had to pay the
one-time collapse of a large full index later. That collapse is cheap only
when the index has a valid cache-tree; after any index-mutating command it is
much slower, and undoing a hydrated full enlistment is more expensive still.
Build the sparse index during clone instead. After the full index is written
by ForceCheckout, and gated on --sparse-index, collapse it in place so the
enlistment is already sparse before its first mount:
1. Build the minimal cone (root files only) from an empty modified-path set
with the existing ConeBuilder, and write info/sparse-checkout in git's
cone format with ConeFileWriter.
2. Set core.sparseCheckout=true. The three sparse-index keys are already
written by TrySetRequiredGitConfigSettings when --sparse-index is set.
3. Collapse the full index with 'update-index --force-write-index' and the
GVFS hooks disabled (no mount is running yet). The fresh index has
skip-worktree on every entry and a valid cache-tree, so git collapses the
out-of-cone directories without walking the working tree.
This runs before the ProjFS provider is registered, so it uses the unmounted
incantation: core.virtualfilesystem and core.hookspath are cleared, and
sparse.expectFilesOutsideOfPatterns=true keeps skip-worktree on the present
files so the collapse is not undone. It is the mirror of ForceExpandSparseIndex.
All of this stays behind gvfs.auto-sparse-index (default off). Any failure in
the construction fails the clone loudly; the index remains a valid full index,
so there is no data-loss risk. Adds two unit tests for the collapse command.
Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Prove 'gvfs clone --sparse-index' constructs a sparse index during the clone, so the enlistment is already sparse before its first mount, with no post-hoc collapse. The functional-test harness already provisions enlistments by running a real 'gvfs clone'. Extend that path with an optional --sparse-index and --no-mount, then add a test that clones, reads the on-disk index before the first mount, mounts, and asserts the mount projects the complete HEAD tree from the collapsed index. - GVFSProcess.Clone: add a sparseIndex/noMount overload. The old three-argument signature delegates to it, so every existing caller is unchanged. - GVFSFunctionalTestEnlistment: add CloneNoMount, which clones without mounting so the caller can inspect the on-disk index before the first mount and time the first mount itself. - CloneSparseIndexTests: an end-to-end test plus a plain-clone control. It reads the DIRC header and 'git ls-files --sparse' directly, never plain 'git ls-files', which forces the sparse index to expand. It asserts the pre-mount index is sparse, the required config is set, the first mount reaches and stays Ready, the projected working tree equals 'git ls-tree -r HEAD' by set equality, and a file inside a collapsed directory reads with the committed bytes. Live result against the test remote: the --sparse-index clone produced a 31-entry index before the first mount versus 864 for a plain clone, its first mount projected all 864 files with none missing or extra, and it reached Ready. Unit suite 1139 passed, 0 failed. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The sparse-index collapse in CloneVerb runs after ForceCheckout, not before. Explain why in a comment so a future reader does not "optimize" by moving cone and sparse-checkout config ahead of the checkout: doing so leaves core.virtualfilesystem active during checkout, apply_virtualfilesystem re-expands the index in-process, and the resulting full index can no longer be collapsed. The clean-checkout-then-collapse order is required, not incidental. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
An earlier comment stated that setting the cone and core.sparseCheckout before ForceCheckout can never produce a sparse index. That is true only for a git whose virtualfilesystem force-expands a collapsed index. With a sparse-index-aware git, checkout writes a sparse index directly. Restate the comment accurately: checkout-then-collapse is kept because it is robust across git versions (it yields a sparse index with both), not because direct construction is impossible. A direct mechanism test confirms the two-git behavior; see decisions/0015. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The pre-command hook parses a git command line, extracts the paths it names, and sends a ConeWiden request to the mount before git runs (ConeNarrow after). Nothing on the mount answered those requests, so the hook sent widen requests into the void and the cone builder was never called. This adds the missing mount-side handler and wires it into the request dispatch. On widen, the handler resolves the requested pathspecs to repository paths, adds them as transient entries, recomputes the cone as the union of the modified-path entries plus all live transient entries, writes it, collapses the on-disk index in place, reparses the projection, and only then replies so the hook can let git proceed. On narrow, it drops the session's transient entries and recomputes. The collapse uses update-index --force-write-index, never sparse-checkout reapply, which walks the working tree and fails on a live ProjFS mount. It runs with the virtual filesystem on, the hooks path at its default, and the read-object hook on; disabling any of the three fails the in-process index write against the live provider. Recursion is prevented by disabling only the pre-command hook (COMMAND_HOOK_LOCK), so the collapse git cannot re-enter the mount pipe. The handler holds no GVFS lock. The triggering git command already holds it through the pre-command hook, so acquiring it here would deadlock; the held lock also satisfies the mount's index-rename gate, which is why the collapse can write the index. Cone operations are serialized by a process-internal lock. Everything is behind gvfs.auto-sparse-index (default false); with the flag off the handler replies NotEnabled and behaviour is unchanged. Every failure is fail-open: the previous sparse-checkout file is restored and git proceeds, transiently expanding and re-collapsing the index. Adds ConePathspecResolver and TransientConeState with unit tests, and a functional test that widens for an out-of-cone git add and asserts the index stays partially expanded. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
When gvfs.auto-sparse-index is on, GVFS owns .git/info/sparse-checkout. A user who runs git's own sparse-checkout tooling, or edits the file by hand, hits silent overwrites, a full-index degrade, or cryptic failures. Hook-side guard: extend CheckForLegalCommands to block the mutating sparse-checkout subcommands (set, add, reapply, init, disable) and redirect to gvfs sparse-index, naming gvfs sparse-index --disable for disable. Read the subcommand from GitPathspecParser so global options before the subcommand resolve correctly. Allow the read-only and working-tree subcommands (list, check-rules, clean) and any unknown/future subcommand. Block only when gvfs.auto-sparse-index is true, read once inside the sparse-checkout case to protect the per-command hook budget. With the flag off, git sparse-checkout behaves exactly as before. Mount-side drift detection: at cone recompute, compare the on-disk cone against what GVFS last wrote with a cheap ordinal compare of the small file. On drift, warn and overwrite, because GVFS owns the file. Detect the non-cone-format hand-edit that silently disables the sparse index via git's is_sparse_index_allowed(), and warn about that cause specifically. Internal GVFS git calls are not blocked: git skips the pre-command hook when COMMAND_HOOK_LOCK is set (usePreCommandHook: false), and the mount-side collapse invokes update-index, not sparse-checkout. Adds 45 unit tests and one functional test proving set is refused with the feature on and runs with it off. Full unit suite: 1207 passed, 0 failed. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The automatic sparse-index cone-widen hook waited 250 ms for the mount's reply and discarded the result. The mount reparsed the projection synchronously before replying, putting the O(repo) reparse on the reply path, so at large-repo scale the reply took seconds and the hook always timed out -- Git then ran against a not-yet-confirmed cone. The index collapse alone (~257 ms of the 272.8 ms reply, 95%) also exceeded the old budget even at small scale, so the "block Git until applied" guarantee was soft. The 250 ms budget was modeled on the cached-hydration-status display, a cosmetic feature whose timeout is genuinely benign; applying it to this functional feature (whose timeout costs a full index expansion) was the error, and it also dropped that precedent's result check. Decouple the reparse from the reply path: the handler now triggers an asynchronous projection invalidate (RequestIndexProjectionUpdate) instead of blocking on ForceIndexProjectionUpdate. Widening changes only which entries the index carries, never what ProjFS projects (GVFS projects the full HEAD tree regardless of cone), so the reply is correct as soon as the on-disk index is widened; the background parse thread reconciles GVFS-internal state. This keeps the bounded hook wait covering only the O(cone) index collapse at every scale. Raise the widen wait budget 250 -> 1000 ms (Task.Wait returns as soon as the reply arrives, so fast widens pay nothing), capture the previously discarded wait outcome (restoring the fuller result check from the hydration-status precedent), and emit an over-budget warning so a breach is visible in the mount log rather than silent. Scope the warning to the pre-command widen only; the post-command narrow's budget breach is benign, so narrow keeps its 100 ms budget. Add phase timing (cone build / write / collapse) to the mount log. Constants moved to GVFSConstants.ConeManagement so the hook and mount share one budget. All behind gvfs.auto-sparse-index (default false). Unit suite 1162/1162/0; functional test WidenHandlerCoversOutOfConePathsWithoutFullyExpandingTheIndex passes. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
gvfs clone --sparse-index now fails fast when the configured git cannot produce a sparse index, and constructs the sparse index directly during checkout instead of writing a full index and collapsing it afterward. Capability gate: - Add GitProcess.SupportsVfsSparseIndex / HasVfsSparseIndexCapability. They probe 'git version --build-options' for the exact line 'feature: vfs-sparse-index'. Stock git omits it; a git build with the VFS sparse-index changes emits it. The parser is separated from the spawn so it is unit-testable without a git binary. The same helper is shared so 'gvfs clone --sparse-index' and a future 'gvfs sparse-index --enable' do not diverge. - CloneVerb.Execute probes the enlistment's configured git before any network work and exits with a clear message when the capability is absent, instead of silently producing a full index. Direct sparse construction (reorder): - Write the minimal cone (/*, !/*/) and enable cone-mode sparse checkout before ForceCheckout, so a capable git materializes only in-cone paths and writes a sparse index directly. This removes the full-index write and the post-hoc collapse. - Replace TryConstructSparseIndex (checkout-then-collapse) with TryWriteInitialSparseCone (pre-checkout cone write) plus an observe-only LogClonedSparseIndex that records the index and warns, never fails or rewrites. The capability gate guarantees a capable git reaches the checkout, so the collapse step is dead code and is removed. GitProcess.CollapseSparseIndex is retained for reuse elsewhere. Tests: - Unit: 5 parser tests for HasVfsSparseIndexCapability (present, CRLF, absent, near-miss, null/empty). - Functional: the sparse-construction test asserts a sparse on-disk index before first mount and a complete projection; it Assert.Ignores when the configured git lacks the capability. A new fail-fast test asserts the clone is rejected on an incapable git. The control (plain clone) is unchanged. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The pre-command hook sent a cone-widen request on every command that named a path, then waited for the reply. Some of those widens cannot help, so the round-trip and the wait were pure cost. Decide locally, before any IPC, when the widen is pointless and skip it. Two exclusions, both provable from existing measurements: 1. Paths already in the cone. The widen exists to cover out-of-cone paths; a command naming only in-cone paths makes the mount rebuild an identical cone and reply "no change". The new GVFS.Common.Sparse.ConeCoverage helper lets the hook reach the same answer from the on-disk sparse-checkout file, using the same ConePathspecResolver and cone semantics the mount uses. Because the file already equals Serialize(ConeBuilder(existing)) and ConeBuilder is deterministic, "all named paths covered" locally is observably identical to a no-op reply, so the skip is provably safe. Any case the hook cannot decide (non-cone or hand-edited file, pathspec-from-file, -C/--git-dir/--work-tree, a linked worktree, a missing file, or any exception) defers to the mount. This is applied to the widen only: the cone can change between the separate pre- and post-command hook processes, so re-deriving a narrow-skip from the current file could leak a transient session. 2. stash. builtin/stash.c expands the index unconditionally when a pathspec is present, so widening cannot prevent the expansion, and it was measured net-negative (it un-hides the skip-worktree change and gives Git more work). The parser already identifies the verb, so stash is skipped on both the widen and the narrow. A per-command allowlist of expand-capable verbs was considered and rejected: the in-cone check is path-based and therefore command-agnostic, which tracks Git's actual conditional-expansion behaviour instead of a list that drifts as Git changes. A skipped widen was previously indistinguishable from a failed one. An opt-in one-line stderr diagnostic, emitted only when GIT_TRACE is 1, 2, or true, now separates a deliberate skip from a budget breach without adding IPC or tracer setup to the fire-and-forget hook. Measured: the local in-cone check is 0.3-0.7 ms at realistic modified-directory counts (<= 1000 dirs), about 10-20x cheaper than the ~5.9 ms warm no-op round-trip it replaces, and it never touches a busy mount. The cone is O(modified dirs), not O(repo), so the check stays sub-millisecond at large-repo scale while the round-trip it replaces can block up to the full wait budget on a mount busy servicing a prior widen's background reparse. All behind gvfs.auto-sparse-index (default false). Full unit suite 1186 passed, 0 failed (+24 ConeCoverage tests). The out-of-cone widen functional test still passes: it names out-of-cone paths, so neither exclusion fires and the full widen still runs. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
…-day2 Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The clone-time and cone-management branches each added a CoreSparseCheckoutName constant to GitConfigSetting with the same value. Git auto-merged both declarations without reporting a conflict, which produced CS0102 (duplicate definition) at build time. Keep one declaration and retain both explanatory comments: the sparse index gating group documents why the setting exists, and the clone-time note records that clone-time construction writes it while enable and disable leave it untouched. Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
ConeBuilder is size-unaware: a modified file always contributes a parent-only ancestor chain. Where a directory is worked wholesale that chain accumulates many patterns and is rewritten often, even though a single recursive include would describe the same set of index entries. Add a post-pass that replaces a parent-only sub-chain with one recursive include at the shallowest ancestor that qualifies. A directory qualifies on either gate: size + activity subtree_files(D) <= 250 and modified_under(D) >= 8 density modified_under(D) / subtree_files(D) >= 0.5 Both gates are entry neutral by construction. They fire only where a recursive include costs about the same index entries the parent-only chain already carries, so index size is unchanged. The payoff is pattern count and cone churn: up to 458x fewer patterns where a directory is modified wholesale, and up to 3.5x fewer cone rewrites. Measurement showed there is no case where recursing a whole top-level directory wins, because every top-level directory holds tens to hundreds of thousands of entries. The heuristic therefore only ever collapses at a deep, small directory, and never at an ancestor of a directory that is already recursive. The heuristic needs subtree file counts, which only the mount has, so ConeBuilder takes an IConeSubtreeFileCounter and stays pure and unit testable. It needs only a bounded answer, not an exact count, so the counter caps the walk: the size gate caps at the threshold, and the density gate caps at modified/density, which is the largest subtree that could still satisfy the ratio. Both are O(cap) per candidate regardless of real subtree size. ProjectionSubtreeFileCounter implements the counter over the in-memory projection through TryGetProjectedItemsFromMemory, so it does no I/O, fetches no objects, and never takes the projection write lock. A folder it cannot resolve returns false, and the heuristic treats false as "do not collapse", so an unresolvable path always yields the narrower cone. Gated behind gvfs.sparse-index-cone-granularity, default false. With no counter supplied the output is byte-for-byte the parent-only cone, which a test asserts directly. Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Implement the cone granularity heuristic (W12), opt-in and default off
Integrate all sparse-index workstreams into feature/sparse-index
Two independent fixes found while preparing an end-to-end validation run.
Native projects pinned PlatformToolset to v143 in an unconditional
PropertyGroup, so the five C++ projects cannot build on a machine that
has a newer Visual Studio. Visual Studio 2026 ships v145 and no longer
provides v143, so Build.bat fails at the first native project with
MSB8020. An unconditional PropertyGroup also outranks an environment
variable, so there was no way to override it without editing the file;
Build.bat passes no /p:PlatformToolset either.
Guard the assignment with a condition. The default stays v143, so every
existing machine and the build pipeline are unaffected, but a newer
toolset can now be selected without editing project files:
$env:PlatformToolset = 'v145'
src\scripts\Build.bat Debug <version>
The benchmark harness supported sixteen operations but not revert, even
though revert is a routine developer operation and appears in about 5%
of real pull requests in a large repository. Revert is also interesting
for the sparse index specifically: it names the paths of a commit that
is already in history, so it can reach outside the cone in ways a
cherry-pick of a foreign commit does not.
Add revert alongside cherry-pick. Its cleanup must be checked before the
cherry-pick cleanup, because both operations use .git/sequencer and an
in-progress revert would otherwise be aborted with the wrong verb.
Assisted-by: Claude Opus 5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
RunFunctionalTests-Dev.ps1 put a hard-coded "C:\Program Files\Git\cmd" on PATH. That path is only the default, and it is not stable: the Git for Windows installer relocates an existing installation when it is pointed at a different directory rather than installing alongside it. After such a move the hard-coded directory no longer exists, PATH gains a dead entry, and the script fails its own git prerequisite check even though git is installed and working. Resolve the installation the same way the product does, from the GitForWindows registry key that WindowsGitInstallation already reads, and fall back to the previous default when the key is absent. When neither resolves, leave PATH alone so the existing prerequisite check reports the problem instead of masking it with a dead entry. Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Settings.cs hard-coded C:\Program Files\Git for PathToGit and PathToBash, the same assumption the dev test script made. The Git for Windows installer relocates an existing installation rather than installing alongside it, so after such a move every functional test fails on a missing git rather than on the behavior under test. Resolve the install root from the GitForWindows registry key, exactly as WindowsGitInstallation does, and fall back to the previous default when the key is absent or unreadable. Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Allow the native PlatformToolset to be overridden, and benchmark revert
GitProcessTests hardcoded C:\Program Files\Git for one case. On a machine where git is installed anywhere else the test called Assert.Ignore and reported success, so it had stopped checking anything. Resolve the path the same way the product does instead. Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
A sparse clone writes a 66-entry index and never descends into the collapsed trees, so the first mount has to read roughly 500,000 tree objects cold to build its projection. Measured on a 2.4 million entry repository that cost 30.5s against 21.1s for a full-index clone. The work is not new, only relocated: a full-index clone reads exactly those trees inside checkout, because writing 2,245,934 index entries requires them. Write the same index during clone and the first mount can parse it instead of walking the trees again. Add GitProcess.WriteProjectionSeedIndex, which runs read-tree against GIT_INDEX_FILE so git reads and writes the seed rather than the repository index. --index-output is not sufficient: it redirects only the write, and git still takes .git/index.lock, which fails the concurrent checkout with "Unable to create index.lock: File exists". read-tree writes no working-tree files, so there is no working-tree contention either, and the seed runs alongside the checkout on its own GitProcess instance. A shared instance cannot be used: GitProcess keeps the running child in an instance field, so concurrent invocations overwrite each other and one reads the other's streams. The seed carries no skip-worktree bits, which is expected rather than a defect. Under a virtual filesystem those bits are not stored state: apply_virtualfilesystem() recomputes them on every index read by setting CE_SKIP_WORKTREE on every entry and clearing it only for the paths the virtual-filesystem hook reports as present. The parser performs that same computation when it consumes the seed, using the modified-paths database as the set of present paths, so the projection it produces matches one built from an index git wrote. The comparison runs on the raw path buffer rather than decoding a string per entry, which would allocate once for every entry in a multi-million entry index. The seed is consumed at most once and deleted whether or not the parse succeeds, so a stale or partial seed can never affect a later mount, and every failure falls back to building the projection from the trees. Measured on a 2.4 million entry repository, two samples per arm: clone + first mount full index 42.8s sparse 43.5s (+0.7s) before this change the same comparison was +3.4s, and first mount alone went from +9.4s to +0.33s. The projection built from a seed matches git ls-tree HEAD exactly at every level checked. Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Seed the first projection from an index written during clone
The seed runs read-tree with no git-level timeout, and clone waited on it with an unconditional GetResult(). A read-tree that never finished would therefore hang clone itself. An unbounded wait is defensible for work clone cannot proceed without, but the seed is only an optimization, so this turned a speedup into a new way for clone to fail. Wait at most five minutes, then abandon the seed and let the first mount build its projection from the trees, which is the behavior without a seed. Measured seed cost on a 2.4 million entry repository is about ten seconds, so the bound only engages when something is wrong. Abandoning is safe because git writes the index to a lock file and renames it into place: the mount finds either a complete seed or none. It never finds a truncated one, which matters because the index parser trusts the entry count in the header and reads recycled page bytes rather than reporting an error when the file is short. A seed that arrives after clone stopped waiting is still complete and still describes the same HEAD, so it is correct whether or not the mount consumes it. Also delete the lock file when discarding a seed. An abandoned or failed seed leaves it behind, and while nothing reads it, it would block a later seed write. Assisted-by: Claude Opus 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Bound the clone-time wait for the projection seed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Prototype review] Automatic sparse index for VFS for Git
Draft — for code review of the hackathon prototype. Not for merge.
This is the consolidated VFS for Git side of the prototype. The git-side changes
are in a companion draft PR in the
microsoft-gitfork.Base is
sparse-index-review-base, a branch pinned at the pre-work commit(
d25926da, theupstream/vnextmerge). Diffing againstmasterinstead wouldadd ~100 unrelated
vnextcommits to the review; against this base the diff isonly our work: 38 commits, 76 files.
What it does
VFS for Git virtualizes the working directory, but
.git/indexstill carriesevery file — 193 MB and 2,245,934 entries on os.2020. Every index-proportional
git operation pays for all of them.
This keeps the ProjFS projection full (no visible change to the enlistment)
while letting
.git/indexgo sparse, using git's cone-mode sparse index.The mount manages the cone automatically from the modified-path set; the
pre-command hook widens it when a command names an out-of-cone path and the
post-command hook narrows it back.
Measured on two os.2020 enlistments at the same commit, differing only in
--sparse-index:git statusgit rebaseClone and first mount are now at parity. The first-mount cost was never new:
writing a 2.2M-entry index during
checkoutrequires reading every tree, so thebaseline does that work inside clone and the sparse arm merely deferred it. The
final commit writes a full index to a temp file concurrently with the
checkout, and the first mount parses it instead of walking ~500,000 trees cold.
Clone + first mount: +3.4 s (5.3%) -> -0.12 s (-0.3%). First mount alone:
+9.4 s -> +0.30 s. Timed from GVFS's own
GVFSClone (Stop)/ExecuteMount (Stop)durations, not wall clock.
The seed is not free, and should be argued as a trade: a sparse clone used to be
14% faster than a full clone, and is now only 1.8% faster. The seed spent that
saving to buy back the mount penalty. Clone is long and runs once; mount gates
the first usable moment.
Everything is behind
gvfs.auto-sparse-index, default false.Suggested reading order
The commit history is workstream-shaped, not review-shaped (re-cutting it into
reviewable PRs is itself a productization task). To read it by area:
Cone model —
GVFS.Common/Sparse/ConeBuilder,ConePatternSet,ConeFileWriter,ConePathspecResolver,ConeCoverage,ConeDriftDetector,TransientConeState. Pure, no I/O,heavily unit-tested. Start here; everything else consumes these.
Projection —
GVFS.Virtualization/Projection/GitIndexProjection.SparseDirectoryExpander.csis the core change: asparse-directory index entry is expanded from its tree so the projection
stays complete.
GitIndexParsergains sparse-entry detection and fail-fast.Mount-side cone management —
GVFS.Mount/AutoSparseIndexConeManager.csHandles widen/narrow requests from the hooks. Note the concurrency comment at
the top: ordering is coneLock -> sparse-checkout write -> git's index.lock,
and the projection reparse is deliberately off the reply path.
Hooks —
GVFS.Hooks/Program.ConeManagement.cs,GVFS.Common/GitPathspecParser.csTable-driven git command-line parsing to decide which paths a command names,
plus the exclusions that skip the IPC entirely when widening cannot help.
CLI —
GVFS/CommandLine/SparseIndexVerb.cs,CloneVerb.csgvfs sparse-index --enable/--disable/--status, andgvfs clone --sparse-indexwhich builds the cone before the initial checkout so theindex is never written full.
Guard —
GVFS.Common/SparseCheckoutCommandGuard.csBlocks the
git sparse-checkoutsubcommands that would fight the mount forownership of the cone file.
Things worth a reviewer's attention
AutoSparseIndexConeManageris the most concurrency-sensitive code here.It deliberately does not take the GVFS lock (the pre-command hook already
holds it, so acquiring it would deadlock the very command waiting on the
reply).
SparseDirectoryExpandermust never produce a partial projection. Itfails the build rather than serving an incomplete tree; a partial projection
would be silent data loss.
decisions/0014): ordinary and memory-mappedindex writes do not re-enter
ModifiedPathsDatabase, so dropping amodified path without converting the file back to a placeholder loses data.
ConeGranularityOptions) is opt-in anddefault off. It is entry-neutral by construction — it optimises pattern count
and cone churn, not index size.
Known gaps
stash push -- <path>4.9x slowerbuiltin/stash.cexpands unconditionally on any pathspec; cone widening cannot fix it. Needs an upstream git change.Verification
gvfs clone --sparse-indexagainst a realmount.
git ls-tree HEADat every level checked.design choice was made, including the ones that were rejected.
Review guidance
This PR is one consolidated diff for reading, not a proposal to merge as-is.
A production breakdown into 12 reviewable PRs — with ordering, dependencies,
branch targeting, and a review-load estimate — is in
presentation/production-pr-plan.md. The two PRs carrying most of the risk arethe git-side non-expanding lookups and the projection tree expansion.
Four gaps must close before any of this is proposed for production: prune must
dehydrate (data-loss risk, ADR 0014), cache replay, the two measured pathspec
regressions, and the one uncoverable rebase expansion. All four are documented
in the plan.