Skip to content

npm_crawler classifies node_modules store children three times, and the copies drifted on store entries and legacy/global stores #1899

Open

Description

[agent] Filed by the scheduled architecture audit (audit-core). Register: register comment.

Kind: refactor. Register ID: A41.

Problem

crawlers/npm_crawler.rs decides which child of a node_modules is a package-manager store (pnpm-shaped .pnpm/.store, vlt .vlt, npm-linked .store, .modules.yaml, legacy pnpm store names) in three hand-written copies:

  • the find_by_purls resolver, nested_node_modules_of: L3166-L3225
  • the scan, gather_node_modules: L3576-L3624
  • the peer-variant finder, find_store_peer_variant_copies_reusing: L4478-L4520

The copies have drifted:

  • Store entries. Every arm of the scan is gated on !store_entry, so a store entry's own node_modules/.pnpm (or .vlt, .store, .modules.yaml, a legacy name) is never treated as a store. The resolver has no store_entry parameter and walks such a child as a store. apply can therefore resolve copies that scan never reports. This is unexecuted and an edge case.
  • Missing kinds. The peer finder handles neither legacy pnpm stores nor the pnpm global virtual store.
  • Empty .modules.yaml arm. The resolver's arm pushes an empty entry list (L3207), and visit_resolver_dir then has to guard against it (L3037).

The same file also has:

  • four shapes for one store entry: StoreEntryDir, StoreEntry, (String, PathBuf) and (OsString, PathBuf). They are converted at L1751-L1788, L3673, L3692, L4087 and L4131, and list_npm_store_entries decodes each name, discards the result, and decodes it again (L4243, then L4619);
  • about 7 copies of the "package-dir child" filter (not hidden, not node_modules, a real dir): L3082, L3226-L3241, L3508, L3772, L4153, L4200, L600. They drift on whether is_dir follows symlinks (L4077 vs L4160/L4226) and on whether a scoped child named node_modules is excluded (L3238);
  • 5 copies of the @scope/name expansion: L3088-L3102, L3234-L3245, L3645/L3762, L4213-L4222, L4719.

Proposed change

  • Add one StoreKind enum with a classify(name, file_type, in_store_entry) -> Option<StoreKind> and use it in all three walks. The resolver passes in_store_entry the way the scan does.
  • Keep one store-entry type (StoreEntryDir), drop the tuple shapes, and decode each name once.
  • Add one package_children(dir, listing) iterator that yields unscoped and @scope/name package dirs, and use it for the filter and expansion copies.

The peer-variant section (L4391-L4791) can then move to crawlers/npm_crawler/store.rs.

Size: 1 file (plus a new submodule), about −300/+150 lines.

Acceptance criteria

  • crawl_all, find_by_purls and find_store_peer_variant_copies return the same results on every existing test in npm_crawler.rs and tests/crawler_npm_e2e.rs.
  • A new test pins the store-entry rule: a store entry holding a nested node_modules/.pnpm is neither reported by crawl_all nor resolved by find_by_purls. Alternatively, if both are meant to walk it, the test asserts that both do.

Activity

  1. added
    arch-auditFiled by a scheduled architecture audit routine (see the architecture review discussion)
    refactorStructural change: duplicated code or logic, missing abstraction, layering, dead code
    on Nov 1, 2026
  2. mikolalysenko commented on Nov 1, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Triage: priority:p1 (npm crawler). No duplicate or open PR found. This is a large single-file refactor (about -300/+150), so it's a candidate to split by sub-cause (StoreKind classify, store-entry type, package_children iterator).


    Generated by Claude Code

  3. mikolalysenko commented on Nov 2, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Claiming this issue (shared root cause: npm_crawler classifies node_modules store children in hand-written copies that drifted). First slice: one StoreKind classifier used by the resolver and the scan. Branch: agent/fix-npm-store-kind-classify. Claim-ID: 2026-11-02T12:36:10Z-a7faf4


    Generated by Claude Code

  4. mikolalysenko commented on Nov 2, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Draft PR for the first slice (one StoreKind classifier shared by scan and resolver, plus the store-entry rule test): #1986. The peer-finder, store-entry type and package_children slices remain open here.


    Generated by Claude Code

  5. mikolalysenko commented on Nov 2, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] First slice merged in #1986. Scan, the apply/vex resolver and the apply peer fan-out now share one StoreKind::classify and its store-entry rule: a store nested in a store entry is never walked. Regression tests cover every store name and every store-entry shape.

    Still open here (I'm releasing the claim so a later run can take them):

    • the peer fan-out still doesn't cover pnpm <=3 .registry.* stores or the pnpm global virtual store;
    • one store-entry type (StoreEntryDir) instead of four shapes, decoding each name once;
    • one package_children iterator for the ~7 package-dir filters and the 5 @scope/name expansions.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:triagedarch-auditFiled by a scheduled architecture audit routine (see the architecture review discussion)priority:p1refactorStructural change: duplicated code or logic, missing abstraction, layering, dead code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions