Skip to content

miner(discover): supplementWithDiscoveryIndex's dedup set is never updated during its own filter pass #10334

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

packages/loopover-miner/lib/discover-cli.ts's supplementWithDiscoveryIndex (around line 190)
merges hosted discovery-index candidates into the local fan-out results, dropping any candidate
that duplicates an issue already present in fanOut.issues:

const seen = new Set(fanOut.issues.map((issue) => dedupeKey(issue.repoFullName, issue.issueNumber)));
const supplemented = aiAllowed
  .filter((candidate) => !seen.has(dedupeKey(candidate.repoFullName, candidate.issueNumber)))
  .map((candidate) => ({ ...candidate, assignees: [...(candidate.assignees ?? [])], labels: [...candidate.labels] }) as RawCandidateIssue);
if (supplemented.length === 0) return fanOut;
return { ...fanOut, issues: [...fanOut.issues, ...supplemented] };

seen is built exactly once, from fanOut.issues, before the .filter() call, and is never added
to during that filter. The filter therefore only guards against a candidate duplicating something
already in the LOCAL fan-out — it does nothing to catch two candidates that duplicate each
other
within the SAME discovery-index response. If the hosted discovery index ever returns two
entries for the same (repoFullName, issueNumber) pair (a real possibility for a hosted, shared,
eventually-consistent index that this file's own doc comments already acknowledge as a
best-effort/fail-open source), both pass the filter, both get appended to fanOut.issues, and the
run proceeds with a duplicated candidate.

This has concrete downstream effects: recordDiscoveryTelemetry's droppedAiBanned count and the
overall fanOut.issues.length become inflated, and every later stage that keys off issue identity
(ranking, signal-tracking writes via contribution-profile-cache, portfolio-queue enqueue) sees
the same (repo, issue) pair twice in one run.

Requirements

  • In supplementWithDiscoveryIndex, update the seen set as candidates are accepted during the
    filter/map pass, so a duplicate within aiAllowed itself is caught the same way a duplicate
    against fanOut.issues already is. Do not change the function's existing "local results always
    win" precedence for candidates that duplicate fanOut.issues — this issue is only about
    duplicates within the hosted response itself.
  • Preserve the existing behavior for every other branch in this function (the
    isDiscoveryPlaneEnabled early return, the aiPolicyAllowed !== false filter and its
    droppedAiBanned telemetry, the supplemented.length === 0 early return) exactly as-is.

Deliverables

  • supplementWithDiscoveryIndex no longer appends two entries for the same
    (repoFullName, issueNumber) pair when the hosted discovery-index response itself contains a
    duplicate.
  • A new regression test in the existing discover-cli test file asserting that a mocked
    queryDiscoveryIndex response containing two candidates with the same repoFullName +
    issueNumber results in exactly one entry being appended to fanOut.issues, and that a
    duplicate of an existing fanOut.issues entry is still dropped in favor of the local result
    (the pre-existing behavior, to guard against a regression while fixing the new case).

Test Coverage Requirements

This repo's Codecov patch gate is 99%+ (branch-counted) on packages/loopover-miner/lib/**. Add
the new regression test under test/unit/** (this package's tests live in the shared root test/
directory; find and extend the existing discover-cli test file rather than creating a new one). The
new branch you add to update seen during the filter/map pass must be exercised by the new test.

Expected Outcome

A hosted discovery-index response containing an internal duplicate produces exactly one candidate
in the fan-out output, matching the same dedup guarantee supplementWithDiscoveryIndex already
provides against duplicates of the local fan-out.

Links & Resources

  • packages/loopover-miner/lib/discover-cli.tssupplementWithDiscoveryIndex (~line 190),
    dedupeKey (the existing helper to reuse).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions