feat(scorecard): add new GitHub metrics for issues, opened/closed prs and GitHub actions - #3472
feat(scorecard): add new GitHub metrics for issues, opened/closed prs and GitHub actions#3472fullsend-ai-coder[bot] wants to merge 2 commits into
Conversation
|
Important This PR includes changes that affect public-facing API. Please ensure you are adding/updating documentation for new features or behavior. Changed Packages
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3472 +/- ##
==========================================
+ Coverage 61.38% 61.41% +0.02%
==========================================
Files 2520 2529 +9
Lines 100936 101338 +402
Branches 28304 28326 +22
==========================================
+ Hits 61958 62234 +276
- Misses 38427 38553 +126
Partials 551 551
*This pull request uses carry forward flags. Click here to find out more. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
🤖 Finished Review · ✅ Success · Started 11:49 AM UTC · Completed 12:02 PM UTC |
ReviewFindingsMedium
Low
Previous runReviewFindingsMedium
Low
Info
Previous run (2)ReviewFindingsMedium
Low
Info
Previous run (3)ReviewFindingsHigh
Medium
Low
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
christoph-jerolimov
left a comment
christoph-jerolimov
left a comment
There was a problem hiding this comment.
Address the following issues:
GitHubActionsProviderneeds to be splitted because it includes metrics that counts actions and then metrics that creates a ratio. But the MetricProvider API only supports oneThresholdConfigfor all metrics, at the moment. So splitting this into two providers would allow to define the rightThresholdConfigfor these metrics.GithubCIPassRateProvidershould be renamed toGitHubPRPassRateProviderand the metrics should be named something like "GitHub PR CI first time pass rate (7d)". The description should also include that this is about FTPR (first time pass rate).- Add unit tests for GitHubClosedIssuesProvider.ts, GitHubClosedPRsProvider.ts, GitHubOpenedIssuesProvider.ts and GitHubOpenedPRsProvider.ts
|
/fs-fix |
|
🤖 Finished Fix · ✅ Success · Started 12:51 PM UTC · Completed 1:10 PM UTC |
- Split GithubActionsProvider into GithubActionsCountProvider (with COUNT_THRESHOLDS) and GithubActionsRatioProvider (with RATIO_THRESHOLDS) so each provider gets the correct ThresholdConfig - Rename GithubCIPassRateProvider to GithubPRPassRateProvider with updated metric IDs (github.pr_ci_first_time_pass_rate_*) and titles mentioning FTPR (first time pass rate) - Add individual unit test files for GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider - Apply encodeURIComponent to owner/repo in REST URL construction Addresses review feedback on #3472
🔧 Fix agent — iteration 1 (human-triggered)Addressed all 3 human review requests: (1) split GithubActionsProvider into count and ratio providers with correct thresholds, (2) renamed GithubCIPassRateProvider to GithubPRPassRateProvider with FTPR naming, (3) added individual unit tests for 4 search-count providers. Also fixed SSRF vulnerability with encodeURIComponent. Disagreed with 4 bot-only findings as out of scope or unnecessary. Fixed (4):
Disagreed (4):
Tests: passed Updated by fullsend fix agent |
|
🤖 Finished Review · ✅ Success · Started 1:13 PM UTC · Completed 1:26 PM UTC |
fullsend-ai-review
Bot
added
the
requires-manual-review
|
/fs-fix inspect and fix e2e playwright tests |
|
🤖 Fix · ❌ Terminated · Started 6:50 AM UTC · Ended 8:25 AM UTC |
- Split GithubActionsProvider into GithubActionsCountProvider (with COUNT_THRESHOLDS) and GithubActionsRatioProvider (with RATIO_THRESHOLDS) so each provider gets the correct ThresholdConfig - Rename GithubCIPassRateProvider to GithubPRPassRateProvider with updated metric IDs (github.pr_ci_first_time_pass_rate_*) and titles mentioning FTPR (first time pass rate) - Add individual unit test files for GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider - Apply encodeURIComponent to owner/repo in REST URL construction Addresses review feedback on #3472
006906c to
5f9d7ac
Compare
|
🤖 Finished Fix · ❌ Failure · Started 6:50 AM UTC · Completed 8:25 AM UTC |
|
🤖 Finished Review · ✅ Success · Started 8:27 AM UTC · Completed 8:40 AM UTC |
| } | ||
| } | ||
| } | ||
| `; |
There was a problem hiding this comment.
[medium] edge-case
The getPullRequestsWithReviews and getPullRequestsWithCommitStatuses GraphQL queries use search(first: 100) which limits results to 100 PRs. For active repositories, this silently truncates the dataset, producing inaccurate metrics. No pagination is implemented unlike getWorkflowRuns.
Suggested fix: Implement GraphQL cursor-based pagination or document the 100-item limit and add a log warning when the result count equals 100.
| }>(query, { | ||
| q: fullQuery, | ||
| }); | ||
|
|
There was a problem hiding this comment.
[medium] edge-case
getWorkflowRuns pages through all workflow runs using a while loop with no upper bound on page count or total results. High-activity repos could cause unbounded HTTP requests, memory exhaustion, and rate limit depletion.
Suggested fix: Add a maximum page count (e.g., 10 pages = 1000 runs) to cap the loop.
Add explicit type annotations to `response` variables in getPullRequestsWithReviews and getPullRequestsWithCommitStatuses. The `cursor` variable created a circular type dependency (response depends on cursor as argument, cursor is reassigned from response), causing TS7022 "implicitly has type 'any'". Moving the type from a generic parameter to a variable annotation breaks the cycle. Addresses review feedback on #3472
1. Remove all const METRIC_IDS / METRIC_ID and inline metric IDs directly into getMetrics() and calculateMetrics() calls across all new providers. 2. Split GithubActionsCountProvider thresholds: started and successful metrics use ACTIVITY_THRESHOLDS (Success >=0, Warning ==0), while failed metric keeps FAILED_COUNT_THRESHOLDS (<10/10-50/>50). 3. Switch GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider to new INFORMATIONAL_NUMBER_THRESHOLD (info status for any value >=0) added to scorecard-common threshold.ts. 4. Change GithubPRLifecycleProvider duration from hours to days, update DURATION_THRESHOLDS to Success <7, Warning 7-14, Error >14, and prefix all metric titles with "GitHub PR". Addresses review feedback on #3472
8e2436f to
ce5ff3a
Compare
|
/fs-fix
|
|
🤖 Finished Fix · ✅ Success · Started 9:38 PM UTC · Completed 9:53 PM UTC Commit: |
Update getProviderId() in all 9 new metric providers to use the 'github.' prefix consistent with the existing GithubOpenPRsProvider pattern and the datasource ID convention. Add changeset for scorecard-backend-module-github (verbose, minor) describing all 13 new metrics, and changeset for scorecard-common (minor) for the new INFORMATIONAL_NUMBER_THRESHOLD export. Addresses human review feedback on #3472
🔧 Fix agent — iteration 7 (human-triggered)Addressed both human review findings: (1) prefixed all 9 new provider getProviderId() return values with 'github.' to match the data source ID convention (e.g., 'ActionsCountProvider' -> 'github.actionsCount'), updated 4 related test assertions; (2) added verbose changeset for scorecard-backend-module-github and short changeset for scorecard-common. All 47 tests pass. Fixed (12):
Tests: passed Updated by fullsend fix agent |
|
/fs-fix
|
|
🤖 Finished Fix · ✅ Success · Started 5:19 AM UTC · Completed 5:35 AM UTC Commit: |
- Split GithubActionsProvider into GithubActionsCountProvider (with COUNT_THRESHOLDS) and GithubActionsRatioProvider (with RATIO_THRESHOLDS) so each provider gets the correct ThresholdConfig - Rename GithubCIPassRateProvider to GithubPRPassRateProvider with updated metric IDs (github.pr_ci_first_time_pass_rate_*) and titles mentioning FTPR (first time pass rate) - Add individual unit test files for GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider - Apply encodeURIComponent to owner/repo in REST URL construction Addresses review feedback on #3472
Rebase on main and migrate all new metric providers to the updated MetricProvider interface from #3560. Remove deprecated single-metric methods (getMetricType, getMetric, getMetricThresholds, calculateMetric, getMetricIds) and use only getMetrics() with thresholds on each Metric object and calculateMetrics() returning a Map. All 10 test suites (37 tests) pass. Addresses review feedback on #3472
🔧 Fix agent — iteration 8 (human-triggered)Rebased branch onto latest main (no conflicts). Added color 'info.main' to INFORMATIONAL_NUMBER_THRESHOLD info rule. Regenerated API report. All 47 tests pass. Fixed (2):
Tests: passed Updated by fullsend fix agent |
Align all new metric provider IDs with the lowerCamelCase naming convention established in the codebase. The providers were using snake_case IDs which is inconsistent with the existing pattern (e.g. github.openPRs). Renames: - github.open_issues -> github.openIssues - github.opened_issues_7d -> github.openedIssues7d - github.closed_issues_7d -> github.closedIssues7d - github.opened_prs_7d -> github.openedPRs7d - github.closed_prs_7d -> github.closedPRs7d - github.time_to_review -> github.timeToReview - github.time_to_approve -> github.timeToApprove - github.time_to_merge -> github.timeToMerge - github.actions_started_7d -> github.actionsStarted7d - github.actions_successful_7d -> github.actionsSuccessful7d - github.actions_failed_7d -> github.actionsFailed7d - github.actions_success_ratio_7d -> github.actionsSuccessRatio7d - github.actions_success_ratio_24h -> github.actionsSuccessRatio24h - github.pr_ci_first_time_pass_rate_7d -> github.prCiFirstTimePassRate7d - github.pr_ci_first_time_pass_rate_24h -> github.prCiFirstTimePassRate24h Addresses review feedback on #3472
- Fix RATIO_THRESHOLDS gap in GithubActionsRatioProvider and GithubPRPassRateProvider: change warning from '50-79' to '>=50' to prevent backend crash from uncovered threshold range - Add cursor-based pagination to getPullRequestsWithReviews and getPullRequestsWithCommitStatuses GraphQL queries (max 10 pages) - Cap getWorkflowRuns REST pagination at 10 pages to prevent unbounded API calls on active repos - Update getProviderId() in all new providers to return a unique identifier matching the filename without the Github prefix - Decouple metric IDs from provider IDs in single-metric providers by introducing METRIC_ID constants Addresses review feedback on #3472
Add explicit type annotations to `response` variables in getPullRequestsWithReviews and getPullRequestsWithCommitStatuses. The `cursor` variable created a circular type dependency (response depends on cursor as argument, cursor is reassigned from response), causing TS7022 "implicitly has type 'any'". Moving the type from a generic parameter to a variable annotation breaks the cycle. Addresses review feedback on #3472
1. Remove all const METRIC_IDS / METRIC_ID and inline metric IDs directly into getMetrics() and calculateMetrics() calls across all new providers. 2. Split GithubActionsCountProvider thresholds: started and successful metrics use ACTIVITY_THRESHOLDS (Success >=0, Warning ==0), while failed metric keeps FAILED_COUNT_THRESHOLDS (<10/10-50/>50). 3. Switch GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider to new INFORMATIONAL_NUMBER_THRESHOLD (info status for any value >=0) added to scorecard-common threshold.ts. 4. Change GithubPRLifecycleProvider duration from hours to days, update DURATION_THRESHOLDS to Success <7, Warning 7-14, Error >14, and prefix all metric titles with "GitHub PR". Addresses review feedback on #3472
Update getProviderId() in all 9 new metric providers to use the 'github.' prefix consistent with the existing GithubOpenPRsProvider pattern and the datasource ID convention. Add changeset for scorecard-backend-module-github (verbose, minor) describing all 13 new metrics, and changeset for scorecard-common (minor) for the new INFORMATIONAL_NUMBER_THRESHOLD export. Addresses human review feedback on #3472
Add color 'info.main' to the info threshold rule so it satisfies the requirement that non-standard keys (anything other than success, warning, error) must specify an explicit color. Regenerated API report after rebase. Addresses review feedback on #3472
6037490 to
8785123
Compare
|
/fs-fix fix api report. In the last commit, just the new color should be added but the order of some other attributes also changed. |
|
🤖 Finished Fix · ✅ Success · Started 5:45 AM UTC · Completed 5:51 AM UTC Commit: |
Revert spurious attribute reordering in scorecard/report.api.md that was introduced when the API report was regenerated. The regeneration was only supposed to add the new color to INFORMATIONAL_NUMBER_THRESHOLD in scorecard-common, but also reordered aggregationId/metricId, kind/type, and layouts ExtensionInput entries in the scorecard plugin report. Addresses review feedback on #3472
🔧 Fix agent — iteration 9 (human-triggered)Reverted spurious attribute reordering in scorecard/report.api.md introduced during API report regeneration. Only the new INFORMATIONAL_NUMBER_THRESHOLD in scorecard-common/report.api.md remains as intended. Fixed (1):
Tests: passed Updated by fullsend fix agent |
|
/fs-fix rebase and resolve conflicts |
|
🤖 Finished Fix · ✅ Success · Started 6:36 PM UTC · Completed 7:01 PM UTC Commit: |
Extend the scorecard GitHub module with new metrics across four domains: issue/PR counts, PR lifecycle timing, GitHub Actions, and CI pass rate.
Squash-rebased all 10 PR commits onto latest main, resolving conflicts: - types.ts: merged main's deployment types with PR's metric types - GithubClient.ts: kept main's constructor (logger, cached credentials), deployment methods, and getWorkflowRuns; added PR's new methods (getOpenIssuesCount, getSearchCount, getPullRequestsWithReviews, getRecentWorkflowRuns, getPullRequestsWithCommitStatuses, getRestConfig) - module.ts: kept main's collectors and added all new metric providers - All new providers: adapted fromConfig to accept logger parameter, matching the existing GithubOpenPRsProvider pattern from main - Renamed PR's getWorkflowRuns to getRecentWorkflowRuns to avoid name conflict with main's existing getWorkflowRuns method - Applied prettier formatting fixes Addresses human review feedback on #3472
🔧 Fix agent — iteration 10 (human-triggered)Successfully rebased PR onto latest main, resolving merge conflicts in GithubClient.ts, types.ts, and module.ts. Adapted all new providers to match main's logger pattern. All 70 tests pass, TypeScript compiles, prettier clean. Fixed (1):
Tests: passed
Decision points
Updated by fullsend fix agent |
094461e to
51fa86f
Compare
|
4 participants
Extend the scorecard GitHub module with 13 new metrics across four domains:
Issue/PR counts (5 metrics):
PR lifecycle timing (3 metrics, batch provider):
All computed from PRs updated in the last 7 days.
GitHub Actions (5 metrics, batch provider):
Non-terminal runs (pending/running/cancelled) are excluded from success/failure counts and ratio calculations.
CI pass rate (2 metrics, batch provider):
Checks CI status on the last commit of the first push to each PR. PRs without CI checks are excluded.
New GithubClient methods use GraphQL for issue/PR queries and REST API (fetch) for workflow runs. All providers follow the existing MetricProvider pattern and are registered in the module init. Batch providers use getMetrics/calculateMetrics for efficient multi-metric computation.
Closes #3468
Post-script verification
agent/3468-extend-github-metrics)7ccaff17753df64c7ab288cdcba34cee5a657254..HEAD)