Add GitHub Actions test history - #10778
Conversation
Persist bounded test history through workflow-managed artifacts and use prior run outcomes to annotate failures and flakiness. Fixes #10770 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 777e8acd-fc17-4ed7-a588-c973cc686388
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| private readonly IClock _clock; | ||
| private readonly ILogger _logger; | ||
| private readonly GitHubActionsHistoryScope _scope; | ||
| private IReadOnlyDictionary<string, GitHubActionsHistoryStats> _statsByTest = EmptyStats; |
There was a problem hiding this comment.
Pull request overview
Adds bounded, workflow-managed test history to the GitHub Actions reporting extension.
Changes:
- Adds history CLI options, persistence, scoped statistics, and failure annotations.
- Preserves history through retry and multi-module processing with unit coverage.
- Updates help output, localization resources, package guidance, glossary, and changelog.
Show a summary per file
| File | Description |
|---|---|
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsSummaryReporterTests.cs |
Tests history post-processing and retries. |
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsHistoryTests.cs |
Tests storage, aggregation, and metadata. |
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsCommandLineProviderTests.cs |
Tests history option validation. |
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsAnnotationReporterTests.cs |
Tests historical annotation context. |
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs |
Updates help/info expectations. |
src/Platform/SharedExtensionHelpers/CiRunSummaryModels.cs |
Adds history fragment models and settings. |
src/Platform/SharedExtensionHelpers/CiRunSummaryAggregateReader.cs |
Validates history fragment data. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.zh-Hant.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.zh-Hans.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.tr.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ru.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.pt-BR.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.pl.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ko.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ja.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.it.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.fr.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.es.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.de.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.cs.xlf |
Adds localized resource units. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/GitHubActionsResources.resx |
Defines history messages and descriptions. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PACKAGE.md |
Documents artifact-based persistence. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/InternalAPI/InternalAPI.Unshipped.txt |
Updates tracked internal API signatures. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs |
Collects and writes history results. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryArtifactPostProcessor.cs |
Merges retry and module history. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsHistory.cs |
Implements history storage and aggregation. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsExtensions.cs |
Registers the history service. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsCommandLineProvider.cs |
Exposes and validates history options. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsCommandLineOptions.cs |
Defines option names and defaults. |
src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsAnnotationReporter.cs |
Appends historical failure context. |
docs/glossary.md |
Documents the history feature. |
docs/Changelog-Platform.md |
Records the feature addition. |
Review details
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 32/32 changed files
- Comments generated: 5
- Review effort level: Balanced
| run: dotnet test --report-gh --report-gh-history .test-history/history.json | ||
|
|
||
| - name: Publish test history | ||
| if: always() && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) |
| const Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsCommandLineOptions.GitHubActionsHistory = "report-gh-history" -> string! | ||
| const Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsCommandLineOptions.GitHubActionsHistoryWindow = "report-gh-history-window" -> string! | ||
| const Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsCommandLineOptions.GitHubActionsStepSummarySections = "report-gh-step-summary-sections" -> string! | ||
| const Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsCommandLineOptions.HistoryWindowDefaultDays = 30 -> int |
|
|
||
| The extension intentionally reads and writes only a local snapshot. The workflow owns GitHub authentication and artifact transfer. This keeps `GITHUB_TOKEN` out of the test process and allows pull request runs to consume history without granting them write access. Snapshots retain at most 1,000 samples per test and the latest 10,000 samples overall, in addition to the configured age window. | ||
|
|
||
| The following steps restore the newest default-branch snapshot, run tests, and publish the updated snapshot only from the default branch: |
| GitHubActionsHistorySample[] samples = | ||
| [ | ||
| .. configuredGroup.SelectMany(module => module.HistoryTests.Select(test => new GitHubActionsHistorySample |
| GitHubActionsHistorySample otherAssembly = CreateSample("Tests.Flaky", GitHubActionsHistoryOutcome.Failed, Now); | ||
| otherAssembly.AssemblyName = "OtherTests"; |
🧪 Expert test review — PR #10778No new or modified test methods were identified in the changed regions of this PR. Nothing to review. Re-run with
|
🧵 Parallel-safety audit — PR #10778Parallelization — audited assembly:
No Findings: A (global-state) I audited every changed/added test method across the four changed unit-test files (
Nothing to flag for parallel-safety. Advisory only — heuristic, non-blocking. Re-run with
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
Review Summary
Well-structured feature addition implementing bounded GitHub Actions test-history snapshots with proper file locking, retry logic, and atomic writes. The design correctly separates concerns between the history service, the store, and the consumers (annotation reporter, summary reporter, artifact post-processor).
| # | Dimension | Verdict |
|---|---|---|
| 1 | Algorithmic Correctness | ✅ Clean — merge deduplication and bounded retention are correct |
| 2 | Threading & Concurrency | ✅ Clean — Volatile.Read/Write for _statsByTest, file-level advisory locking |
| 3 | Security & IPC Contract Safety | ✅ N/A — paths come from user CLI args; user already has FS access |
| 4 | Public API & Binary Compatibility | ✅ Clean — all types are internal; no public API surface added |
| 5 | Performance & Allocations | GetIdentity()/GetRunIdentity() allocate per-call in LINQ chains (see inline) |
| 6 | Cross-TFM Compatibility | ✅ Clean — uses System.Text.Json source gen, no TFM-gated APIs |
| 7 | Error Handling | ✅ Clean — retryable exceptions with exponential backoff; corrupt files reset gracefully |
| 8 | Localization | ✅ Clean — new strings in .resx, XLF files regenerated |
| 9 | Test Coverage | ✅ Good — dedicated GitHubActionsHistoryTests covers merge, aggregation, window, and boundary cases |
| 10 | Configuration & CLI | ✅ Clean — validation, dependency enforcement (--report-gh-history-window requires --report-gh-history) |
| 11 | Documentation | ✅ Clean — changelog, glossary, and PACKAGE.md updated |
| 12 | Naming & Conventions | CreateHistoryService (see inline) |
| 13–22 | Remaining dimensions | ✅ N/A or clean |
Overall: No blocking issues. Two minor items flagged inline. The PR is ready for human sign-off.
| } | ||
|
|
||
| private static GitHubActionsHistoryService CreateHistoryService(IServiceProvider serviceProvider) | ||
| => new( |
There was a problem hiding this comment.
Style (SA1028): This line uses 7-space indentation for the => body instead of the expected alignment. The .editorconfig enforces consistent indentation.
| => new( | |
| => new( |
| .. samples | ||
| .GroupBy( | ||
| static sample => $"{sample.GetIdentity()}\0{sample.GetRunIdentity()}", |
There was a problem hiding this comment.
Performance (§5): GetIdentity() and GetRunIdentity() allocate interpolated strings on every invocation within GroupBy. For large history snapshots (up to MaxTotalSamples = 10,000 samples), this creates significant GC pressure during the merge operation. Consider caching the identity per sample (e.g. computed once and stored in a transient field or local dictionary) or using a composite key tuple instead of string concatenation.
Summary
Validation
net8.0andnet462buildsnet462,net8.0, andnet10.0test assetsFixes #10770