Skip to content

Preserve CTRF duplicate results and test attachments - #10769

Open
Amaury Levé (Evangelink) wants to merge 5 commits into
microsoft:mainfrom
Evangelink:dev/amauryleve/ctrf-report-extension
Open

Preserve CTRF duplicate results and test attachments#10769
Amaury Levé (Evangelink) wants to merge 5 commits into
microsoft:mainfrom
Evangelink:dev/amauryleve/ctrf-report-extension

Conversation

@Evangelink

@Evangelink Amaury Levé (Evangelink) commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary

  • preserve every terminal test execution when multiple results share an MTP UID instead of inferring retry/flaky relationships
  • serialize per-test FileArtifactProperty values as schema-compliant CTRF attachments with MIME inference, fallback metadata, and descriptions
  • update focused unit and acceptance coverage for duplicate UIDs and real file artifacts

Testing

  • Microsoft.Testing.Extensions.UnitTests: 52 CTRF reporter tests passed on net8.0
  • Microsoft.Testing.Platform.Acceptance.IntegrationTests: 15 CTRF acceptance cases passed across net462, net8.0, and net10.0

Fixes #10764

Preserve duplicate-UID executions, emit test attachments, and avoid splitting Unicode surrogate pairs during truncation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Record the attachment projection in the internal API baseline and align the regression assertion with current analyzer rules.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings August 26, 2026 11:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Preserves duplicate-UID CTRF executions and adds test attachment serialization, addressing #10764.

Changes:

  • Removes UID-based retry collapsing.
  • Adds MIME-aware CTRF attachments.
  • Expands unit and acceptance coverage.
Show a summary per file
File Description
CapturedAttachment.cs Adds attachment DTO.
CapturedTestResult.cs Stores captured attachments.
TestResultCapture.cs Captures attachment metadata and MIME types.
CtrfReportEngine.JsonTestWriter.cs Serializes CTRF attachments.
CtrfReportEngine.JsonSerializer.cs Counts and emits every execution.
CtrfReportEngine.TestCollapsing.cs Removes UID-based collapsing.
InternalAPI.Unshipped.txt Tracks new internal APIs.
CtrfReportEngineTests.cs Tests attachments and duplicate UIDs.
CtrfReportTests.cs Adds end-to-end attachment and duplicate coverage.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 9/9 changed files
  • Comments generated: 2
  • Review effort level: Balanced

@Evangelink
Amaury Levé (Evangelink) enabled auto-merge (squash) August 26, 2026 12:25
@Evangelink Amaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Aug 26, 2026
Correct the shared capture documentation and cover attachment path-resolution failures without dropping later valid artifacts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 26, 2026 12:31
Assert that MSTest retry attempts remain distinct CTRF results instead of expecting UID-based flaky inference.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 11/11 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Comment thread src/Platform/Microsoft.Testing.Extensions.CtrfReport/CapturedAttachment.cs Outdated
Restore the required source BOM and cover every MIME mapping plus surrogate-safe truncation for attachment names, paths, and descriptions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 26, 2026 14:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

src/Platform/Microsoft.Testing.Extensions.CtrfReport/CtrfReportEngine.JsonSerializer.cs:104

  • Hard-coding flaky to zero reintroduces #10292 for MSTest [Retry]. These updates carry RetryAttemptProperty (MSTestTestNodeConverter.cs:116-122), which was added specifically so CTRF can emit retryAttempts/flaky; its contract still documents that behavior (RetryAttemptProperty.cs:12-16). Stop UID-only inference for untagged duplicate UIDs, but retain retry collapsing for explicitly tagged attempt sequences. As written, a fail-then-pass retry is incorrectly reported as two unrelated tests rather than one flaky pass.
            // MTP UIDs are not unique per execution, so they cannot establish retry
            // relationships. Keep flaky at zero until explicit attempt correlation is
            // available rather than dropping duplicate-UID results or inventing retries.
            writer.WriteNumber("flaky", 0);
  • Files reviewed: 11/11 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

return attachments;
}

private static string GetAttachmentContentType(string path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I used this list for our default mappings: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/MIME_types/Common_types

We're going in the opposite direction, since we are combining attachment name + media type into a filename, so we're mapping media types to file extensions, but it seems like you could use roughly the same list. We also do an on-Windows-only fallback to look things up in the registry under HKEY_CLASSES_ROOT.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state/needs-review Awaiting review from the team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CTRF reporter should preserve duplicate results and attachments

3 participants