Add support for multiple upgrade artifact source URIs

[lorienhu]

What is the problem this PR solves?

Currently, only one source for upgrade artifacts can be specified, meaning upgrades fail if that source is down.

How does this PR solve the problem?

Adds support for sending multiple upgrade artifact source URIs in an upgrade action with a new sources that deprecates source_uri. Backwards compatibility is maintained.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes [Fleet Server] Allow multiple artifact URLs to be configured for a single policy #2586
• Relates [Elastic Agent] Allow multiple artifact URLs to be configured for a single policy elastic-agent#4752
• Relates Add support for multiple upgrade artifact source URIs elastic-agent#16097

[ycombinator]

Currently, this array is unbounded. A client could send thousands of sources and eat up memory in Fleet Server unnecessarily. Consider setting a practical limit like, IDK, 20?

[lorienhu]

Added a check in 2eeb96d.

[ycombinator]

Could we add a test case for where neither sources nor source_uri is set?

[ycombinator]

Also, this is more of a naming nit, but would source_uris be more consistent than sources?

[lorienhu]

Also, this is more of a naming nit, but would source_uris be more consistent than sources?

I picked sources because the naming for sourceURI vs actual sourceURI was colliding in downloadArtifact before with the old name, since it's technically the base path/base URI.

[github-actions]

TL;DR

All three failed Buildkite jobs are caused by the same deterministic unit-test regression in this PR: TestConvertActionData/upgrade_action_handles_empty_source_uri_and_sources fails in internal/pkg/api. Immediate action is to align the test expectation with current omitempty serialization behavior (or explicitly set defaults in code if that behavior is intended to change).

Remediation

• Fix the failing test case at internal/pkg/api/handleCheckin_test.go:165-168 by updating the expected payload for {"version":"1.2.3"} to match actual output {"version":"1.2.3"}.
• If the intended contract is to always emit source_uri and sources, then instead update convertActionData in internal/pkg/api/handleCheckin.go:971-995 to set those defaults when both fields are absent, and keep the current test expectation.
• Re-run unit jobs (Run unit tests, Run unit tests with FIPS, Run fips140=only unit tests) after the fix.

Investigation details

Root Cause

convertActionData only backfills upgrade fields when one of them is present:

• If sources exists, it derives source_uri from first item.
• Else if source_uri exists, it derives sources.
• If neither exists, neither field is set.

Code path: internal/pkg/api/handleCheckin.go:977-994.

The new test case in this PR expects defaults even when input is only {"version":"1.2.3"}:

• internal/pkg/api/handleCheckin_test.go:165-168

That expectation conflicts with actual output and fails assertion at:

• internal/pkg/api/handleCheckin_test.go:241

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16731
• Jobs/steps:
  • :smartbear-testexecute: Run unit tests
  • :smartbear-testexecute: Run unit tests with FIPS
  • :smartbear-testexecute: Run fips140=only unit tests
• Key log excerpt from reproduced unit run on PR head (104cc1d):
  • --- FAIL: TestConvertActionData/upgrade_action_handles_empty_source_uri_and_sources
  • expected: {"source_uri":"","sources":[],"version":"1.2.3"}
  • actual : {"version":"1.2.3"}
  • FAIL github.com/elastic/fleet-server/v7/internal/pkg/api

Verification

• Reproduced on PR head commit 104cc1d7de06d23f08f8e625fd557d098af403b2.
• Reproduced with FIPS-tagged test invocation:
  • go test -tags=grpcnotrace,requirefips -run 'TestConvertActionData/upgrade_action_handles_empty_source_uri_and_sources' -v ./internal/pkg/api

Follow-up

• I checked open flaky-test issues in elastic/fleet-server; none matched this test name/path, so this appears to be a real regression rather than known flake.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.