Skip to content

fix: return array pod build errors - #8072

Merged
Sovietaced merged 2 commits into
flyteorg:masterfrom
1fanwang:1fanwang/fix-array-subtask-build-error
Sep 24, 2026
Merged

Sovietaced merged 2 commits into
flyteorg:masterfrom
1fanwang:1fanwang/fix-array-subtask-build-error

Conversation

@1fanwang

Copy link
Copy Markdown
Contributor

Tracking issue

None.

Why are the changes needed?

A legacy container_array task with an invalid pod specification panics instead of reporting the validation error.
The executor now returns the original task-specification error. Allocation cleanup no longer discards it or replaces it with a cleanup error.

What changes were proposed in this pull request?

Check the pod-builder error before asserting the returned object type, and preserve that error while releasing the allocation token. Regression coverage includes successful and failed cleanup.

How was this patch tested?

Testing Done

The file-backed replay in this PR starts the real array executor from empty state. It reads task and input protobufs from local storage, advances through the normal phases, and reaches the real pod builder with a missing primary-container configuration.

This is a plugin-level replay. Only surrounding execution metadata uses the existing test fixture; neither the executor nor the pod builder is mocked. It does not start Admin or Kubernetes, because the error occurs before a Kubernetes request.

To reproduce the before case from the PR checkout, copy only the replay test onto the base. The implementation stays unchanged:

git worktree add --detach ../flyte-array-before 6f352060a580be92c080e033791c3233e0d9ea59
cp flyteplugins/go/tasks/plugins/array/k8s/integration_test.go \
  ../flyte-array-before/flyteplugins/go/tasks/plugins/array/k8s/integration_test.go
(cd ../flyte-array-before/flyteplugins && \
  CGO_ENABLED=0 go test -p 2 ./go/tasks/plugins/array/k8s \
  -run '^TestExecutorPodBuildError$' -count=1 -v)

The base reaches round 4 and produces:

panic: interface conversion: client.Object is nil, not *v1.Pod [recovered, repanicked]

Run the same case on this branch:

(cd flyteplugins && \
  CGO_ENABLED=0 go test -p 2 ./go/tasks/plugins/array/k8s \
  -run '^TestExecutorPodBuildError$' -count=1 -v)

It returns the validation error instead:

round=4 returned_error=[BadTaskSpecification] invalid TaskSpecification, config missing [primary_container_name] key in [map[]]
Raw replay output

Selected base output, with stack frames omitted:

    integration_test.go:155: round=1 phase=PhaseRunning
    integration_test.go:155: round=2 phase=PhaseRunning
    integration_test.go:155: round=3 phase=PhaseRunning
panic: interface conversion: client.Object is nil, not *v1.Pod [recovered, repanicked]

Selected branch output:

    integration_test.go:155: round=1 phase=PhaseRunning
    integration_test.go:155: round=2 phase=PhaseRunning
    integration_test.go:155: round=3 phase=PhaseRunning
    integration_test.go:152: round=4 returned_error=[BadTaskSpecification] invalid TaskSpecification, config missing [primary_container_name] key in [map[]]

Labels

fixed

Setup process

Go 1.26.2. The replay uses a temporary directory and needs no Kubernetes configuration.

Screenshots

Not applicable.

Check all the applicable boxes

No documentation update is needed for this error-path fix. The test run covers the affected array packages and ordinary pod package.

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

None.

Stack

If you do use git town to manage PR Stacks, the stack relevant to this PR
will show below. Otherwise, you can ignore this section.

Docs link

Not applicable.

Check the pod-builder error before asserting the returned object type. Preserve that error when releasing the allocation token so cleanup cannot hide the failed launch.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@github-actions github-actions Bot added the flyte label Sep 23, 2026
@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.32%. Comparing base (6f35206) to head (62c6e12).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8072      +/-   ##
==========================================
+ Coverage   56.50%   57.32%   +0.81%     
==========================================
  Files         772      931     +159     
  Lines       52285    58315    +6030     
==========================================
+ Hits        29546    33429    +3883     
- Misses      19974    21833    +1859     
- Partials     2765     3053     +288     
Flag Coverage Δ
unittests-datacatalog 53.62% <ø> (ø)
unittests-flyteadmin 53.24% <ø> (ø)
unittests-flytecopilot 48.05% <ø> (ø)
unittests-flytectl 64.16% <ø> (?)
unittests-flyteidl 76.63% <ø> (ø)
unittests-flyteplugins 60.59% <100.00%> (+0.13%) ⬆️
unittests-flytepropeller 53.84% <ø> (ø)
unittests-flytestdlib 64.39% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Sovietaced
Sovietaced merged commit b2be54c into flyteorg:master Sep 24, 2026
79 of 80 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants