Skip to content

storage: bump stow to v0.5.0, pass the context to stow - #8115

Open
SVilgelm wants to merge 1 commit into
mainfrom
svilgelm/stow-v0.5-context
Open

SVilgelm wants to merge 1 commit into
mainfrom
svilgelm/stow-v0.5-context

Conversation

@SVilgelm

@SVilgelm SVilgelm commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Two things, both tied to moving flytestdlib to stow v0.5.0:

  1. stow v0.5.0 moves its S3 provider from aws-sdk-go v1 to aws-sdk-go-v2 (s3: migrate to aws-sdk-go-v2 stow#29), so the errors it returns are smithy errors and no longer awserr.Error. awsBucketIsNotFound and awsBucketAlreadyExists match on awserr.Error. With the new stow they keep compiling but never match:
    • a write to a missing bucket no longer creates it;
    • creating a bucket that already exists becomes a hard error.
  2. The stow Location, Container and Item methods take no context, so a request to the blob store could not be cancelled or given a deadline by the caller, even though every StowStore method has a ctx. stow v0.5.0 adds functions that take a context (Add methods that take a context stow#31).

What changes were proposed in this pull request?

  • Bump github.com/flyteorg/stow to v0.5.0.
  • awsBucketIsNotFound and awsBucketAlreadyExists match on smithy.APIError and its ErrorCode() (NoSuchBucket, BucketAlreadyOwnedByYou). flytestdlib/storage no longer imports aws-sdk-go v1; flyteplugins/go/tasks/aws still does.
  • Every call to stow in StowStore goes through the stow functions that take a context: CreateContainerContext, ContainerContext, ItemContext, ItemsContext, PutContext, RemoveItemContext, OpenContext, MetadataContext and ETagContext. These are all the stow calls in the repository. For the s3, google, azure, swift and oracle backends the context reaches the request; for a backend without context support (local) it is checked before the call.

Behaviour to be aware of:

  • The context passed to ReadRaw now also covers reading from the returned reader, so it must stay alive until the reader is closed. The callers in this repository (copyImpl.CopyRaw, the cached raw store, ReadProtobuf, flytecopilot's DownloadFileFromStorage) read within the same context.
  • A cancelled or expired context now fails Head, List, ReadRaw, WriteRaw, CopyRaw and Delete with the error of the context.
  • NewStowRawStore takes no context and still loads the base container with context.TODO().
  • stow v0.5.0 brings its own dependency changes: cloud.google.com/go/storage v1.56.0 → v1.69.0, azblob v1.4.0 → v1.8.2, ncw/swift → ncw/swift/v2, and the aws-sdk-go-v2 S3 client. Its S3 provider uploads objects between 5 MB and 16 MB in one request instead of a multipart upload.

How was this patch tested?

  • The existing table tests for awsBucketIsNotFound / awsBucketAlreadyExists and the WriteRaw tests for a missing and an already owned bucket now build their errors as smithy.GenericAPIError. With the stow bump alone and the old awserr.Error checks these tests fail.
  • New TestStowStore_PassesContext: with a location, container and item that have the stow methods with a context, each of WriteRaw, Head, ReadRaw, List, Delete, CopyRaw and LoadContainer passes the caller's context to stow; with mocks that have no such methods, a cancelled context stops every operation and nothing is written.
  • go build ./... passes; go test ./flytestdlib/storage/... ./dataproxy/... ./flytecopilot/... passes.
  • Not run against real S3, GCS or Azure buckets. In stow the context functions were run against MinIO, Azurite, fake-gcs-server and OpenStack Swift.

Labels

  • changed
  • fixed

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

stow v0.5.0 moves its S3 provider to aws-sdk-go-v2, so the errors it
returns are smithy errors and no longer awserr.Error. The checks for a
missing bucket and for a bucket the caller already owns matched on
awserr.Error and would silently stop matching: a write to a missing
bucket would not create it, and creating an existing bucket would
become a hard error. Match on smithy.APIError and its error code.

stow v0.5.0 also adds functions that take a context. Call the stow
location, containers and items through them, so a request to the blob
store is cancelled with the context of the caller. The context of
ReadRaw now covers reading from the returned reader as well.

Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Copilot AI balanced review requested due to automatic review settings October 2, 2026 20:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added the flyte2 label Oct 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants