Conversation
stow v0.5.0 moves its S3 provider to aws-sdk-go-v2, so the errors it returns are smithy errors and no longer awserr.Error. The checks for a missing bucket and for a bucket the caller already owns matched on awserr.Error and would silently stop matching: a write to a missing bucket would not create it, and creating an existing bucket would become a hard error. Match on smithy.APIError and its error code. stow v0.5.0 also adds functions that take a context. Call the stow location, containers and items through them, so a request to the blob store is cancelled with the context of the caller. The context of ReadRaw now covers reading from the returned reader as well. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai>
kumare3
approved these changes
Oct 2, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Two things, both tied to moving flytestdlib to stow v0.5.0:
aws-sdk-gov1 toaws-sdk-go-v2(s3: migrate to aws-sdk-go-v2 stow#29), so the errors it returns are smithy errors and no longerawserr.Error.awsBucketIsNotFoundandawsBucketAlreadyExistsmatch onawserr.Error. With the new stow they keep compiling but never match:Location,ContainerandItemmethods take no context, so a request to the blob store could not be cancelled or given a deadline by the caller, even though everyStowStoremethod has actx. stow v0.5.0 adds functions that take a context (Add methods that take a context stow#31).What changes were proposed in this pull request?
github.com/flyteorg/stowto v0.5.0.awsBucketIsNotFoundandawsBucketAlreadyExistsmatch onsmithy.APIErrorand itsErrorCode()(NoSuchBucket,BucketAlreadyOwnedByYou).flytestdlib/storageno longer importsaws-sdk-gov1;flyteplugins/go/tasks/awsstill does.StowStoregoes through the stow functions that take a context:CreateContainerContext,ContainerContext,ItemContext,ItemsContext,PutContext,RemoveItemContext,OpenContext,MetadataContextandETagContext. These are all the stow calls in the repository. For the s3, google, azure, swift and oracle backends the context reaches the request; for a backend without context support (local) it is checked before the call.Behaviour to be aware of:
ReadRawnow also covers reading from the returned reader, so it must stay alive until the reader is closed. The callers in this repository (copyImpl.CopyRaw, the cached raw store,ReadProtobuf, flytecopilot'sDownloadFileFromStorage) read within the same context.Head,List,ReadRaw,WriteRaw,CopyRawandDeletewith the error of the context.NewStowRawStoretakes no context and still loads the base container withcontext.TODO().cloud.google.com/go/storagev1.56.0 → v1.69.0,azblobv1.4.0 → v1.8.2,ncw/swift→ncw/swift/v2, and theaws-sdk-go-v2S3 client. Its S3 provider uploads objects between 5 MB and 16 MB in one request instead of a multipart upload.How was this patch tested?
awsBucketIsNotFound/awsBucketAlreadyExistsand theWriteRawtests for a missing and an already owned bucket now build their errors assmithy.GenericAPIError. With the stow bump alone and the oldawserr.Errorchecks these tests fail.TestStowStore_PassesContext: with a location, container and item that have the stow methods with a context, each ofWriteRaw,Head,ReadRaw,List,Delete,CopyRawandLoadContainerpasses the caller's context to stow; with mocks that have no such methods, a cancelled context stops every operation and nothing is written.go build ./...passes;go test ./flytestdlib/storage/... ./dataproxy/... ./flytecopilot/...passes.Labels
Check all the applicable boxes
Related PRs
errors.Aschecks this PR changes)