Repository navigation
chore: resync 2026.08.26 c7n with July's tagging fixes - #26
Merged
Merged
Conversation
…1089) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…loud-custodian#11090) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d-custodian#11050) - kms-key: AWS managed keys (KeyManager == 'AWS', e.g. the keys behind alias/aws/s3, alias/aws/ebs) reject tag mutations, key policy changes, rotation changes, and ScheduleKeyDeletion - AWS owns their full lifecycle. - resolver-rule: Route 53's global auto-defined rules (e.g. rslvr-autodefined-rr-internet-resolver) have no owning account and reject tag mutations the same way. Fixes cloud-custodian#11027 Co-authored-by: Usman Qureshi <usman.nja.14@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Centralize dynamic tag lookup logic where possible * Extend the "default-value with no key" pattern from AWS to GCP & Azure * Fix aws.asg-specific tag logic, and extract shared behavior between ASG & generic tag logic into a mixin
### what `modify_bucket_tags()` tolerates the same transport failures `BucketAssembly.assemble()` already does, on both the tag re-fetch and the write. A bucket whose endpoint doesn't answer is logged and skipped; the rest of the batch is tagged as normal. `ConnectionClosedError` joins that set in both places. A degraded endpoint can accept a connection and then drop it rather than refusing or hanging, and that class was in neither handler - so it would still abort the tag action, and still abort collection for the whole account, which is what cd79871 set out to stop. `ProxyConnectionError`, `SSLError` and `ResponseStreamingError` are deliberately left to raise: the first two say something is wrong with the path to s3 rather than with one bucket, and the third can't plausibly happen on a response this small. ### why One bucket whose endpoint hangs silently costs healthy buckets their tags, on every run. `modify_bucket_tags()` caught only `ClientError`, so a connect timeout raised out of it and abandoned the rest of its batch. `_common_tag_processer` chunks at `Tag.batch_size` - 25 for `tag`, 100 for `remove-tag` - and walks each chunk serially, so the loss is every bucket after the hung one in its own chunk. How many that is depends on where the hung bucket sorts in `list_buckets` order, so it is silent, it varies, and it repeats every run. Measured against an account matching the reported one, 76 buckets with one timing out on every call: | hung bucket's position | before | after | | ---------------------- | ------ | ----- | | last | run errors, 75 of 75 tagged | succeeds, 75 of 75 | | eleventh | run errors, 61 of 75 tagged | succeeds, 75 of 75 | The run also reported error either way, so an account that was in fact being tagged still looked like a failure to anything gating on run status. This is what remains after cd79871, which stopped the same bucket from aborting resource collection outright. `list_buckets` is global, so every run in every region enumerates the whole account - one bucket whose endpoint hangs is on the path of every s3 policy, not only ones targeting its region. The bucket that couldn't be read is skipped rather than written to: tags are set atomically for the whole set, so writing one we couldn't re-read would drop whatever tags it has. ### testing New tests cover a hung bucket on the re-fetch and on the write leaving the rest of the batch tagged; a connection closed mid response surviving both collection and the tag action; and an end to end case - collection then tag action, with one bucket timing out on every call - asserting that a hung bucket at the head of a chunk costs none of the 29 buckets behind it. That last one fails against the previous behaviour with the timeout propagating out of the action, which is the reported symptom. ### docs No docs needed. --------------------------- 🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
There was a problem hiding this comment.
All reported issues were addressed across 66 files
Architecture diagram
sequenceDiagram
participant Policy as Policy Engine
participant Lookup as c7n.lookup
participant ASG as ASG Tag Action
participant EC2 as EC2 CreateTags
participant RGTA as Resource Groups Tagging API
participant CW as CloudWatch API
participant S3 as S3 API
participant KMS as KMS API
participant R53 as Route53 API
participant GCP as GCP/Knative API
participant Azure as Azure API
Note over Policy,Lookup: Dynamic Tag Value Resolution (shared across providers)
Policy->>Lookup: resolve_value(spec, resource, name, current_tags)
alt Lookup with key
Lookup->>Lookup: Extract value via jmespath
alt Key found
Lookup-->>Policy: Resolved value
else Key not found, default-value present
Lookup-->>Policy: Default value
else Key not found, no default
Lookup-->>Policy: Raise error
end
else Conditional default (no key)
alt Tag already present
Lookup-->>Policy: SKIP (omit tag)
else Tag absent
Lookup-->>Policy: Default value
end
else Static scalar
Lookup-->>Policy: Value as-is
end
Note over Policy,EC2: EC2 Native Tag Action (deleted resource handling)
Policy->>EC2: Process resource batch
EC2->>EC2: retry_skip_bad_ids()
EC2->>EC2: call CreateTags with resource IDs
alt Success
EC2-->>Policy: Tags applied
else ClientError with InvalidXXX.NotFound
EC2->>EC2: Extract bad IDs from error message
EC2->>EC2: Remove bad IDs from batch
opt Remaining IDs
EC2->>EC2: Retry with remaining IDs
EC2-->>Policy: Tags applied to survivors
end
end
Note over Policy,RGTA: Universal Tag Action (KMS, Route53, Elasticache)
Policy->>RGTA: Process resources with tags
alt AWS-managed KMS key
Policy->>Policy: Skip (KeyManager == 'AWS')
else Auto-defined resolver rule
Policy->>Policy: Skip (Id starts with rslvr-autodefined-rr-)
else Normal resource
RGTA->>RGTA: Call TagResources
alt Resource already deleted
RGTA->>RGTA: Skip failed ARN from FailedResourcesMap
RGTA-->>Policy: Partial success
else Success
RGTA-->>Policy: Tags applied
end
end
Note over Policy,CW: CloudWatch Dashboard Tagging
Policy->>CW: ListDashboards
CW-->>Policy: Dashboard list
Policy->>CW: ListTagsForResource (per dashboard ARN)
alt Dashboard deleted between calls
Policy->>Policy: Skip dashboard
else Success
CW-->>Policy: Existing tags
end
Policy->>CW: TagResource / UntagResource (native API)
CW-->>Policy: Confirmation
Note over Policy,S3: S3 Bucket Tagging with Transport Error Tolerance
Policy->>S3: get_bucket_tagging
alt Transport error (timeout, connection closed)
Policy->>Policy: Skip bucket, continue batch
else Success
S3-->>Policy: Existing tags
end
Policy->>S3: put_bucket_tagging
alt Transport error on write
Policy->>Policy: Log error, continue batch
else Success
S3-->>Policy: Confirmation
end
Note over Policy,GCP: GCP Cloud Run Label Actions
Policy->>GCP: Get resource
GCP-->>Policy: Service/Job object
Policy->>Policy: knative_body() - filter to schema fields
alt Unrecognized root field
Policy->>Policy: Raise PolicyExecutionError
else Valid body
Policy->>Policy: Strip c7n annotations
Policy->>GCP: Replace service/job (whole-object)
GCP-->>Policy: Updated object
end
Note over Policy,Azure: Azure Tag Actions
Policy->>Azure: Get resource
Azure-->>Policy: Resource with existing tags
Policy->>Azure: Resolve tag values via Lookup
alt Conditional default and tag exists
Policy->>Policy: Skip tag write
else Resolved tags to write
Policy->>Azure: Update resource tags
Azure-->>Policy: Confirmation
end
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ENG-8974
what
Cherry-picks six changes carried by the July release line onto
release/2026-08-26and stamps the workspace0.9.52a5.post2+a83dda5c9—fc8d0ac72already took.post1.Two are Stacklet-authored and will never arrive from upstream:
50d330ad1aws(s3) —modify_bucket_tags()tolerates the sametransport failures
BucketAssembly.assemble()already does, so onebucket whose endpoint hangs no longer costs the rest of its chunk
their tags (chore: aws(s3) - per bucket tolerance in the tag action #19).
1b5ad409cgcp(cloud-run) — label mutations send only fields theKnative schema defines, so the annotations c7n leaves on a resource
don't 400 the whole-object replace (gcp - cloud-run - drop non-schema fields from label mutations cloud-custodian/cloud-custodian#11052).
Four are upstream:
459879a59tags — dynamic tag lookups centralized inc7n.lookup,the "default-value with no key" form extended from AWS to GCP and
Azure, and asg's tag action moved onto the shared resolver (tags - standardize dynamic tagging support cloud-custodian/cloud-custodian#10968).
bf3a18095aws kms, route53 — AWS-managed KMS keys and Route 53auto-defined resolver rules excluded from the actions they reject
(aws - kms, route53 - exclude AWS-managed resources from actions cloud-custodian/cloud-custodian#11050).
16f20d030aws cloudwatch-dashboard — tags read and written throughthe cloudwatch apis, which cover dashboards where the resource
groups tagging api does not (aws - cloudwatch-dashboard - use native cloudwatch apis for tagging cloud-custodian/cloud-custodian#11090).
1dde3003caws tagging — a resource deleted between enumeration andthe tag call is dropped from the batch rather than failing it, in
place of the per-resource handling ebs carried (aws - tagging - tolerate resources deleted mid-run cloud-custodian/cloud-custodian#11089).
A seventh candidate, cloud-custodian#11016 (s3
augment()connection errors),needed no pick: it reached integration before August's base was cut,
so the content is already present under a different SHA.
why
August was cut from a later integration point, which is not a
superset of July's. Its base predates most of the upstream tagging
work July picked up, and none of that has been applied since, so a
deployment moving from the July line to August loses all six changes.
The two Stacklet-authored ones are the same pair the June-to-July
port existed to rescue; without this, they would be lost a second
time.
testing
Each pick brings its own coverage: per-bucket tolerance on the s3 tag
action, including an end-to-end case through collection and tagging;
cloud run replace bodies rejecting root fields absent from the
schema; dynamic tag resolution across AWS, GCP and Azure and on the
asg-specific path; AWS-managed kms keys and auto-defined resolver
rules skipped by their actions; cloudwatch dashboard tag and
remove-tag against the native apis; and batched ec2 tagging dropping
the ids an error names as gone.
What the diff cannot settle is whether cloud-custodian#10942, already on the August
line, and cloud-custodian#10968 cohere when applied in this order — the reverse of
the order July received them in. That is the question the suite run
on this branch answers.
docs
No docs needed.
🤖 Generated with Claude Code