Repository navigation
Conversation
…emporalio#12089) ## What changed? Two allow-lists — `allowedMethodsDuringHandover` and `selectedAPIsForwardingRedirectionPolicyWhitelistedAPIs` — are now keyed by full gRPC method instead of bare method name. `SelectedAPIsForwardingRedirectionPolicy` gains `WithAdditionalWhitelistedMethods`, reachable through `Redirection.WithAdditionalWhitelistedMethods`, and `DCRedirectionPolicy` takes `fullMethod` where it took `apiName`. `common/api.ParseFullMethod` is new and exported, for callers that need to check their own entries. ## Why? A bare method name cannot say which service it means. `GetSearchAttributes` is on both WorkflowService and AdminService and the frontend serves both, so one bare entry was covering two services by accident. An embedder registering its own services on this server shares the same name space, and had no way to opt its methods into forwarding at all: the whitelist is private, so a registered redirect response never forwarded. Service attribution came from the descriptors of the services the frontend registers, which also found that `ReplicateEventsV2` is a HistoryService method and has never matched anything here. `localAPIResponses` and `globalAPIResponses` stay bare-keyed — `Intercept` has already narrowed the request to WorkflowService before either is consulted. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) `common/rpc/interceptor/allowlist_test.go` covers both lists: the multi-service entry, that one service's entry no longer reaches another's same-named method, the extension point, and that every entry parses and names a real method. ## Potential risks - **A caller passing bare names to `additionalAllowedMethodsDuringHandover` will find its entries inert.** They match nothing now, which for that list fails open — a method it meant to let through a handover stops being let through. The constructor logs a warning for such entries rather than refusing to start, since failing on an arbitrary restart is the larger failure; there is a TODO to fail at deploy time once callers verify their lists first. That check catches a malformed entry or an unknown method on a known service, but not a misspelled service name, so callers should also assert their own list in a test — `api.ParseFullMethod` is exported for that. - **A multi-service method needs one entry per service.** `GetSearchAttributes` was the only one, found by enumerating descriptors rather than by hand; missing one would silently start gating that service's call during a handover. - **`DCRedirectionPolicy` implementers now receive a full method.** The interface is exported; an implementation matching on the bare name needs updating. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…2233) ## What changed? Exclude expected FailedPrecondition responses from cancel/terminate activity batch tasks from batcher_processor_errors. ## Why? These terminal or duplicate outcomes are retained in batch-operation failure totals but should not inflate the processor error-rate metric. Fixes temporalio#12232. ## How did you test it? - [X] built - [ ] run locally and tested manually - [ ] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s)
…fires at When() (temporalio#11990) ## What changed? Derive the per-key switch time once in a new `switchTime()` helper used by both `Value` and `When` in `common/dynamicconfig/gradual_change.go`, so that `Value(key, When(key)) == New` holds by construction. ## Why? `Value` computed a truncated `uint32` threshold with strict `fingerprint < threshold`, while `When` mapped the fingerprint back to a time with different rounding. `Value(key, When(key))` was therefore deterministically still Old, and stayed Old for up to `(End-Start)/2^32` after `When` (~200µs for a 10-day rollout). The subscription wrapper only schedules a new timer when `When(key)` is strictly in the future, so when the runtime timer fired within that quantum of its deadline — common, Go timers routinely fire sub-millisecond around the deadline — it reevaluated to Old, scheduled nothing, and the subscriber never received the New value until an unrelated config change. This affects the new-matcher and fairness rollouts consumed by `task_queue_partition_manager`. Switch times remain uniformly distributed, strictly after `Start`, and capped at `End`. ## How did you test it? Two regression tests: `Value(key, When(key))` now returns New for all tested keys, and a fake-clock subscription test where the timer fires exactly at the deadline now observes the transition. Both fail on `main` and pass with the fix; `go test ./common/dynamicconfig/... -count=1` is green and `go vet` is clean. ## Potential risks Keys switch at most one time-quantum earlier than before (still within the configured window); no API change. Closes temporalio#11989
## What changed? Adds bidirecitonal linking for Nexus handler calls made via the `NexusHandler`-variant completion callback. e.g. if a Workflow Update had a completion callback to invoke the `MyCompletionHandler` service's `OnComplete` operation, the Nexus handler would receive a link to the Workflow Update. And if it spawned any Temporal resources, the Workflow Update would receive the links. ## Why? Without this capability, you wouldn't be able to cross reference Temporal resources that were created as part of handling a completion callback. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks None that I can think of.
## Summary Build `temporal-server` releases with the `disable_grpc_modules` tag. | Target | Before | After | Raw saving | Reduction | | --- | ---: | ---: | ---: | ---: | | Linux AMD64 | 137,777,312 B (131.39 MiB) | 126,029,984 B (120.19 MiB) | 11.20 MiB | 8.53% | | Linux ARM64 | 127,991,968 B (122.06 MiB) | 116,785,312 B (111.38 MiB) | 10.69 MiB | 8.76% |
…emporalio#12083) ## What changed? When matching SQL calls fail because the context was canceled or timed out, we now keep that error unwrap-able instead of turning it into Unavailable. That covers `txExecute`, `lockTaskQueue`, `GetTaskQueue`, matching `GetTasks`, and `CompleteTasksLessThan` (v1 and v2). Classic persist-ack also skips the store-failure log on cancel. This does not change every SQL Unavailable wrapper. Other stores can still flatten cancel. Fixes temporalio#12201. ## Why? Matching persist-ack, task GC, and idle-unload ownership checks (`periodicSync` → `GetTaskQueue`) can race task-queue shutdown. Wrapping cancel as Unavailable made that look like a persistence outage and broke `IsContextCanceledErr`. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…o#12263) ## What changed? Upgrade `go.temporal.io/auto-scaled-workers` to the [`7196792` merge commit](temporalio/temporal-auto-scaled-workers@7196792) from [temporalio#138](temporalio/temporal-auto-scaled-workers#138). That PR removed hundreds of Go packages from the build. ## Why? | Target | Before | After | Raw saving | Reduction | | --- | ---: | ---: | ---: | ---: | | Linux AMD64 | 125,792,416 B | 107,151,520 B | 17.78 MiB | 14.82% | | Linux ARM64 | 116,523,168 B | 98,893,984 B | 16.81 MiB | 15.13% |
## What changed? Bump the default Temporal CLI version used by Docker image builds on `main` from 1.6.1 to 1.9.1.
## What changed 1. clean up stale todos for chasm 2. move propagation util method to chasm path (prepare to support schedule time skipping) 3. history timer task supports chasm-generated tasks with time skipping 4. cherry-pick back add task regeneration in passive clusters ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s)
) ## What changed? - Validate `system.secondaryVisibilityWritingMode` when loading dynamic configuration. - Log a warning and treat unrecognized values as `off` instead of allowing them to fail during visibility writes. - Prevent invalid values such as `duel` or `dual ` from causing visibility tasks to repeatedly fail and eventually land in the DLQ. ## Why? `system.secondaryVisibilityWritingMode` only accepts `off`, `on`, and `dual`, but invalid values were previously accepted by configuration validation and only surfaced when a visibility write was attempted. This could cause visibility tasks to fail repeatedly and eventually be sent to the DLQ, while workflow execution and visibility reads continued to work normally. Validating the value when loaded makes configuration errors visible at startup, while falling back to `off` keeps the cluster functional and avoids losing visibility writes entirely. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests ## Potential risks - An invalid `secondaryVisibilityWritingMode` value will now be treated as `off`, so secondary visibility writes will be disabled until the configuration is corrected. - Existing valid values (`off`, `on`, and `dual`) retain their current behavior. fixes temporalio#12109
… annotate workflow types with versioning behaviours (temporalio#12260) ## Summary - accept workflow task completions that report a versioning behavior without a valid versioned deployment - normalize the behavior to `UNSPECIFIED` for explicit `UNVERSIONED`, absent deployment metadata, and invalid legacy deployment metadata - preserve behavior for versioned workers and valid legacy deployments, while rejecting malformed present deployment options ## Testing - `go test -tags test_dep ./service/history/api/respondworkflowtaskcompleted -run '^TestWorkflowTaskCompletedHandlerSuite$/^TestVersioningBehaviorNormalization$' -count=1` - `go test -tags test_dep ./service/history/api/respondworkflowtaskcompleted -run '^TestWorkflowTaskCompletedHandlerSuite$/^TestVersioningBehaviorWithoutWorkerVersioningModeIsRejected$' -count=1` - `go test -tags test_dep ./service/history/api/respondworkflowtaskcompleted -count=1` <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes worker versioning validation on a core history API path; behavior is softened for unversioned/misconfigured workers but malformed deployment options still fail fast. > > **Overview** > **Workflow task completions** can now succeed when workers report a **versioning behavior** (e.g. from workflow-type defaults) but run with **`useVersioning: false`** / unversioned deployment options. The server **normalizes** `VersioningBehavior` on `RespondWorkflowTaskCompleted` instead of rejecting those requests outright. > > When behavior is set, it is **kept only for versioned workers** (or a **valid legacy** `Deployment`). Otherwise it is cleared to **`UNSPECIFIED`**—including **unversioned** `WorkerVersioningMode`, **missing** deployment metadata, and **invalid** legacy deployments. **`DeploymentOptions` with behavior but unspecified/unknown versioning mode** still returns **`InvalidArgument`**. > > Normalization was **moved out of** `AddWorkflowTaskCompletedEvent` in the workflow task state machine so history and metrics use the **already-adjusted** request field. **Tests** cover normalization paths and rejection of malformed deployment options. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 7aae54d. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
## What changed? Skip the namespace failover version comparison in `validateStateMachineRefWithoutTransitionHistory` when the ref carries no meaningful value to compare. This lets the existing fallback handle completion tokens converted from CHASM to HSM the same way it already handles native HSM ones. ## Why? A Nexus completion token converted from a CHASM component ref carries a zero-value versioned transition, since CHASM has no HSM equivalent to populate it with. After a reset rebuilds the operation as HSM under a nonzero namespace failover version, comparing that real value against the token's zero placeholder falsely rejects the completion as stale, even though the existing reset fallback (retry without a run ID) would otherwise resolve it correctly. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) ## Potential risks Currently no risk since CHASM to HSM migration is not started and there is no cross framework completions enabled at the moment. The fix in this PR is against a rare edge during the rollout where a nexus operation is initially created in a CHASM tree and then after + failover and reset and also turning off the rollout flag results in the operation ending up in a HSM tree and the completion comes via CHASM nexus completion token.
…n-write (temporalio#11579) ## What changed? Just mutate outstandingTasks in place. ## Why? We already were fully guarded by the lock, the extra copy was entirely unneeded. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed? Use each cached client's gRPC connection to replace the client after that connection shuts down. ## Why? A generated client can remain cached after its shared internode connection closes, causing requests to reuse that connection until the cache is evicted. ## How did you test it? - `go test -tags test_dep ./common ./client` - `make lint-code-fast` --------- Co-authored-by: Yichao Yang <yichao@temporal.io>
## What changed? There are more error types from the Worker Controller Instance client (for serverless worker configuration) that would probably be good to expose to users as ApplicationErrors than just `InvalidArgument` without needing to explicitly configure `frontend.maskInternalErrorDetails` to false. Start by adding the `FailedPrecondition` code. The conversion logic is moved to a common function, so this can be extended easily as needed. ## Why? Currently only the `InvalidArgument` gRPC code gets propagated from WCI client calls without getting wrapped in an `Internal` error by the calling code. This means that any other error types returned by the WCI client will by default get masked by the frontend to the generic internal error message, "something went wrong, please retry", which isn't particularly helpful. This happens even if the WCI client itself returns a more useful error message that isn't of type `InvalidArgument`. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s)
## What changed? Adds a log for V2 schedules when they migrate, to differentiate between being created vs migrated ## How did you test it? - [X] built - [X] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks low risk, just a log
## What changed? Add ConditionRangeID to CompleteTasksLessThanRequest and make CompleteTasksLessThan conditional on unchanged rangeid, just like CreateTasks. ## Why? When fair task reader gc's tasks, it's not protected by write fencing like adding tasks. So there's a small race window: a node advances its ack level to C, but it's not persisted yet. Another node takes over and loads the previous ack level, A. It writes a task above A but below C, call it B. The first node now deletes below C, deleting B. Note that this only happens with fair queues because fifo queues always write above the rangeid. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) - [x] validated in tla+ model (not in repo yet)
## What This requires callbacks to be associated with the namespace in which they were created. ## LLM Summary Added `callback.internal.sameNamespaceArchetypes`, a dynamic-config list of fully-qualified CHASM archetype names whose internal callbacks must target the callback source namespace. The default list contains the Scheduler archetype. Internal callback dispatch now applies the generic configured policy without referring to Scheduler directly. Component references are also structurally validated before dispatch. ## Why? Some CHASM features require same-namespace callbacks while others may support cross-namespace delivery. Keeping this as an archetype policy avoids imposing Scheduler constraints on unrelated components and lets the list evolve without changing dispatch logic. ## How did you test it? - [x] covered by existing tests - [x] added new unit test(s) - [X] Manual testing --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## What changed? The V1 (worker-service) scheduler workflow now copies `VersioningOverride` from the schedule action into the `StartWorkflowExecutionRequest` it builds. Fixes temporalio#12148 ## Why? `startWorkflow` in `service/worker/scheduler/workflow.go` builds the start request field by field, and `VersioningOverride` was not among those fields. The frontend accepts the override, validates it and persists it, and `DescribeSchedule` reads it back intact, so nothing signals that it is ignored. The started workflows simply carry no override and route to the Current version. temporalio#11308 added this forwarding to the CHASM scheduler (`chasm/lib/scheduler/invoker_tasks.go`) and added frontend validation, but its diff never touched `service/worker/scheduler/workflow.go`. Since `history.enableCHASMSchedulerCreation` defaults to `false`, new schedules are still created on V1, where the field was dropped. This is the same one-line omission that was previously fixed for `UserMetadata` (temporalio#6462) and `Priority` (temporalio#7134) in the same request builder. ## How did you test it? - [x] built - [x] added new unit test(s) `TestStart` in `service/worker/scheduler/workflow_test.go` is the existing test for "all fields in the start request", so the assertion was added there alongside the `UserMetadata` one. Verified it fails without the fix: ``` Error: Proto mismatch (-want +got): (*workflow.VersioningOverride)(Inverse(protocmp.Transform, protocmp.Message{ + "@invalid": bool(true), "@type": s"temporal.api.workflow.v1.VersioningOverride", - "pinned": s`{behavior:PINNED_OVERRIDE_BEHAVIOR_PINNED, version:{...}}`, })) ``` `go test -tags test_dep ./service/worker/scheduler/...` and `make lint-code-fast` both pass with the fix. ## Potential risks **No new scheduler workflow version.** The change only alters the input to the `StartWorkflow` local activity. Local activity markers record activity ID, type, replay time, attempt and backoff, not the input, so replay cannot diverge on it. This matches how `UserMetadata` and `Priority` were added to the same request without a `SchedulerWorkflowVersion` bump. All existing replay fixtures pass. **Behavior change for existing schedules.** Schedules that already carry a stored override begin honoring it as soon as the new binary picks them up. If the pinned version does not serve the action's task queue, the start is now rejected, where before it silently ran on the Current version: ``` Pinned version 'my-deployment:build-a' is not present in task queue 'my-queue' of type 'Workflow' ``` That is the intended behavior and matches both a direct `StartWorkflowExecution` and the CHASM scheduler, but it does surface as a newly failing start rather than a silent one. **No V1 kill switch.** CHASM gates this behind the `EnableVersioningOverride` tweakable, which is operator-flippable because `scheduler.tweakables` is a namespace dynamic-config setting. V1's `CurrentTweakablePolicies` is a static Go var, so adding a matching field there would be a compile-time constant rather than a real switch. I left it out; happy to wire a dynamic config setting through `schedulerDynamicConfig` instead if you would like the rollout control. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lio#12073) ## What changed? Reapply now skips a Nexus event whose operation is in neither the HSM nor the CHASM tree, instead of failing the whole batch. `cherryPickChasmEvent` treats a `serviceerror.NotFound` from an event definition as "this tree does not own the operation", which is what makes it safe for `cherryPickHSMEvent` to fall back on `hsm.ErrStateMachineNotFound` again. Also a change was made on the CHASM, an invalid transition on an operation CHASM owns is now skipped (parity with how HSM already treats `hsm.ErrInvalidTransition`). And if CHASM is disabled for the workflow we no longer return Internal error. ## Why? While working on HSM to CHASM migration for Nexus ops in workflows we introduced temporalio#11384 and had to drop CHASM backed Nexus Ops during workflow reset. This change is filling that gap without reintroducing the workflow replication errors observed originally. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks Low. New tests reproduce the replication error described in temporalio#11384, and tests previously skipped for lack of CHASM support are re-enabled. Replication validation via bench tests in Temporal Cloud should still be monitored after the merge. `executable_sync_versioned_transition_task.go:238` still reads any escaping `*serviceerror.NotFound` as "workflow absent in source cluster" and builds a deletion task. No Nexus `NotFound` escapes reapply after this change, but the overload itself is untouched and worth its own issue.
## What changed? Lookup dynamic config for using chasm for nexus ops in workflows once for a batch of event applications. ## Why? Dynamic config for using chasm for nexus ops in workflows could be changed during a mutable state rebuild. Looking up the value once before the batch of event applications will direct the Nexus op schedule events consistently to either hsm or chasm. ## How did you test it? - [X] built - [ ] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Low, we are reducing dynamic config lookup during mutable state rebuild into one and make the tree selection consistent.
## What changed? Added the GrantEagerDispatch matching service RPC and request/response messages, including generated clients, mocks, helpers, metrics, retry, and log-tag plumbing. ## Why? History needs to ask a task queue partition whether workflow or activity tasks may be dispatched eagerly while accounting for matching rate limits. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) Ran: - `go test -tags test_dep ./api/matchingservice/v1 ./api/matchingservicemock/v1 ./client/matching ./common/rpc/interceptor/logtags` - `make lint-code-fast` (blocked by an unrelated existing staticcheck deprecation in `tests/timeskipping_propagation_test.go:702`) ## Potential risks The RPC is additive, but older matching hosts return Unimplemented; callers must treat failures as a denial of eager dispatch as documented.
) ## Summary - preserve the incoming HLC when replicated V2/V3 task queue user data wins conflict resolution - clone the clock so persisted user data cannot alias the replication request - retain existing nil-clock selection and V1 build ID revival behavior ## Tests - go test -tags test_dep ./service/matching -run TestMatchingEngine_Classic_Suite/TestApplyTaskQueueUserDataReplicationEventPreservesSelectedClock -count=1 - go test -tags test_dep ./service/matching -run TestMatchingEngine_Classic_Suite/TestApplyTaskQueueUserDataReplicationEventRevivalUsesMaxClock -count=1 - make lint-code-fast <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes multi-cluster replication merge semantics for task queue user data clocks; incorrect behavior could cause stale writes or conflict resolution bugs, but scope is narrow and covered by new tests. > > **Overview** > Fixes **hybrid logical clock (HLC) handling** when applying replicated task queue user data in matching: if the incoming event wins last-writer conflict resolution (V2 assignment/redirect rules and V3 `PerType`), the merged result now **copies the incoming clock** via `common.CloneProto` instead of leaving a stale or nil clock on persisted user data. > > Adds regression tests that **incoming clock wins** when it should (with a cloned, mutation-safe copy), **current clock is kept** when it is newer, and **build ID revival** still advances the merged clock with `hlc.Max` against the revival timestamp. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1ab908b. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
…io#12334) ## What changed? `replication_stream_stuck` now sets its `from_cluster` / `to_cluster` tags to cluster **names** instead of numeric cluster IDs (initial failover versions). The IDs are resolved via `ClusterMetadata.ClusterNameForFailoverVersion`, only on the stuck path. This adds `metrics.FromClusterTag` / `metrics.ToClusterTag` string helpers that use the same tag keys as the existing `FromClusterIDTag` / `ToClusterIDTag`. Other metrics that use the ID tags are unchanged. ## Why? Alerts built on this metric showed labels like `from_cluster=102402, to_cluster=102437`, so on-call had to decode the IDs before they knew which cluster pair was stuck. With names, the alert identifies the pair directly. Alert expressions that group by these labels keep working without changes. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) Ran `go test -tags test_dep ./service/history/replication/... ./common/metrics/...` and `go vet`. New test `TestEvaluateStreamStatus_StuckMetricUsesClusterNames` checks that the tag values are names. ## Potential risks - Any consumer that filters `replication_stream_stuck` on specific numeric `from_cluster` / `to_cluster` values will stop matching. As far as I can tell, the only internal consumer is an alert that groups by these labels, which is unaffected. - While a cell is being upgraded, the series label values change, so alerts that compare consecutive windows may fire up to one window late for that cell. - If an ID doesn't resolve to a known cluster, `ClusterNameForFailoverVersion` logs a warning and returns an `unknown-cluster-<id>` name. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary - report that a Worker Deployment Version is missing only for an actual NotFound error - the way errors that are not found are being surfaced right now, from the deployment workflow, are quite confusing because they always start with the message: "version not found in deployment, <actual_error>". This is confusing. - describe throttling, timeout, and other lookup failures as DescribeVersion failures during task-queue validation - cover both message paths with a focused table-driven test ## Why IsVersionMissingTaskQueues currently prefixes every DescribeVersion failure with "version not found in deployment". This makes operational failures such as BusyWorkflow and namespace throttling look like missing-version failures even when the version exists. The change preserves the existing FailedPrecondition type and retry behavior while making the message truthful. For example: <img width="542" height="168" alt="image" src="https://github.com/user-attachments/assets/9610fc20-4065-46e8-9ec4-fdb8adf3063c" /> ## Testing - GOWORK=off go test -tags test_dep ./service/worker/workerdeployment -run ^TestDescribeVersionFailedPrecondition$ -count=1 - golangci-lint v2.13.0 built with Go 1.27: 0 issues - scoped errortype go vet: pass <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Behavior change is limited to error message text for the same FailedPrecondition surface; no change to retry logic or missing-version detection logic. > > **Overview** > **`IsVersionMissingTaskQueues`** no longer labels every `DescribeVersion` failure as a missing deployment version. Failures are mapped through **`describeVersionFailedPrecondition`**, which keeps the existing *version not found* `FailedPrecondition` only when the underlying error is **`NotFound`**, and otherwise returns a message that the describe step failed while checking missing task queues (e.g. throttling / busy workflow). > > Table-driven tests in **`util_test.go`** lock in both message paths. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 628f31c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
…anch (temporalio#12214) ## What changed? When replication resolves a conflict by forking a new branch (or switching to another existing branch), it seeded the first node's `prev_txn_id` from the tip of the branch it was moving off. Now it reads the tip of the branch it's actually appending to. ## Why? The bad link breaks the reverse chain, so reverse reads fail with DataLoss after failovers with replication lag. Forward reads are fine. We see this in Cloud. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) New unit tests for the fork and switch cases. I also have an end-to-end repro in saas-temporal that fails without this change and passes with it. `tests/xdc` passes. ## Potential risks On a branch change this reads the target branch from the first event, because forward reads can't start in the middle of a node. Branch changes are rare, and `StateRebuilder` already does the same read. That may be expensive but since this happens rarely - I don't expect it to be a problem. On my machine a 4K-node history read took about 600 ms (but this was just a seat-of-pants check - not a performance evaluation).
…alio#12337) ## What changed? Adds a `replication_dlq_dropped` counter, tagged with `source_cluster`. It is emitted when `ExecutableTaskImpl.MarkPoisonPill` hits `MarkPoisonPillMaxAttempts` and discards the replication task instead of writing it to the DLQ. ## Why? Today a replication task that can't be written to the DLQ is dropped silently from a metrics point of view. The only signals are the `MarkPoisonPill reached max attempts` error log and a wide event that depends on dynamic config. `replication_dlq_enqueue_failed` doesn't fill the gap: it counts every failed attempt, including ones where a later retry succeeds, so it can't tell us how many tasks were actually lost. A dedicated counter makes it possible to alert on lost replication tasks. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) `TestMarkPoisonPill_MaxAttemptsReached` now checks that the counter is not emitted on a failed attempt that will be retried, and is emitted exactly once, with only the `source_cluster` tag, when the task is dropped. Ran `go test -tags test_dep ./service/history/replication/... ./common/metrics/...` and `go vet`. ## Potential risks None expected: this only adds a low-cardinality counter on an already-rare error path. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## What changed? Nexus log messages that embedded the method now use one static message per condition; the method stays in the `operation` tag. ## Why? Interpolating the method into the message split one condition into several log lines when aggregating. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…12357) ## What changed? Inject a destination blocked function for chasm nexus operations, check it in the describe. ## Why? describe nexus operations should include when the operation is blocked by the circuit breaker ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s)
…alio#12321) ## What changed? Regular poller saturation can consume the concurrency slots needed by per-namespace system workers. Internal workflow, activity, and Nexus polls now use separate frontend counters and quotas, per namespace and API method. Sticky workflow polls use `TaskQueue.NormalName` for classification. Add `frontend.internalPerNSNamespaceCount` (default `1200` per frontend instance) and `frontend.globalInternalPerNSNamespaceCount` (default `0`, disabled). Pending-request metrics distinguish the two budgets with `concurrency_limit_group="default"|"internal_per_ns"`. Both groups are counted per namespace and API method; the tag identifies the budget rather than the source of a configuration override. `service_errors_resource_exhausted` also includes this label: `default` or `internal_per_ns` for this frontend concurrency limiter's rejections, and `not_applicable` for other exhaustion errors. Existing cause and scope labels are preserved, and all recordings carry the group label for Tally/Prometheus compatibility. ## Why? The incident exhausted namespace concurrency limits while RPS remained within its limits. Keep per-namespace system workers polling when regular concurrency slots are saturated. ## How did you test it? - [x] Covered by existing tests - [x] Added new unit tests - [x] Added new functional tests Complete metrics, interceptor, dynamicconfig, frontend, and history replication unit suites passed with race detection and shuffled execution. The full namespace interceptor functional suite passed with SQL/SQLite. `make lint-code-fast` passed with automatic fixes disabled. Regressions cover namespace/method isolation, sticky classification, independent local/global quotas, and quota cleanup. Temporary classification and shared-budget mutations failed the expected unit and functional cases. Metric regressions directly test five resource-exhausted recording cases, covering both concurrency groups and unrelated exhaustion attribution, with exactly one counter recording and preserved inherited, cause, and scope tags.
## What changed? This changes the encoding to the newer and not-buggy callback. This probably should always have been true, but was set to false and in production we turned it manually on everywhere. ## Why? This fixes a bug in CHASM callbacks, in which schedules didn't have enough information if the workflow did a CAN ## How did you test it? - [ ] built - [ ] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks This is for a feature not expected to be enabled for OSS users yet, so it should be safe, however it's a nontrivial default change, so warrants a little thought. That said, the actual codepath is well tested in our production env.
## What changed? - Added `ScheduleCHASMSuite` and its `parallelsuite.Run` entry point. - Migrated only `TestScheduleNextActionTimeVisibility` into the suite. - Kept the schedule environment helpers and the test body unchanged. ## Why? This is the bootstrap layer for parallelizing schedule functional tests. Keeping it to one representative CHASM test makes the suite wiring independently reviewable before the V1, CHASM, and shared migrations above it. ## How did you test it? - [x] built - [x] covered by existing functional tests Commands: - `go test -tags test_dep ./tests -run '^TestScheduleCHASMSuite/TestScheduleNextActionTimeVisibility$' -count=1 -timeout=10m` - `make fmt-imports` - `make lint-code-fast` - `git diff --check` ## Stack 1. temporalio#11102 (this PR) — bootstrap parallel CHASM suite 2. temporalio#12065 — migrate V1-only schedule tests 3. temporalio#12066 — migrate CHASM-only schedule tests 4. temporalio#12067 — migrate shared V1/CHASM schedule tests ## Potential risks Test-only change. The main risk is altered test execution ordering; this layer intentionally migrates only one isolated CHASM visibility test. Co-authored-by: chaptersix <13949480+chaptersix@users.noreply.github.com>
## What changed? Added tests for Query links ## Why? Improve coverage for Query-backed Nexus ops ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) ## Potential risks
## What changed? Reduced the activity start-to-close timeout to 5 seconds in `TestActivityBatchReset_RunningWorkflowsResetAttempts`, and wait for all activities to return to `SCHEDULED` after stopping the old worker before resetting them. ## Why? An old worker can claim an activity immediately before stopping. The former 15-minute start-to-close timeout left that activity `STARTED`, so the replacement worker could not retry it during the test. Waiting for it to time out ensures reset operates on a scheduled activity and keeps the reset-attempt assertion race-free. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 of 3 tasks
Automatically generated by the optimize-test-sharding workflow. Co-authored-by: Temporal Data <commander-data@temporal.io>
## What changed? Add a composite action that retries artifact uploads twice with 5-second and 15-second backoff. ## Why? GitHub intermittently returns `403 Forbidden` while finalizing artifact uploads, and `actions/upload-artifact` classifies that response as non-retryable. In [run 34251956559](https://github.com/temporalio/temporal/actions/runs/34251956559), this made three otherwise-successful functional test jobs fail during post-test reporting. There's an open issue on the action: actions/upload-artifact#560
## What changed? Added generic `Rcv`/`Snd` methods to `parallelsuite` and migrated all calls. ## Why? More ergonomic/shorter test helper. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed? Migrated activity reset suite to parallelsuite. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed? Makes `FunctionalTestBase` private. ## Why? The reset-suite migration removed the last external caller. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed? - `SignalWithStart` now honors the first workflow task backoff for workflow-initiated continue-as-new runs. The signal is recorded immediately, but the workflow task is not scheduled until the configured backoff expires. - Existing behavior is preserved for initial workflow start delays, retry backoffs, cron backoffs, and expired backoffs: `SignalWithStart` continues to schedule a workflow task immediately. - The workflow start event is loaded only when the run is still before its execution time and pending on workflow task backoff. - The existing skip-delay metric is emitted only when a delay is actually bypassed, and throttled logs distinguish skipped delays from honored continue-as-new backoffs. - Added `tdbg execution force-schedule-first-workflow-task`, which connects directly to a history host and calls `ScheduleWorkflowTask` with `is_first_workflow_task=true` as an operator mitigation. - Updated the `tdbg` client factory to support history-service connections and derive TLS host verification from the selected target address. ## Why? A signal sent through `SignalWithStart` during a workflow-initiated continue-as-new backoff could schedule the first workflow task immediately, bypassing the backoff requested by the workflow. This change keeps the signal durable while preserving that backoff. Retry and cron backoffs retain the existing signal-driven wake-up behavior. The new `tdbg` command provides an explicit operator recovery path when the first workflow task must be forced onto a specific history host. ## How did you test it? - [x] Built successfully - [x] Added unit coverage for execution-time boundaries, workflow-initiated continue-as-new, retry and cron initiators, overdue backoffs, and skip-delay metrics - [x] Added `tdbg` command coverage for successful requests, RPC failures, and required flags - [ ] Run locally and tested manually ## Potential risks - Signals delivered during a workflow-initiated continue-as-new backoff will not be processed by workflow code until the backoff expires. This is intentional and restores the workflow-requested behavior. - The new `tdbg` command bypasses first-workflow-task timing and connects directly to a history host. It is an operator-only mitigation requiring the namespace ID, workflow ID, and target history address.
## Problem When a schedule appears not to have fired, visibility shows what ran but does not explain what should have run or whether an overlap policy legitimately suppressed or delayed an action. Investigating this across a namespace currently requires reconstructing each schedule's expected fire times and correlating them with workflow executions by hand. ## What this adds Adds `tdbg schedule audit`, which reconstructs nominal and jittered fire times for a bounded window using the server's schedule spec compiler, correlates them with workflows carrying `TemporalScheduledById` and `TemporalScheduledStartTime`, and emits one self-contained JSONL row per flagged schedule. Started actions are classified as: - `on_time`: eligible and started below the effective late threshold (1 minute by default) - `late`: dispatch delay reached the effective late threshold - `delayed_overlap`: started after waiting for earlier actions under a delaying overlap policy Unmatched fire times are classified as: - `real_miss`: no workflow was observed and the overlap policy does not explain its absence - `skip_overlap`: an overlap policy discarded the fire, including older queued fires superseded by `CancelOther` or `TerminateOther` - `pending`: a buffering, canceling, or terminating policy is still blocked by a running workflow - `inconclusive_overlap`: leading-edge `BufferOne` occupancy cannot be reconstructed from visibility - `paused`: the schedule was already paused before the window when `--include-paused` is requested Schedule-level `audit_status` is `complete` or `inconclusive_schedule_changed`. Changed schedules remain visible even when the current spec has no fires in the window. Jittered fire times use the modern V1/CHASM seed, `<namespace UUID>-<schedule ID>`. Historical V1 actions before `NewCacheAndJitter` used an empty seed and are outside this reconstruction contract. Each row includes nominal/jittered scheduled-time pairs, the effective delay threshold, observed executions, per-category counts, and delay decomposition (`jitter_offset`, `overlap_wait`, `dispatch_delay`, and end-to-end delay). ## API usage and scalability Input targets and output rows stream with bounded memory. Whole-namespace audits: - page through `ListSchedules` - issue one `DescribeSchedule` per schedule - skip visibility for paused, exhausted, no-fire, or over-100k-fire schedules - combine up to 25 eligible schedule IDs into each paginated `ListWorkflowExecutions` query For example, 25 schedules with five workflows each require one visibility request when the 125 rows fit on one page, rather than 25 requests. Describe calls use `--rps`; list and visibility calls use the separate `--list-rps` limiter. Four schedule batches per namespace may run concurrently, independent of the rate limits. Transient service errors use the server's standard retry predicate and backoff policy. Targets can be supplied directly with `--namespace` / `--schedule-id` or streamed as JSONL through `--file` or stdin. Window bounds accept durations (`--lookback-start 24h`) or RFC3339 timestamps (`--start-time ...`); future end bounds are rejected. ## Boundaries - The window start is clamped to namespace retention to avoid treating purged visibility records as misses. - Currently paused and exhausted schedules are skipped by default. - Schedule changes produce schedule-level inconclusive results instead of unreliable per-fire verdicts. - Fire-time reconstruction targets modern V1 and CHASM jitter semantics. - Visibility cannot prove whether a pre-window `BufferOne` slot was occupied, so affected leading-edge fires are marked `inconclusive_overlap`. ## How did you test this - Target parsing rejects unknown fields so a misspelled schedule ID cannot silently expand the audit to a namespace. Command output tests verify configured writers and quiet mode. - Visibility query coverage parses generated queries and checks that IDs containing quotes, backslashes, SQL-like text, and Unicode retain their literal values. Batched/paginated loading includes 25 schedules with five workflows each. - V1 and CHASM functional coverage checks a normal scheduled action, a pre-window running blocker, and a buffered action whose nominal time is in the window but whose actual start follows the window end. - Classification coverage checks jittered window boundaries, modern jitter seeds, buffering and replacement overlap policies, chained BufferAll delay decomposition, trailing-edge grace, and changed/paused/retention-filtered schedules. Loader coverage checks transient retries and rate limiting on every attempt; dispatcher coverage checks bounded batching and failure isolation. Carries forward the original work from temporalio#10376 by @fretz12. --------- Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io> Co-authored-by: David Porter <contact@davidporter.id.au>
## What changed and why? 1. run admin batch in temporal-system so that we can refresh tasks in passive clusters 2. delegate maintenance ops of user batch to admin batch ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s)
…lio#12381) ## What Add a `SupportsVersioning()` check in `ensureRegisteredInDeploymentVersion` to skip deployment registration for worker commands task queue. ## Why This is to handle a SDK bug where version deployment was incorrectly attached to poll requests on worker command task queue. This is a follow up to a previous fix we did earlier, but it was not complete. Noticed this in ring0. Details: [temporalio#11227](temporalio#11227) fixed the partition manager's poll routing to skip `getVersionedQueue()` for worker commands partitions using `SupportsVersioning()`. However, this introduced a regression: the poll now falls through to the default physical queue, whose `PollTask` unconditionally calls `ensureRegisteredInDeploymentVersion`, registering the worker commands TQ with the deployment. This hits the 100 TQ per deployment limit with `ResourceExhausted`. Before temporalio#11227, `getVersionedQueue` had its own `SupportsVersioning()` guard (line 2155) that rejected the call, so the registration path was never reached. After temporalio#11227, the poll correctly skips versioned routing but still triggers registration on the default queue. Turns out this behavior is true for sticky queue as well. But it doesn't create a problem because the registration uses the task queue family. Since sticky queues share the tq family name, they deduplicate to a single entry. Worker commands queues have unique family names (including namespace and worker instance key), so each one registers as a distinct TQ in the deployment, quickly exhausting the limit. ## How did you test it? Functional test (`TestPollWorkerCommandsWithDeploymentOptions`): enables deployment versions with worker service, registers a deployment via a normal TQ poll, polls a worker commands TQ with the same deployment options, verifies cancel command delivery still works, and asserts via `DescribeWorkerDeploymentVersion` that the worker commands TQ is not in the deployment's task queue list. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…alio#12292) When an existing workflow is loaded for the first time in CHASM, the Component.InitialVersionedTransition has a NamespaceFailoverVersion of 0. This comes from the currentVersion number being empty/0. When the new tree is built, tree.initSerializedNode sets the failover version to the currentVersion. Change: When creating a ref for the root object, set the initalVersion to nil instead of an incorrect number ## What changed? - Ref.forConsistencyLevel(RefConsistencyLevelComponentCreation) sets executionLastUpdateVT=nil - tree.Component -> root components now don't return error will componentInitialVT == nil - tree.structuredRef/invokeSideEffectTaskFn returns componentInitialVT == nil for roots - tree.closeTransactionSerializeNodes root components on first save get their InitialVersionedTransition set - FF flag EnableCHASMWorkflowRootOnStart - mutable_state_impl.ApplyWorkflowExecutionStartedEvent. -> when FF is on, mark new workflows dirty for first time save ## Why? CHASM workflows being loading don't have their initialVersionTransition.NamespaceFailoverVersion set correctly. The value is being taken from currentVersion during initial component creation. - Because there are bad values saved, we now ignore the roots initialVersionTransition in refs. Don't use these values - When root node is created for existing workflows, make closeTransactionSerializeNodes save a proper value - Mark workflow roots as dirty on first load (behind FF EnableCHASMWorkflowRootOnStart) ## How did you test it? - [X] built - [ ] run locally and tested manually - [ ] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) ## Potential risks
…12313) The async resend could not report a source NotFound back to the standby task, which then retried until its discard delay. Record it on the resend scheduler so the next retry finishes the way the inline path does. ## What changed? Cache the parent’s source `NotFound` on the resend scheduler so the next standby retry can finish. ## Why? Match the inline path and avoid retrying until the discard delay. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks - Stale source NotFound could cause premature ACK. Entries are checked against the current source cluster and expire after 10 minutes. - Cache eviction or restart may delay ACK; tasks fall back to existing retries and discard handling. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
EnsureChasmWorkflowComponent is unused. The check for UnspecifiedArchetypeID is never true. Both NewTreeFromDB and NewEmptyTree set the archeType ## What changed? Remove dead code because the Archetype is always set. ## Why? Code is not reachable. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks If we missed any code paths where EnsureChasmWorkflowComponent is used, but based on static analysis by claude, this is unlikely.
## What changed? Link `AGENTS.md` to the shared code guidelines. ## Why? Reduce duplicated and generic guidance while keeping one source of truth for shared coding standards.
…ified during earlier bug hunt (temporalio#12077) ## What changed? Following bug fixes across `service/worker/batcher` and the batch APIs in `service/frontend`. - **Batch terminate/cancel now call the frontend client directly.** Previously they went through the SDK client, which substituted the worker's own `pid@host` for the requester's identity, had no way to send terminate `details`, and used a fresh cancel request ID on every attempt, e.g., cancel retry recorded a second cancel-requested event. On the other hand, the frontend calls forward identity, details, reason, and a deterministic cancel request ID. This also allowed removing the SDK client in the task-processing path from `batchWorkerProcessor` → `startTaskProcessor` → `processTaskWithRetries` → `processSingleTask`. It still remains used for the visibility list/count. - **Batch signal forwards the operation's `Header`.** `BatchOperationSignal.Header` was previously not copied to `SignalWorkflowExecution`. This silently dropped tracing context and any auth tokens attached by the caller before they reached the `WorkflowExecutionSignaled` event. - **`StopBatchOperation` verifies that the job ID is a batch job.** Previously, it turned the caller-supplied job ID into a workflow ID and terminated it in process, skipping the authorization check that a `TerminateWorkflowExecution` API call does — so the API could terminate any workflow in the namespace whose ID is known. It now requires a batcher workflow type, a batcher namespace division, and the internal per-namespace worker task queue. It rejects anything else with `InvalidArgument`. Since, the division and type are caller settable, using the task queue provides an extra guard. It also pins the terminate to the validated run, so a same-ID run started in between cannot be terminated in its place. - **`DescribeBatchOperation` verifies the job ID too.** It had the same gap, which allowed a caller-supplied job ID to be used as a generic workflow ID, while skipping the authorization check the real API performs. The non-batch workflow came back with empty reason/identity, but the real `StartTime`, `CloseTime`, and `State`. Both the APIs now share one validated function, also removing the second describe call this path was making earlier. - **An explicit `AttemptsOnRetryableError` of 1 is honored.** Previously, `setDefaultParams` treated `<= 1` as unset. This silently raised a requested single retry to the default of 50. The fix is to use `< 1` so that value of 1 is honored, instead of treating it as default. - **Batch activity options are built per execution.** Previously, the `BatchWorkflowProtobuf` wrote the heartbeat timeout into a package-level `ActivityOptions` before passing it to `workflow.WithActivityOptions`. This caused a race condition - concurrent batch workflows on a worker used the same value, and one execution could schedule its activity with another's timeout. The race was already fixed in PR 12045, but here, new unit tests are added to cover per-call independence, and the extraction of options into `batchActivityOptions`. - **The heartbeat frequency derives from the activity's heartbeat timeout.** Previously, this was hardcoded to `defaultActivityHeartBeatTimeout / 4`, so any configured `ActivityHeartbeatTimeout` shorter than the default would timeout the activity mid-batch while it was making progress. - **An `InvalidArgument` from a target's operation is no longer retried.** It was classified retryable, so a permanently failing target used up the whole `AttemptsOnRetryableError` budget, i.e., 51 calls by default, paced only by the batch's rate limiter, and logged the identical failure each time to reach the identical outcome. It is now non-retryable for every batch type, matching how `fetchPage` already treats an `InvalidArgument` from the visibility query. - **The options-update `UpdateMask` is forwarded unchanged.** Both the workflow- and activity-options paths dereferenced `UpdateMask.Paths` directly and unguarded, on a goroutine, so a request without a mask took down the worker host rather than failing the batch. Using a getter fixes the panic error but converts nil mask to a non-nil but empty one. Now the nil mask is passed to the server, which rejects it as `InvalidArgument`. `ValidateBatchOperation` requires a mask only on the workflow-options path, so the activity-options path was reachable from a plain `StartBatchOperation`. ## Why? These bugs were identified during a bug hunt earlier. In these cases, a batch operation quietly did something unexpected or not requested for, e.g., dropped or ignored a field that the caller set, retried something that could never succeed, or crashed the worker host. The `StopBatchOperation` gap let one batch API act as a general-purpose terminate for any workflow in the namespace, without any authorization check of the similar API calls. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Each fix is covered by unit tests, and was confirmed to fail against the code without the fix, e.g., the nil-`UpdateMask` panic, the 51 retry attempts on `InvalidArgument`, the activity-options data race under `-race`, and the missing heartbeat inside a short timeout. New functional tests assert that the terminate/cancel/signal history events carry the requester's identity, reason, details and header, and that `StopBatchOperation` refuses to stop a non-batch workflow ID, and actually stops a real throttled batch job. ``` go test -race -count=1 ./service/worker/batcher/... ./service/frontend/ go test -tags disable_grpc_modules,test_dep -count=1 ./tests/ -run 'TestWorkflowAPIBatch|TestActivityAPIBatch|TestActivityApiBatch' ``` ## Potential risks These user-visible behaviors are changed in one form or the other in the fixes above. - Terminate/cancel history events now record the requester's identity instead of the batcher worker's pid@host, and are empty when the caller set no identity. Cancel events gain the job's reason as their cause, and terminate events carry the request's details. - Signaled events now carry the caller's header. - StopBatchOperation returns InvalidArgument for a job ID that names a non-batch workflow, where it previously terminated it. Any caller relying on that to stop arbitrary workflows will break, which is intended. - A batch task failing with InvalidArgument now fails immediately instead of after 51 attempts. A batch that previously appeared to succeed because a later retry happened to pass would now be reported as failed. - `DescribeBatchOperation` returns `InvalidArgument` for a job ID naming a non-batch workflow, where it previously returned that workflow's state and timing. - An `options-update` batch submitted without an `UpdateMask` now fails its targets with `InvalidArgument`. Previously, this crashed the worker host. - `errBatchJobIDNotValid`'s message changed from "JobId is not valid." to "JobId does not refer to a batch operation."
## What changed? Add request IDs to existing HSM and CHASM outbound callback failure and HTTP trace logs. ## Why? Correlate failed completion delivery with the Nexus request. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Prevent inbound replication from modifying local namespaces. Test: `go test ./common/namespace/nsreplication ./service/history/replication`
…oralio#12112) Stack: PR0 ## Summary - extract the namespace replication decision into `ShouldReplicateNamespace` - extract legacy persistence-to-detail assembly into `NamespaceDetailFromTransmissionTask` - extract detail-to-wire conversion into `NamespaceDetailToTaskAttributes` - establish shared helpers for reuse by the stacked CHASM namespace-replication implementation - preserve existing legacy replication gating, payload, queue publication, and lifecycle-event behavior ## Tests - `go test -race -count=1 -tags test_dep ./common/namespace/nsreplication` - `make fmt-imports` - `make lint-code-fast` - full CI and cross-repository compatibility checks ## Operational impact No intended behavior change. The existing legacy queue path continues to make the same replication decisions and emit the same wire payload; this PR only centralizes that behavior for reuse by later stack layers. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What Fix data races in `perNamespaceWorker` initialization and refresh: - Hold `worker.lock` while registering dynamic-config callbacks and initializing `ns`, `count`, and `opts`. - Build the worker cache key from `w.opts` while holding the lock, matching the options used by `startWorker`. ## Why Without the lock, a dynamic config callback can fire on another goroutine before `ns` is set, leading to a nil dereference panic in `refresh`. ## How did you test it? Unit tests (`go test -race ./service/worker/`). --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Set EnableCHASMWorkflowRootOnStart to default to true. Edit test which did not flip EnableCHASM to false when running under 'false' ## What changed? Enable write for CHASM workflows root. Otherwise, it will wait until there is an action. Test update: The test did not properly CHASM disabled because the default is now true. It was relying on old behavor. ## Why? Create the new CHASM object on existing workflows so they can have a proper initialVersionTransition value. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks There might be extra writes, but we have a FF to disable this.
## What changed? Use short, deterministic task queue, deployment, and build identifiers in `TestPollWorkerCommandsWithDeploymentOptions`. ## Why? The generated identifiers inherited the full test name. Once encoded into the versioned physical task queue key, they produced a 318-byte `task_queue_id`, exceeding MySQL’s 272-byte column limit. This caused deployment registration to fail and the test to time out on the subsequent describe call. This fixes the failure from https://github.com/temporalio/temporal/actions/runs/37529324205/job/112495638069. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s)
## What changed? Remove `addsearchattributes` workflow. ## Why? Since admin handler is forwarding search attributes API to operator handler (temporalio#10747), this workflow is no longer necessary. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks
Timers with equal deadlines are grouped into one state machine timer, and the wall clock on macOS does not always tick between two nodes of one refresh walk, so the task refresher test counted two timer groups where it expected three.
moetemp
force-pushed
the
moe/hsmtest-distinct-timer-deadlines
branch
from
October 7, 2026 02:33
3137ba4 to
4925e20
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR gives each regenerated test timer a distinct deadline.
What changed?
service/history/hsm/hsmtest/sm.go:RegenerateTaskspushes every timer deadline out by one more nanosecond than the last one it handed out, through a package-level counter, instead of usingtime.Now().Add(time.Hour)for all of them.Part of AI-198 (epic AI-37).
Why?
Timers with equal deadlines are grouped into one state machine timer. The task refresher test regenerates the timers of several nodes in one walk and expects one timer group per node. On macOS the wall clock has microsecond resolution, so two
time.Now()calls inside one walk can return the same value, two nodes then share a deadline, andTestRefreshSubStateMachineTaskscounts two groups where it expects three. The failure is intermittent and only shows on a fast machine. The counter makes the deadlines distinct without depending on the clock.How did you test it?
Link to a test plan if any -
Ran
TestTaskRefresherSuite/TestRefreshSubStateMachineTasksin a loop on macOS on this branch, all passing. The same loop on the base helper fails intermittently with "should have 3 item(s), but has 2", with two deadlines a few microseconds apart and the third equal to one of them.