Add: TMR kernel-mode execution coordination and teardown（K7） - #2199
Draft
Leaf-Salix wants to merge 17 commits into
Draft
Add: TMR kernel-mode execution coordination and teardown（K7）#2199Leaf-Salix wants to merge 17 commits into
Leaf-Salix wants to merge 17 commits into
Conversation
Kernel mode is simpler's second execution identity: instead of owning
the device, a context borrows the caller's already-current device and
stream to enqueue one bounded asynchronous operator per launch, so a
PyPTO program is capturable by ACLGraph as an ordinary node. This
change freezes the public surface that identity hangs off and gives the
context a write-once identity the guards can key on. It creates no
resources. The program path gains one call — simpler_init latches
PROGRAM — and no behavior: latching a fresh context always succeeds, is
idempotent, and nothing on that path reads the latch.
- runtime_c_api.h declares the lifecycle entries
simpler_kernel_mode_{supported,init,prepare_callable,launch} and adds
the host-band code PTO_RUNTIME_ERR_INVALID_STATE. The existing
finalize_device stays the fifth lifecycle entry, and a kernel context
now reaches it. Kernel-mode capacity is
a mode invariant rather than a gated state: config is context-static,
so each pooled arena region is committed at most once, and
setup_static_arena reports a grow or release request on a committed
region under kernel mode as an internal invariant break; capacity
intent travels in CallConfig.runtime_env like everywhere else.
- Execution identity is a write-once property of the context rather
than a state that evolves. ExecutionModeLatch (platform/include/host/
execution_mode_latch.h) replaces the four-state claim: the first init
entry to run latches the mode, and it never changes — not on finalize,
not on error. simpler_init latches PROGRAM before touching any
process or runner state, so the program/kernel mutual exclusion is
enforced on every program init instead of resting on a separate
declaration call. There is no unlatch, which the latch documents as a
consequence: a handle from a failed kernel init can never be recycled
into a program context. SimplerExecutionMode now has one definition
(task_interface/execution_mode.h) that both the wire header and the
latch consume, so the host-side identity and the value that travels to
the AICPU can no longer disagree.
- device_id_ records which device a context is on, not a claim on it —
ownership is what the latch carries. attach_current_thread splits
accordingly: bind_current_thread does the per-thread rtSetDevice and
nothing else; attach_current_thread is the program-mode adopt
(bind plus the one-shot op-execute watchdog and identity write) and
refuses on a kernel latch; adopt_borrowed_device records the device a
kernel context runs on without binding the thread and without
configure_aicore_op_timeout, whose aclrtSetOpExecuteTimeOutV2 would
rewrite the watchdog for every other user of a borrowed card. It does
resolve the timeout config, because the stream and scheduler timeouts
derived from it are read on both identities. DeviceRunner::finalize()
is the one caller that runs under both identities and skips the bind
on a kernel latch, so the kernel close path reaches its no-reset
branch instead of being turned away by a device bind it never needed.
- ensure_acl_ready(), force_reset_device(), and finalize()'s rt-layer
device reset refuse on a kernel-mode context (a2a3 + a5): the ACL
lifecycle belongs to the caller, and every call site of the five ACL
lifecycle APIs falls into three enumerable classes (below the
ensure_acl_ready guard, inside force_reset_device behind its own
guard, or gated on acl_ready_ which only the guarded path sets), with
finalize's rt-layer reset intercepted by its own kernel-mode branch —
so poison recovery can never reset the device out from under the
host process.
- kernel_invocation_header.h pins the envelope every kernel launch
ships to the AICPU (mode / callable / generation / payload length /
int32_t arg counts). Both sides of the wire come from one
build_runtimes.py build, so the struct carries no version or size
negotiation and the POD/standard-layout guards are its only
compile-time checks. generation is the occupancy counter of the
residency slot callable_id resolves to - a property of the slot, not
of the callable in it, so a generation carried by the callable could
not detect slot reuse - with zero reserved for "not recorded".
ChipCallable's sig_count includes the scalar entries and its
scalar_count reads 0 both for a scalar-free orchestration and for an
artifact built before the field existed, so a consumer derives the
effective scalar count - the field when nonzero, otherwise the
signature's SCALAR entries, the split count_callable_tensor_args
already computes - and compares tensor_count against sig_count minus
it. Subtracting the field directly would count an unrecorded
callable's scalars as tensors.
- Kernel-entry argument validation is shared by all eight host-runtime
components through kernel_entry_validation.h (one copy of the
null/range/image-size/alignment checks; a binary pointer and its size
must be present or absent together, and a callable image must be
aligned for ChipCallable so its CALLABLE_CHILD_ALIGN-relative storage_
lands aligned too), so a stub and a real implementation accept and
reject exactly the same arguments.
- KernelExecutionState and ExecutionModeClaimState carry the kernel
context phase machine (New/Collecting/ReadyEnqueued/Poisoned/
Closing/Closed with sticky, retriable Closing and separate
runtime-error and teardown-error slots) and the two restricted
operation vocabularies; synchronize, allocation, capture queries,
and model attachment stay unrepresentable in those tables, and a
launch implementation is obligated to route through them. Every
kernel-mode guard reads the identity through ExecutionModeLatch::
is_kernel() rather than comparing an enumerator at the call site, so
the test lives in one place instead of eight.
- ChipWorker dlsyms the four new symbols from every runtime, so a
component missing one fails at load, and clears them alongside the
other resolved pointers on all three teardown paths so none is left
dangling into the library DlHandleGuard dlcloses.
test_host_runtime_abi.py
asserts the export across all eight components, and table-driven UTs
cover the phase machine (including failed-rollback landing in
Closing with the create error reported and the cleanup error
latched), the shared argument validation, and the wire layout.
- ChipCallable additionally records scalar_count as a cached
derivation of the signature's SCALAR entries: make_callable rejects
a nonzero count that disagrees with the signature, while 0 also
means "not recorded" (legacy blobs read 0). The field occupies four
bytes of historical header tail padding, so every historical offset,
sizeof, and the kernel-cache ABI token are unchanged;
ChipCallable.build gains a trailing scalar_count=0 keyword and a
read-only property.
Two facts a reader should not have to re-derive. The latch refusal returns
PTO_RUNTIME_ERR_INVALID_STATE (-1003) rather than PTO_RUNTIME_ERR_INTERNAL
(-1000) on purpose: conftest.py scrapes "simpler_init failed with code <N>"
and treats -1000 as a poisoned card, so an identity conflict must not look
like one. And kernel_execution_state.cpp stays compiled into all four host
runtimes even though grepping KernelExecutionState now finds only its own
header and .cpp — it is the persistent-state change's foundation, not an
orphaned translation unit.
Every kernel-mode branch this adds is provably dead in this commit: no
production site latches KERNEL (`git grep 'latch(SIMPLER_MODE_KERNEL)' src`
is empty) because both simpler_kernel_mode_init stubs return before any latch
call, so is_kernel() is false on every context and the program path takes the
same branch it took before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Compute TMR kernel arena requirements from the existing per-architecture sizing and reserve-only layout without acquiring device resources. Keep candidates call-local and publish output only on success. Share mode-aware contract and topology validation while keeping the TMR resource set explicit. Preserve valid program behavior and reject unserviceable stream declarations before context creation. Define execution modes in a neutral header shared with the unchanged invocation envelope. Connect the internal builder to validating kernel init stubs; HBG remains unsupported and kernel execution stays disabled. Cover sizing bounds, packed input, independent concurrent calls, real loader admission, sim entry behavior, and C/C++ header compatibility. Relevant tests and sim partitions pass. Full CTest retains the existing profiler quiesce failure (139/140); onboard validation remains pending.
Use the upstream execution-mode header as the single definition. Keep resource contracts independent of kernel lifecycle and invocation headers, and verify that boundary through compiler dependencies. Leave invocation layout ownership and its original wire tests with K1; remove the resource-contract test that additionally pins wire offsets. TMR sizing, resource validation, and init preflight remain unchanged.
Separate common invocation admission from the TMR wire codec. Encode independently owned packets with transactional host templates; consume trusted callable and execution-binding views without owning runtime resources or changing program-mode execution. Add bounded decoding, caller-owned argument conversion, unit tests, and an isolated CPU transport snapshot probe. Production kernel launch remains disabled until resource providers and execution are integrated.
Integrate persistent context resources from K2 commit 48b120b, including its four-argument prepare ABI and borrowed-device lifecycle. Preserve resource-contract admission before context mutation. Validate callable signatures and canonical storage before registration. Provide runner-backed CPU transport and trusted-view TMR consumption without enabling public launch or fabricating resource providers. Cover prepare rejection, consumption transactions and bounded templates. Fix shared buffer-pool lint without changing release-once semantics. Co-authored-by: YunjiQin <a1339924773@gmail.com>
Fix persistent argument ownership across partial preparation and failed release, including retryable kernel topology query cleanup. Copy context configuration at init and resolve topology before publishing resident arguments. Consume invocation arguments and callable tables explicitly in the existing executor and scheduler while preserving the program data sources and public layouts. Add bounded provider views, failure-reuse tests, and lifecycle coverage. Keep public kernel launch disabled pending resource providers and the execution binder. Targeted tests, sanitizers, program sim regression, and incremental lint pass within the documented validation scope.
Exercise the production context ops and AICPU loader with a native capture/replay probe. Verify fresh inputs across 100 replays and reject forbidden capture APIs. Source: hw-native-sys#2176 Upstream-commit: 533f67a
Use SCHED_OTHER for active A2/A3 workers after the affinity gate, retaining barrier participation when the platform rejects the change. Include Linux syscall-wrapper tests and troubleshooting documentation. Source: hw-native-sys#2166 Upstream-commit: e750ccf
Share the arena-bank capacity rule between onboard and sim runners. Refuse retained temporary-buffer growth before freeing the old buffer. Keep program-mode growth semantics and add both-architecture tests. Ported alongside the existing TMR resource-contract tests. Source: hw-native-sys#2193 Upstream-commit: eb69687
Silence the libc++ std::function bad_function_call false positive at InitRollbackGuard after verifying both calls check for an empty function. Keep existing program rollback behavior unchanged. No lint rule or test is disabled globally.
Return context-local callable handles from a bounded residency cache. Validate device residency at a separate kernel invocation entry while leaving payload execution unsupported. Port the handle signatures to existing K4 contract fixtures; retain K4 full image validation and K5 static configuration ownership. Source: hw-native-sys#2190 Upstream-commit: 2c87647
Compose gated AICore/AICPU submission, completion joins and bounded Host compensation with an explicit owner lease. Add native adapters, fault-injection tests and capture-safe source guards. Keep the binder independent of context and runtime providers; merge its build targets alongside K2/K4/K5 rather than replacing them. Source: hw-native-sys#2187 Upstream-commit: 136e971
Preflight every kernel arena before allocating a new region. Roll back only regions created by a failing request, including allocator exceptions, so previously published bases and images stay usable. Preserve the program-mode growth and whole-bank rollback policy.
Share the canonical callable validator with cache admission and reject oversized images before traversing their contents. Withdraw borrowing when close begins. Retain the code arena and allocator bookkeeping when free fails, permit close retry, and refuse owner destruction while allocations remain. Do not enter allocator-wide cleanup until persistent arguments and the callable arena are released.
Prefix the unchanged K4 invocation with the K10 dispatch envelope and an explicitly borrowed residency descriptor address. Check the complete SDK length and keep the owned host packet alive through native enqueue. Adapt the gated transport probe with separately published test residency fixtures. Keep public launch fail-closed until real providers and the submission owner are connected; no kernel executor or device pool is introduced. Document the combined prepare and close lifecycle.
Coordinate TMR kernel invocations independently of program lifetime. - Admit trusted prepared context and callable views before publishing invocation inputs to execution threads. - Join initialization, finalization, final-status readers and departure through an epoch-tagged gate with exactly-once cleanup. - Add dedicated AICore entry binaries, bounded control/report clearing and pre-window cancellation with explicit retirement acknowledgments. - Keep runtime diagnostics separate from CANN native return statuses. - Exercise both architecture consumers and add real CANN protocol probes. Public launch stays disabled pending resource-provider integration. Overlapping callers and eager caller error propagation remain explicit integration limits; failed cleanup retains resources instead of reuse.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
List the imported callable-residency page in the launch navigation so the strict documentation build accepts it without suppressing omitted-page validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
K5 separates resident configuration from invocation inputs, but that separation alone does not establish when execution may begin or when borrowed inputs may be released. The existing program path permits orchestration to overlap initialization and uses its own completion lifecycle. Kernel mode needs a uniform initialization verdict, coordinated cancellation, and a final-reader boundary without changing that program behavior.
This PR connects trusted invocation admission → coordinated execution → AICore retirement → final-status publication → last-reader release:
This is an independently reviewable TMR execution implementation, not an enabled public kernel-mode feature.
simpler_kernel_mode_supported()remains0, and public launch stays fail-closed. The real resource-provider/Host-binder connection is not supplied by the test fixtures. Eager caller error propagation and arbitrary overlapping submissions also remain unresolved, as detailed below.Execution contract and coordination
Runtime consumers validate the prepared inputs
The two TMR AICPU implementations expose matching internal native entries:
kernel_prepared_context.hvalidates context identity/generation, fixed topology, resource ranges, capacities, alignment, and RuntimeContext placement. Prepared configuration is not replaced by invocation arguments.kernel_prepared_callable.handkernel_registration.hvalidate residency membership, callable spans, signature counts, slot generation, and the bounded function table. Membership and bounds checks precede cache maintenance or dereferencing an untrusted callable image.kernel_execution.handkernel_execution_inputs.hcarry explicit borrowed execution inputs. K4's packet is converted once into executor-owned storage; config, orchestration, scheduler handshakes, and child dispatch use the same invocation and callable table.The new TMR-only descriptors are separate from the existing public ABI and K9 invocation layout:
host_cancelupdate, not a replacement for device-side error handlingThese are resource requirements and consumer contracts. This PR does not make a test-owned descriptor into a production provider, establish the complete committed-HBM budget, or allocate per-call device argument buffers.
Round coordination without changing program scheduling
kernel_round_gate.hprovides an epoch-tagged gate for all launched workers, including workers filtered out of execution by affinity.kernel_execution_round.his the common coordinator used by both architecture executors.kernel_core_group.hand the two architecture AICore consumers implement independent pre-window cancellation, register-window startup, exit acknowledgement, window closure, and release. Ordinary and early-dispatch paths share the existing child execution logic while observing kernel cancellation. Kernel mode receives its ownaicore_kernel_mode.oand envelope; the loader does not substitute the old program entry when that binary is missing.kernel_native_status.hseparates raw runtime status, cleanup status, logical dispatch classification, and the CANN native return. Only the four new native boundaries map success to0and failure to the SDK'sINNER_ERRORvalue2. Returning logical dispatch value6directly was observed to produce replay EOS; the mapping is checked against the installed SDK by the onboard probe. Program and HBG native return behavior is unchanged.Lifecycle and program-path impact
Lifetime and concurrency: Prepare/release require externally established quiescence; generation validation is not a lifetime pin. One executor/workspace has one active round, and its invocation storage remains borrowed until the last reader leaves. Cleanup failure does not authorize storage reuse or resource release. Gate Idle alone does not prove that all CANN native tasks or graph references have ended. Program and kernel execution in the same runtime SO are not supported concurrently.
Boundary safety: Shared clearing must already be ordered after the previous round and before Start. The clear-plan validator constrains what may be cleared, but a gate reached inside the CPU task cannot prevent another caller from clearing shared memory earlier. Arbitrary cross-caller/graph overlap therefore still requires an admission/identity protocol before clear; the current external non-overlap requirement is not a completed overlap-safety test.
Program-path impact: Existing program wrappers, bind/staging behavior, KernelArgs/Runtime physical layouts, and program cleanup remain in place. Shared execution bodies gain explicit kernel inputs and conditional retirement; program calls retain their original data sources and lifecycle. The five public kernel ABI signatures, execution-mode definitions, and existing K9/K4 packet layouts are unchanged. New context/control records and native exports are internal TMR interfaces; this PR does not implement HBG kernel execution.
Left to follow-up integration: Connect the real stable-resource provider, callable residency lifetime, Host launch/clear-plan adapter, and submission/close protection. Resolve eager error delivery to the caller: on A3, the hidden CPU stream reports
507018while caller synchronization returns success, despite Done-event joins. The corresponding test intentionally fails. Also establish pre-clear protection for overlapping submissions and the recovery policy after native task failure; do not infer safe reuse fromcleanup_status == 0alone.Tests
The final 69-file source snapshot was hash-checked against
fea56ab0. Final-snapshot checks and earlier broad regression runs are distinguished below. These results are not a claim that the full CI matrix or the complete pipeline acceptance suite passed.Coverage added or extended in this PR
test_tmr_kernel_round_gate.cpp/test_tmr_kernel_execution_round.cpp: Epochs, filtered workers, duplicate arrival, delayed initialization/finalization/readers, uniform verdicts, exactly-once cleanup, failure/reuse, and report publication before readers return. Interleavings use explicit synchronization, not sleeps.test_tmr_kernel_prepared_context.cpp/test_tmr_kernel_prepared_callable.cpp: Trusted identity/generation, topology, resource ranges and capacities, callable signature/function-table bounds, residency membership before image access, and failed registration without publication.test_tmr_kernel_clear_plan.cpp: Missing or oversized clear regions, overlap/static-region rejection, retained pollution, and refusal to reuse a busy round.test_tmr_kernel_core_group.cpp/test_tmr_kernel_aicore.cpp: Cache/MMIO protocol models, pre-window cancel, stale register state, acknowledgements, close/release, and both architecture AICore bodies, including ordinary/early-dispatch cancellation.test_tmr_executor_execution_inputs.cpp/test_tmr_scheduler_execution_inputs.cpp: Both real architecture consumers; A/B/A callable-table isolation, invalid admission, initialization/configuration failure followed by reuse, runtime errors, both scheduler bindings, and exact native status mapping. Tests call the production coordinator instead of inserting their own initialization barrier.test_runtime_builder.py/test_runtime_compiler.py: Dedicated kernel binary discovery/build behavior and preservation of program/HBG artifacts.tools/cann-examples/tmr-kernel-control/: Committed hardware probe using production K2 stream operations, binder submission sequence, round coordinator, CoreGroup, and AICore binary. Its executor/resource owner is a protocol fixture, not a full TMR operator or production K3 provider. The probe README documents commands, assertions, and deliberately failing caller-error acceptance.Executed results
-47, cleanup0, dispatch6, native2; hidden CPU returned507018. Caller returned0, diagnostic-only in this variant.507011(MODEL_EXECUTE), with matching current-epoch control, Core-retirement and CPU-result evidence.507018, caller0;native_error=PASS,caller_error=NOT_PROPAGATED. Native failure does not propagate through the current event topology to the eager caller.Hardware validation used
Ascend910_9392with CANN 9.0.0 and exclusive single-device task jobs. Builds used checkout-isolated environments, explicit ccache and four-way parallelism; existing Torch dependencies were reused without installing Torch. A5 onboard compilation is not A5 hardware execution.The L2 skip is the existing sim-only unpublished HBG host-handle guard. Program smoke and broader regression can overlap, so their counts are not summed. The final hardware probe executes the production coordination/control code with a fixture executor; it does not validate full TMR child-task execution through public launch. Terminal native-error tests retain resources until process exit, do not reset or reuse the context, and do not treat readable result slots as proof that CANN collected every worker's native return.
Main reproduction commands, from the repository root after installation and target builds following the testing guide:
For exclusive-device probe builds/runs and the three separate terminal variants, follow the committed probe instructions. The caller-first variant is expected to reproduce the unresolved contract failure, not to be omitted or counted as passed.
Not validated or not implemented: A5 hardware, a real provider-backed public kernel launch and complete operator capture/replay, arbitrary overlapping callers/graphs, native-error recovery/reuse, full-network integration, and the complete CI/manual/SDMA/multi-device matrices. Eager caller error propagation is a demonstrated failure, not merely an unrun test. Program regression and protocol-probe success do not close these gaps.
Commits
One K7 commit is added on top of the fixed integrated parent
e727684b:fea56ab0—Add: TMR kernel-mode execution coordination and teardown（K7）: Prepared-input admission, coordinated initialization and four-phase retirement, AICore control/cancel, exact clear contracts, native status mapping, both-architecture consumer tests, and real CANN protocol probes.Merge order: reconcile the K1/2a/K2/K4/K5 and imported resource/binder prerequisites → this PR (K7). Prior component commits and authors are preserved. This PR does not replace their resource owners or turn the remaining integration requirements into completed work. Keep it Draft until the remaining caller-error, provider/lifetime, and overlap contracts have an agreed implementation and validation boundary.
中文总结
e727684b，本次仅新增fea56ab0（K7）。前序 K1/2a/K2/K4/K5、容量修复、callable residency、binder 等提交及原作者不变；审阅优先看 K7-only diff，合入前对齐依赖并复验。0。