Fix: normalize onboard CANN worker scheduling policy - #2166
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (7)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe change adds a scheduling helper that selects ChangesA2/A3 scheduling control
Priority: ➖ Normal — Schedule the A2/A3 worker scheduling change because it addresses periodic FIFO-polling stalls and real-time CPU throttling, with qualification showing lower worst-case latency. Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to A2/A3 onboard AICPU workers are normalized to normal Linux scheduling after affinity selection, reducing real-time scheduling stalls while safely retaining the existing policy if adjustment fails. No concrete merge-blocking risk remains. Sequence Diagram(s)sequenceDiagram
participant AICPUWorker
participant simpler_aicpu_exec
participant SchedulingHelper
participant LinuxScheduler
AICPUWorker->>simpler_aicpu_exec: invoke kernel
simpler_aicpu_exec->>SchedulingHelper: use_normal_aicpu_scheduling()
SchedulingHelper->>LinuxScheduler: query current policy
LinuxScheduler-->>SchedulingHelper: return policy
SchedulingHelper->>LinuxScheduler: set SCHED_OTHER when needed
SchedulingHelper-->>simpler_aicpu_exec: return status
simpler_aicpu_exec-->>AICPUWorker: continue kernel execution
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 3 files. (4 skipped: 4 unsupported.) ✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. A rabbit checks the worker thread, Comment |
41cc95f to
d3cec5f
Compare
Set active A2/A3 CANN workers to SCHED_OTHER with priority 0 after the affinity gate in simpler_aicpu_exec. Check each invocation and skip the setter when the worker already has the requested policy. Continuous FIFO polling can exhaust the real-time CPU budget and produce periodic stalls during dual-slot decode. Cover TMR orchestration and scheduler workers and HBG scheduling workers. Warn once per loaded runtime SO on syscall failure and continue barrier participation. Add four Linux syscall-wrapper tests and indexed troubleshooting documentation for periodic worker stalls and scheduling tradeoffs.
d3cec5f to
9c1e1f6
Compare
2 participants
Summary
Continuous FIFO polling can exhaust the real-time CPU budget and produce periodic stalls. Hardware runs showed roughly 40 ms pauses near 1 Hz, with a configured RT budget of 950000 us per 1000000 us. The observations are consistent with real-time bandwidth throttling; a kernel throttle event was not captured directly.
After the affinity gate, A2/A3 onboard
simpler_aicpu_execcalls a helper that checks each active CANN worker's policy on every invocation and setsSCHED_OTHER, priority 0, when needed. This covers TMR orchestration/scheduler workers and HBG scheduling workers.Syscall failures produce one warning per loaded runtime SO, guarded atomically, while workers continue barrier participation. The policy persists across calls; contention under normal scheduling may add latency.
Adds four Linux syscall-wrapper tests and indexed troubleshooting documentation.
Hardware Evidence
A3, CANN 9.0, GCC 10; Qwen3-14B, batch 16, 127 decode steps, two slots split 64/63, and 122 steady-state samples per qualification run:
All final candidate runs passed golden checks. Effective latency measures TMR scheduler duration and HBG device wall time; compare each runtime against its own baseline. No candidate run had an effective sample above 50 ms.
One earlier TMR candidate run had a token mismatch at step 112. Across all candidate runs, TMR passed 4/5 golden checks and HBG 4/4. The mismatch did not recur in follow-up runs; its cause was not established.
Hardware evidence used base
39ce891, before rebasing onto7bcdc95. Intervening main changes include DFX configuration, HBG early-dispatch prerequisites, A2/A3 worker retirement, and A5 HBG profiling. The scheduling implementation is unchanged; the only manual conflict resolution preserved both troubleshooting index entries. Hardware tests were not rerun on the rebased integration. These are qualification measurements, not the formal multi-run performance campaign.Validation
BufferError: cannot close exported pointers exist, reproduced individually on both baseline and candidate:test_worker_chip_message_queue,test_worker_chip_orch_comm_stream,test_l3_single_hop_two_lifecycles, andtest_l4_two_hop_two_lifecycles.