Skip to content

test(skywalking): remove worker-0 dependency in backend timer assertion - #13902

Open
janiussyafiq wants to merge 1 commit into
apache:masterfrom
janiussyafiq:test-skywalking2-flaky
Open

test(skywalking): remove worker-0 dependency in backend timer assertion#13902
janiussyafiq wants to merge 1 commit into
apache:masterfrom
janiussyafiq:test-skywalking2-flaky

Conversation

@janiussyafiq

@janiussyafiq janiussyafiq commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Description

t/plugin/skywalking2.t TEST 2 fails intermittently on CI, and since it can fail the flaky-rerun too, it turns whole build jobs red.
Example: run 33146313320 on #13651, where it failed both the initial pass and the rerun with:

Failed test 't/plugin/skywalking2.t TEST 2: trigger skywalking - grep_error_log_out (req 0)'
got: ''
expected: 'start skywalking backend timer'

Cause: the test's patched startBackendTimer logs start skywalking backend timer only on ngx.worker.id() == 0, so TEST 2 needs worker 0 to accept at least one of its 50 keepalive-off requests.
The comment in the test assumed independent uniform accepts ((3/4)^50, "vanishingly unlikely" to miss worker 0), but nginx accept distribution is skewed and correlated: measured locally over 60 runs, the per-worker split of the 50 requests was consistently about 17/16/14/3, with the starved worker receiving as few as 1 request and worker 0 sitting in the starved slot in roughly a quarter of runs.
On a loaded 4-vCPU CI runner the starved worker can receive 0 requests, and when that worker is worker 0 the log line never appears.

Fix: replace the worker-id gate with an atomic ngx.shared.DICT:add() first-starter guard, so whichever worker handles the first sampled request starts the timer and logs, exactly once.
The assertion no longer depends on which worker the kernel picks; it only needs any request to be handled, which the block already asserts via the 50 expected 200s.
The 50-request loop is kept: with multiple workers each calling startBackendTimer on their first request, the expected single log line now also proves the guard dedupes across workers.

Verification (250 local runs):

Experiment old guard new guard
forced worker-0 starvation: 1-request variant, 25 runs 23 FAIL / 2 pass 25/25 pass
fixed test, idle machine, 100 runs - 100/100 pass, one log line each
fixed test pinned to 4 CPUs with background load, 100 runs - 100/100 pass, one log line each

The 1-request variant makes the failure condition (worker 0 gets no request) the common case, reproducing the CI failure on demand with the old guard and never with the new one.

Which issue(s) this PR fixes:

None filed; flake observed on #13651's CI.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants