Skip to content

test: skip CLI parser tests that execute the full pipeline in CI - #727

Merged
michael-richey merged 1 commit into
mainfrom
michael-richey/skip-slow-cli-parser-tests
Sep 28, 2026
Merged

michael-richey merged 1 commit into
mainfrom
michael-richey/skip-slow-cli-parser-tests

Conversation

@michael-richey

@michael-richey michael-richey commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

CI runs are taking 10–134 min for the unit suite and blocking PRs (jobs appear "stuck" up to the 6h GitHub default timeout). This is not caused by any recent code change — it's pre-existing flakiness.

Root cause

Three "flag accepted" parser tests use CliRunner.invoke() to assert click recognizes a flag (exit_code != 2 is click's "unknown option" usage error). But invoke() does not stop after parsing — it executes the whole command. In CI, tox.ini's passenv = DD_SOURCE_*,DD_DESTINATION_* lets the real workflow secrets reach the unit tests, so sync/migrate/diffs/import actually run the full pipeline against live Datadog orgs, retrying up to DD_HTTP_CLIENT_RETRY_TIMEOUT (300s) per call.

Timestamp-gap analysis of completed job logs names the same culprits across independent runs (macos 09-28, ubuntu 09-25):

Test Duration
test_force_missing_dependencies_cli.py::test_sync_accepts_force_missing_deps 50–65 min
test_drop_unresolvable_principals_cli.py::test_migrate_accepts_drop_unresolvable_principals 12–55 min
test_force_missing_dependencies_cli.py::test_migrate_accepts_force_missing_deps 2–10 min

The wide variance (10–134 min for the identical suite across runs) is just how long those real network calls happen to take — classic flakiness, not a deterministic hang. Locally the suite runs in ~13s because the fake API-key defaults 403 immediately.

Change

Skip the affected tests with pytest.mark.skip and a pointer to the root cause, so CI is unblocked while a proper fix is designed.

  • tests/unit/test_force_missing_dependencies_cli.py — module-level skip (all 3 tests execute the pipeline)
  • tests/unit/test_drop_unresolvable_principals_cli.py — per-test skip on the 3 exit_code != 2 tests
  • Kept test_import_rejects_drop_unresolvable_principals running: it asserts exit_code == 2 (click parse error), which returns immediately without executing anything.

Follow-up (not in this PR)

A proper fix would rewrite these to short-circuit after parsing — e.g. CliRunner(env={...}) with fake creds, or a parser-only assertion — and optionally drop DD_SOURCE_*,DD_DESTINATION_* from tox.ini's passenv for the plain [testenv] so real secrets never reach unit tests. Adding timeout-minutes: 30 to the test job in test.yml would also make a future genuine hang fail loudly instead of lingering up to 6h.

Verification

  • tox -e py311 -- tests/unit/test_force_missing_dependencies_cli.py tests/unit/test_drop_unresolvable_principals_cli.py -v → 1 passed, 6 skipped in 0.08s
  • ruff and black clean on the two changed files.

test_force_missing_dependencies_cli.py and test_drop_unresolvable_principals_cli.py
use CliRunner.invoke() to assert click recognizes a flag (exit_code != 2 is
click's "unknown option" usage error). But invoke() does not stop after
parsing — it executes the whole command. In CI, tox.ini's
`passenv = DD_SOURCE_*,DD_DESTINATION_*` lets the real workflow secrets reach
the unit tests, so sync/migrate/diffs/import actually run the full pipeline
against live Datadog orgs, retrying up to DD_HTTP_CLIENT_RETRY_TIMEOUT (300s)
per call.

Measured wall-clock from recent runs (timestamp-gap analysis of completed job
logs): the suite swings from 10 min to 134 min across identical runs on the
same OS, with three tests accounting for ~95-130 min of every run:

  test_sync_accepts_force_missing_deps               50-65 min
  test_migrate_accepts_drop_unresolvable_principals  12-55 min
  test_migrate_accepts_force_missing_deps             2-10 min

This flakiness blocks PRs (jobs appear stuck for up to 6h, the GitHub default
timeout) without any code change being responsible. Skip these tests until
they are rewritten to short-circuit after parsing (e.g. CliRunner with fake
creds via env, or a parser-only assertion).

Kept test_import_rejects_drop_unresolvable_principals running: it asserts
exit_code == 2 (click parse error), which returns immediately without
executing anything.
@michael-richey
michael-richey merged commit 623265f into main Sep 28, 2026
21 of 23 checks passed
@michael-richey
michael-richey deleted the michael-richey/skip-slow-cli-parser-tests branch September 28, 2026 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants