[PRAteek-singHWY]

Fixes #885.

What

preload in application/utils/gap_analysis.py polls /rest/v1/map_analysis until every standard pair is cached. A pair that never succeeds keeps it looping forever: a 404 when CRE_NO_CALCULATE_GAP_ANALYSIS is set or on Heroku, or a job that keeps failing and being re-enqueued. The loop is still on current main.

Change

• What counts as a failure. A non-200, a 200 with neither result nor job_id, or a job_id that changed since the last poll, because map_analysis only enqueues a new job when the last one ended without a result (or its in-flight key expired). The same job_id again means the job is still running, so it is not a failure, and it clears earlier failures.
• The cap. After PRELOAD_MAX_FAILURES (10) failures in a row, the pair is dropped. Counted per pair.
• Reporting. preload returns the dropped pairs. It logs map analysis preloaded successfully only when nothing was dropped, and otherwise logs an error listing them.
• The waiting list is (a, b) tuples, and finished pairs are not polled again. No sleep after the last pass.

Why the first version of this PR was wrong

It counted "job still running" as a failure. GA jobs may run for 36 hours (GAP_ANALYSIS_TIMEOUT), so a healthy job needing 51 polls was dropped at poll 10, and the run then printed "preloaded successfully". Reworked after a self-review on 2026-11-05.

Tests

application/tests/gap_analysis_preload_test.py, 8 tests, no network: all cached means one pass and no sleep; a finished pair is not polled again; a long-running job is not dropped; scattered 503s while a job runs do not drop it; a pair that keeps returning 404 is dropped after 10; a job that keeps failing and re-enqueuing is dropped; failures are counted per pair; success is not logged when pairs were dropped. Six of the first seven fail against the earlier version of this PR.

Proof

Full suite merged with current main: 1105 tests OK (skipped=6). black clean, no new mypy errors.

Limits

• A job stuck under one id (dead worker) is bounded only by the 36h in-flight key expiring.
• cre_main.py ignores the return value, so the process still exits 0 when pairs were dropped; only the error log reports it.

[PRAteek-singHWY]

Hi @Pa04rth, this PR resolves the infinite loop problem reported in #885. The issue has been solved by implementing a 10-retry cap for failing background jobs and optimizing the runner loop with tuples.
Happy to iterate if needed .

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: OWASP/OpenCRE
• Branch: main
• File: .coderabbit.yml
• Commit: facfd46
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Advanced
• Run ID: d4e93d87-d686-4c78-825c-4ef79881fc5a

Commits

Reviewing files that changed from the base of the PR and between facfd46 and f631ed4.

Files selected for processing: 2

• application/tests/gap_analysis_preload_test.py
• application/utils/gap_analysis.py

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn