[DevOps]: Schedule diagnostics linking on Chrysalis and Spin

[tomvothecoder]

Description

Closes #303.

• Add a diagnostics operation to the shared site launcher, reusing protected API environments with independent logs and locks.
• Schedule live Chrysalis diagnostics linking daily at 14:00 UTC for dev and prod; preserve existing ingestion jobs.
• Handled API failures are counted while scanning continues. Deferred state lookups or failed link submissions produce a nonzero exit; later scheduled runs retry the remaining work.
• Group scanner configuration and execution by responsibility, add regression coverage, and document the separate NERSC Spin CronJob using the existing GUI runbook format.

Checklist

• Code follows project style guidelines
• Self-reviewed code
• No new warnings in the checks run
• Tests added or updated (if needed)
• All tests pass (locally and CI/CD)
• Documentation/comments updated (if needed)
• Breaking change noted (if applicable): invalid or empty scanner DRY_RUN values now fail configuration validation

Validation

• Passed: 71 focused launcher/scanner tests using uv run --project backend pytest backend/tests/features/ingestion/test_site_collection_launcher.py backend/tests/features/ingestion/test_diagnostics_link_scanner.py --noconftest --no-cov --capture=sys -q.
• Passed: bash -n backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh.
• Passed: make pre-commit-run (including Ruff and type checking) and git diff --check against main.
• Full make backend-test previously failed during setup because .envs/local/backend.env is absent. Per user instruction, full backend validation is deferred to CI/CD; no full-suite pass is claimed.
• Live Chrysalis/Spin deployment validation has not been run.

Deployment Notes (if any)

• Merge the new diagnostics entries into the installed Chrysalis crontab, preserving local paths and schedules. They explicitly use DRY_RUN=false; valid protected dev/prod API credentials are required.
• Create the separate Spin diagnostics secret and CronJob through Rancher following the runbook. Mount the diagnostics archive at its registered path, use Forbid concurrency and a job backoff limit of zero.
• Monitor archive traversal duration and the summary counters. Handled API failures remain counted without a nonzero exit; discovery failures may leave the discovered count at zero.
• No schedules or GUI workloads are installed by this PR.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved scanner error-handling and whitespace-validation issues remain, along with a launcher lock regression-test gap.

Review effort: Lite
Findings: 1

Open (1)

• Handled candidate failures incorrectly produce nonzero exit · New

What changed in this PR

Adds scheduled diagnostics linking for Chrysalis and NERSC Spin using the shared launcher, scanner, separate logs/locks, and operational documentation.

Changes:

• Adds diagnostics scheduling and launcher support.
• Improves scanner configuration and failure reporting.
• Adds regression tests and deployment runbooks.

File	Summary
docs/​operations/​test-ingestion-operations.md	Documents diagnostics validation steps.
docs/​operations/​setup-ingestion-operations.md	Documents scheduled diagnostics operations.
docs/​operations/​nersc-spin-runbook.md	Adds the Spin diagnostics CronJob runbook.
backend/​tests/​features/​ingestion/​test_site_collection_launcher.py	Tests launcher behavior, locking, and scheduling; lock-invocation coverage needs strengthening.
backend/​tests/​features/​ingestion/​test_diagnostics_link_scanner.py	Covers scanner configuration and failure handling.
backend/​app/​scripts/​ingestion/​sites/​templates/​crontab.example	Adds development and production diagnostics schedules.
backend/​app/​scripts/​ingestion/​sites/​site_ingestion_launcher.sh	Adds diagnostics routing, logging, and locking.
backend/​app/​scripts/​ingestion/​diagnostics_link_scanner.py	Adds scanner configuration and reporting; handled failures and whitespace validation require changes.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.