fix(sync): stream JSONL sources in bounded batches and ship external feeds disabled

[rubenvdlinde]

What broke

Rehearsal R3 (fresh NC 32.0.13, integriq 0.4.12-unstable.20261015175726): cron.php exited 255 on 2 of 5 runs, Allowed memory size of 2147483648 bytes exhausted in SynchronizationService::parseJsonLines().

Cause confirmed: a fresh install enables 18 jobs that pull external data. One reads Austria's OpenTender full.jsonl.gz (70 MB gzip, 451 MB and 181,741 records decompressed). The JSONL path held the body whole three times (base64 call log body, gunzipped string, every record in one array). Replaying that exact path against the real file under memory_limit=2048M dies at about 90,000 records in the json_decode line, the same line as the cron fatal.

What changes

• Streaming JSONL. A format: "jsonl" synchronization streams the download to a temp file through the call engine's existing sink, and reads it line by line (gzopen/gzgets, gzip or plain). The temp file is removed after the batch.
• One batch per run. sourceConfig.pageSize records (default 5,000), resumed through the existing currentPage. Only a whole file read in one run is a complete fetch; every other batch is incomplete (batch_limit_reached), so the deletion pass never runs on part of the source.
• Resume on the next tick. The run result carries resume; SynchronizationAction returns resumeSoon, and JobService schedules that job on the next tick instead of after its interval.
• External feeds are opt-in. The ten external data-gathering connector jobs (TenderNed, AusTender, BOAMP, Austria OpenTender, Germany Bund, Diavgeia, Latvia IUB, Avropa, DPG registry, Nextcloud App Store) ship isEnabled: false. Sources, mappings and synchronizations still ship. Their in-flight spec (native-data-gathering-provider) already said a connector needing streaming of very large files ships disabled; its "live-verified may ship enabled" scenario now says the Job ships disabled.
• OpenSpec change bounded-jsonl-sync (REQ-006 modified, job-scheduling requirement added); validates.

Not changed, on purpose

• The eight endoflife.date jobs stay enabled: endoflife-date-source spec requires isEnabled: true. They are one small JSON call per product per day.
• Per-job failure isolation already exists: JobService catches \Throwable around each action and per job in the pass (pinned by testRunProcessesAllThreeJobsWhenTheFirstThrows). Memory exhaustion is a fatal no handler sees, which is why the fix is the bounded reader.
• Whole-document formats (JSON, XML, YAML, markdown, HTML, and gzdecode for gzip JSON) still parse one page whole; they are bounded by pagination, maxPages and the existing memory ceiling.
• Existing installs keep their jobs' enabled state (seeded job versions unchanged).

Verified

• Real file through the shipped reader at memory_limit=512M: a 5,000-record batch from the start peaks at 90 MB; from record 175,000 at 82 MB; the whole file is scanned in about 4 s.
• New tests, each with a mutation control (whole-body reader, dropped currentPage carry, dropped resumeSoon, complete forced true all turn them red): stream to a removed temp file, batch and resume, whole-file run complete, failed download keeps its batch, a 24 MB file read under a 2 MB budget with a control proving the fixture exposes a whole-body reader, a batch run deletes nothing and persists the next batch, resumeSoon scheduling.
• composer check:strict (run once, phpmd re-run on the two refactored files after): lint, phpcs, psalm, phpstan, routes, legacy-types pass; phpmd clean after splitting fetchJsonLinesBatch; PHPUnit 5,500 tests, 11 errors, all in tests/Unit/AppInfo/* (TypeError: Cannot assign MockObject_IAppContainer to property App::$container of type DIContainer), a local OCP-stub mismatch in DI tests this PR does not touch.

Inherited: phpcs reports 313 findings in the touched test files on lines this PR does not change (tests are outside the phpcs.xml scope).

[github-actions]

Quality Report — ConductionNL/integriq @ 7500550

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
build		✅
check-specs		✅
test-l10n		✅
format		✅
check-schema-l10n		✅
check-l10n-js		✅
composer			✅	✅ 143/143
npm			✅	✅ 646/646
app:check-code	⏭️
info.xml	✅
REUSE	❌
lockfile sync		✅
PHPUnit					✅ 5499/5499 · 1 skipped
Newman					✅ 130/130
Playwright					⏭️ deferred: E2E runs locally and on the promotion path only. This pull request targets development, so the suite is asked once per promotion into beta and main rather than once per push per open pull request. Run it locally with npx playwright test, or from the Actions tab on a branch with no open pull request into development.
Hydra gates					✅
Hydra gates (axe)					✅

---

Quality workflow — 2026-10-16 10:20 UTC

Download the full PDF report from the workflow artifacts.