Skip to content

perf: upload remote storage files in background and avoid duplicate image loads - #1674

Closed
gambijul wants to merge 5 commits into
PhotoboothProject:devfrom
gambijul:feat/background-remote-upload
Closed

gambijul wants to merge 5 commits into
PhotoboothProject:devfrom
gambijul:feat/background-remote-upload

Conversation

@gambijul

@gambijul gambijul commented Nov 7, 2026

Copy link
Copy Markdown

Prerequisites checklist

What is the purpose of this pull request? (put an "x" next to an item)

  • Documentation update
  • Bug fix
  • New feature
  • Other, please explain: performance on low-end hardware (Raspberry Pi 3)

What changes did you make? (Give an overview)

Remote storage (FTP/SFTP) uploads ran synchronously inside api/applyEffects.php, so guests waited for the network before seeing their picture, and an unreachable server turned every capture into an error after a long timeout.

  • New option ftp.upload_in_background (default off, so existing setups are unchanged). When enabled, the picture is queued in data/remote-queue and uploaded by a detached worker (bin/photobooth photobooth:remote-storage:process). Failed uploads stay queued and are retried on the next capture, or by running the command manually / from a timer. --status lists pending uploads. A lock file prevents concurrent workers.
  • Fix RemoteStorageService::createWebpage() checking for config.php instead of config.inc.php, which re-uploaded the gallery web page on every picture.
  • Result page (assets/js/core.js): the preloaded image now uses the exact same URL as the stage background, and its known dimensions are passed to addImage(), so the full-size picture is downloaded and decoded once instead of twice.

Measured on a Raspberry Pi 3B with an SFTP target:

Before After
applyEffects.php (SFTP online) 4.6 s 2.6 s
applyEffects.php (SFTP offline) error after 13–75 s 2.6 s, upload retried later

Is there anything you'd like reviewers to focus on?

  • RemoteStorageQueueService::startWorker() spawns the worker with nohup … & via exec(); happy to switch to another mechanism if preferred.
  • Only the English translation strings were added (ftp:upload_in_background, manual:ftp:upload_in_background).
  • npm run eslint passes on assets/js/core.js. The PHP code has been running on PHP 8.4 (DietPi) during real use; I could not run php-cs-fixer locally.

AI used to create this Pull Request?

Yes. Claude Code (Claude Opus 5.5) profiled the photobooth on the Raspberry Pi, wrote the queue service, the console command and the core.js change, and drafted this description. I reviewed and tested the changes on my own booth (Pi 3B, Nikon D3200, SFTP storage), including with the SFTP server offline.

🤖 Generated with Claude Code

…mage loads

Remote storage (FTP/SFTP) uploads ran inside applyEffects.php, so guests
waited for the network before seeing their picture, and an unreachable
server turned every capture into an error.

- Add ftp.upload_in_background (default off): pictures are queued in
  data/remote-queue and uploaded by a detached worker, with retries on the
  next capture or via `bin/photobooth photobooth:remote-storage:process`
  (`--status` lists pending uploads).
- Fix createWebpage() checking config.php instead of config.inc.php, which
  re-uploaded the gallery page on every picture.
- Result page: preload the exact URL used as background and pass the known
  dimensions to addImage(), so the full-size picture is downloaded and
  decoded once instead of twice.

Measured on a Raspberry Pi 3 with SFTP: applyEffects 4.6 s -> 2.6 s, and
still 2.6 s with the SFTP server offline (previously an error after 13-75 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@reloxx13 reloxx13 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and for sharing the Raspberry Pi measurements, including the offline-server case. Moving uploads out of the capture request is a useful improvement.

I found four issues that need addressing before merging; details are attached inline.

Validation: reproduced the video URL parsing issue and exercised the unchanged queue service in an isolated harness with test doubles for destination changes and webpage-creation failures. I did not run an actual FTP/SFTP transfer, a browser test, Windows execution, or the full test suite.

Comment thread assets/js/core.js Resolved

@reloxx13 reloxx13 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for the PR. Following up with queue behavior and performance observations for a larger backlog.

The main correctness issue is that one permanently failing file blocks every later upload. The other notes are optimization suggestions, not measured performance regressions: I have not run a load test with thousands of queued images on a Raspberry Pi.

Keeping uploads serial and triggering work on new captures is sufficient for the intended workflow; a timer, parallel uploads, or a queue database is not required. The existing code already reads and sorts once per queue snapshot, rather than once per individual upload, so there is no need to add that optimization again.

@reloxx13
reloxx13 marked this pull request as draft November 16, 2026 09:44

@reloxx13 reloxx13 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I have contributed follow-up fixes and regression tests in 1c39cfa (video QR), ca8e96d (upload queue) and 395a6cc (capture triggers) and replied to all seven review threads. The requested correctness changes are addressed.

Validation: 60 PHPUnit tests / 314 assertions and full PHPStan pass under DDEV; 28 focused tests / 282 assertions pass under native Windows PHP 8.4. Browser and isolated FTP/SFTP checks were also completed during implementation. The final queue write-reduction changes were covered by the Linux and Windows tests; real FTP/SFTP was not rerun after that last optimization.

Raspberry Pi performance, camera hardware and Windows Apache/FastCGI remain unverified. The backlog optimization was evaluated with desktop measurements and explicitly deferred beyond removal of redundant work. No merge performed.

@gambijul could you please review my follow-up changes in 1c39cfa, ca8e96d and 395a6cc and let me know if they fit your intended behavior? Thanks again for the PR!

Bind jobs to destinations, preserve gallery retries, distinguish connection and file failures, and coordinate detached Linux and Windows workers. Prevent duplicate jobs and avoid rewriting jobs on failed attempts.

Add queue, service, process and FTP/SFTP regression coverage plus a reproducible backlog benchmark.
Skip hidden chroma intermediates and enqueue saved chroma results. Cover photo, collage and chroma triggers through the real HTTP endpoints.
@reloxx13
reloxx13 force-pushed the feat/background-remote-upload branch from 41bec38 to 395a6cc Compare November 16, 2026 10:35
Keep destination-change and real worker regressions, and remove the superseded manual result-rendering fixture.
@reloxx13

reloxx13 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Please see the discussion here: #1603 (comment)

Closing this PR for now while we agree on a shared approach. Thanks again for your contribution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants