Skip to content

feat(remote-storage): add async FTP/SFTP uploads with JSON queue and upload tracking - #1603

Open
fmiccolis wants to merge 4 commits into
PhotoboothProject:devfrom
fmiccolis:feature/async-remote-upload
Open

fmiccolis wants to merge 4 commits into
PhotoboothProject:devfrom
fmiccolis:feature/async-remote-upload

Conversation

@fmiccolis

@fmiccolis fmiccolis commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

… debug tab

Prerequisites checklist

What is the purpose of this pull request? (put an "x" next to an item)

  • Documentation update
  • Bug fix
  • New feature
  • Other, please explain:

What changes did you make? (Give an overview)

This PR makes the remote storage (FTP/SFTP) upload asynchronous, so the transition
from capture to the result screen is no longer blocked by a slow uplink, and adds
upload tracking plus two new admin debug panel tabs.

Asynchronous upload with tracking

  • applyEffects.php no longer uploads synchronously: files are enqueued in a persistent
    JSON queue (var/run/remotestorage_queue.json, new RemoteStorageQueueService) and the
    response returns immediately.
  • After rendering the result, the frontend fires a non-blocking request to the new
    api/remoteStorageUpload.php, which drains the queue in the background:
    ignore_user_abort(), single-drainer lock (flock), session_write_close() (critical:
    without it the PHP session lock would freeze the booth UI during uploads), priority for
    the just-captured photo, and a circuit breaker after 2 consecutive failures so an
    unreachable server does not pile up ~90s FTP timeouts.
  • Per-file status (pending/uploading/done/failed), attempts and last error are persisted.
    Retries happen on later drains (max 3 attempts); failed uploads can be listed/retried/
    cleared via the new admin-protected api/remoteStorageAdmin.php.
  • The QR code still appears immediately on the result screen (its URL never depended on the
    upload); a small spinner next to it turns into a check/error icon, driven by a cheap
    status-polling mode of the same endpoint (same pattern as gallery.php?status).
  • deletePhoto.php cancels queued uploads and skips remote round-trips for photos that
    were never uploaded.

createWebpage() fixes/improvements

  • Fixed the existence check (config.php → config.inc.php): before this, the early-return
    condition could never be true, so the webpage template + config were re-uploaded on
    every single photo.
  • The webpage creation now runs at most once per storage folder (flag cached in the queue
    file) instead of doing remote fileExists round-trips per photo.
  • New ftp.template_config_location option (default private/template/config.inc.php):
    an optional PHP file returning an array that is merged over the generated
    config.inc.php defaults — the counterpart of the already-customizable
    template_location for people using a custom remote gallery template. Also applied to
    the test/remote-storage-template.php preview.

New debug panel tabs

  • Environment & network: OS, PHP version, all network interfaces with IP, netmask and
    CIDR network (via net_get_interfaces()), Wi-Fi SSID (best-effort via iwgetid), and
    one QR code per IPv4 address encoding the photobooth URL on that network — so an
    operator can quickly open the booth (or SSH/VNC into it) from another device.
  • Remote storage queue: live table of the upload queue (status, attempts, errors,
    timestamps), compatible with the existing auto-refresh toggle.
  • Added the missing translation label for the existing remote storage log tab.

Behavior changes to be aware of

  • Chroma captures with keying.show_all = false are no longer uploaded to remote storage
    (previously they were uploaded and then deleted locally; the remote gallery now mirrors
    local visibility).
  • After 3 failed attempts an upload stays failed until retried via the admin endpoint —
    deliberately not retried on every capture so an offline server cannot slow down an event.
  • The per-photo QR link is unchanged (<gallery>/images/<file>). A ?photo=<file> deep
    link will be proposed in a separate PR, together with support for it in the bundled
    gallery template.

Is there anything you'd like reviewers to focus on?

  • Session lock: session_write_close() in the drain/status endpoints is load-bearing;
    please double-check I haven't missed a path that would hold the session while uploading.
  • Queue concurrency: single-drainer lock, the enqueue-vs-drainer-exit race (mitigated by
    a 1.5s linger re-check, a one-shot re-trigger from the status poller, and the next
    capture's trigger), and stale uploading recovery (entries reset after 10 min).
  • On Apache/mod_php the drain request occupies one worker while a backlog is uploading;
    set_time_limit(0) cannot override server-level timeouts — leftover entries are simply
    picked up by the next drain.

Tested on a live installation during a real event (FTP over a slow uplink): the result
screen now appears immediately, the queue drains in the background, and the
failure/recovery paths were exercised by taking the FTP server offline mid-event.

AI used to create this Pull Request?

Yes. This PR was developed with Claude Code (Anthropic). The AI wrote the initial
implementation (queue service, endpoints, frontend changes, debug tabs) based on a design
we discussed and I decided on; I reviewed every change and tested the full flow on a real
installation at a live event (successful uploads, failure/retry paths, burst captures,
offline-server recovery). npm run build and npm run eslint were run (clean), and all
touched PHP files were syntax-checked.

@flacoonb

Copy link
Copy Markdown
Contributor

Hi, thanks for working on this. This PR is very similar in scope to my earlier PR #1463, which also implements asynchronous FTP/SFTP uploads with a persistent queue, retries and upload tracking.

The main difference seems to be the architecture: #1603 uses a JSON queue processed through a background HTTP request, while #1463 uses SQLite and a dedicated Symfony/systemd worker. It may be worth comparing the approaches and seeing whether parts can be consolidated.

@fmiccolis

Copy link
Copy Markdown
Contributor Author

Hi @flacoonb and sorry for not digging in the previous PRs before submitting mine.
I was the original creator of the FTP upload feature but I never had the time to review its fallacies (one above all the synchronous upload).
One of the things that I would like to keep of your approach is the semplification of the admin settings regarding folder, baseFolder, website ecc because it can be just one property.
I want to know why you used SQLite/systemd, what are the strengths of this choice?
I preferred to not introduce others services along the existing ones in the photobooth.

@flacoonb

Copy link
Copy Markdown
Contributor

Hi @fmiccolis, no problem at all — it is easy to miss an older open PR, especially in a project with many ongoing contributions. I am glad you brought your approach forward, as the overlap gives us a good opportunity to compare both implementations and potentially combine the strongest parts.

I chose SQLite because it provides a persistent transactional queue without requiring an external database server. It allows atomic job claiming, reliable recovery after crashes or restarts, and structured tracking of attempts, errors and filename mappings.

The systemd worker keeps uploads fully independent from PHP and the web request lifecycle. This avoids request timeouts, session locks and occupied web-server workers during slow or unavailable FTP connections. It also gives automatic startup and restart on failure.

The downside is additional installation complexity and a stronger dependency on Linux/systemd. Your HTTP-based approach is simpler and integrates more naturally into the existing Photobooth setup, so I can understand that choice.

I agree that the remote-storage configuration should be simplified, and it may be useful to combine the strongest parts of both PRs.

@reloxx13

reloxx13 commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the work on asynchronous uploads and for documenting the behavior changes and event testing. The JSON/flock approach avoids introducing SQLite or a dedicated systemd service, so the additional-dependency concern discussed on #1463 does not apply here.

I am not ready to approve merging yet for the following reasons:

  1. QR-link compatibility: api/qrcode.php and api/print.php change the per-photo link from /images/<file> to /?photo=<file>. As noted in the PR description, the bundled gallery template ignores photo, so scanning the QR code opens the gallery rather than the selected image. Please preserve the existing link behavior in this PR, or split the deep-link change into a separate PR that also implements and verifies support in the bundled template. A custom template alone does not cover the default installation.

  2. Remaining verification: I have inspected the architecture and diff, but have not yet independently exercised the complete flow locally. Before approval, I still want to verify simultaneous enqueue/drain requests, deletion while an upload is running, recovery after an interrupted worker/request, and behavior with a slow or unavailable FTP/SFTP server while the booth remains responsive. These are verification gaps, not claims that those paths are broken. Your reported live-event testing is useful evidence; this describes what remains unverified in my review.

This is not a request to introduce SQLite or systemd. The immediate code blocker is the QR-link behavior with the default template; the remaining queue scenarios need focused validation before I can give merge approval.

@reloxx13 reloxx13 changed the title feat: async remote storage upload with tracking, QR indicator and env… feat(remote-storage): add async FTP/SFTP uploads with JSON queue and upload tracking Nov 6, 2026
fmiccolis and others added 4 commits November 9, 2026 15:12
- Environment & network tab: one QR code per IPv4 address encoding the
  photobooth URL on that network, to open it from another device
- New remote storage queue tab showing per-file upload status, attempts,
  errors and timestamps (works with the existing auto-refresh toggle)
- Add missing label for the remote storage log tab

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The build workflow runs "npm run build", which sorts en.json with
localeCompare and then requires a clean working tree. The new
remoteUpload* keys were placed after "rembglog" but localeCompare
orders them after "remoteStorageTemplate", so the build modified the
file and the status check failed. it.json is aligned for consistency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bundled gallery template ignores the photo query parameter, so the
/?photo=<file> link opened the whole gallery instead of the picture.
Keep the existing link; the deep link will be proposed separately
together with template support.
@fmiccolis
fmiccolis force-pushed the feature/async-remote-upload branch from 51d76d5 to 5c8ff6c Compare November 9, 2026 14:23
@fmiccolis

fmiccolis commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author

I've reverted the change on /?photo=<file> by going back to /images/<file> as you suggested.
I've done another event where I used the feature and it was ok but I'll wait your response on the matter.

In the end I think that all the settings regarding the FTP upload should be rewritten to give the power user more flexibility on the configuration. I will provide a new solution in another feature

@reloxx13

Copy link
Copy Markdown
Collaborator

fyi @gambijul @fmiccolis @flacoonb @andi34

Thanks for your work on these improvements! We currently have three overlapping approaches to asynchronous FTP/SFTP uploads:

#1677 builds on #1603 and improves the settings, connection testing and gallery links.

Would you agree with using #1603 as the foundation and bringing over the reliability improvements from #1674? In particular:

  • Bind queued uploads to their original destination.
  • Stop on connection failures while allowing later jobs after individual file failures.
  • Keep failed gallery creation independently retryable.
  • Prevent jobs from being stranded when a worker exits.
  • Avoid duplicate jobs and unnecessary queue writes.

We should also add an admin retry button and migrate existing configurations when introducing the settings from #1677.

For worker execution, I would prefer a detached CLI worker, so uploads do not occupy a webserver worker or depend on HTTP request timeouts. However, this requires PHP CLI and permission to start processes. What would you prefer: keep the HTTP-based approach from #1603 or adapt the detached CLI worker from #1674?

I would leave SQLite and a permanent service out of this initial solution. Could we agree on a shared foundation and adapt the remaining improvements as follow-up PRs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants