Skip to content

feat: recover from WebGL context loss - #622

Open
wouterlucas wants to merge 4 commits into
rdkcentral:masterfrom
wouterlucas:feat/webgl-context-loss-recovery
Open

wouterlucas wants to merge 4 commits into
rdkcentral:masterfrom
wouterlucas:feat/webgl-context-loss-recovery

Conversation

@wouterlucas

Copy link
Copy Markdown
Contributor

Port of lightning-js/renderer#930 (WebGL context-loss recovery) to the Lightning 2 architecture.

  • Context state machine on Stage: running|lost|restoring|failed|destroyed, with a generation counter so stale async work is discarded. All GPU submission is suspended while not running (updateFrame/renderFrame guards, upload loss detection, useProgram guard).
  • preventDefault() on webglcontextlost (otherwise no restore event); one recovery frame outside the normal loop before emitting contextRestored. contextRestoreFailed is non-terminal on timeout (stays lost, default 10s via contextRestoreTimeout, 0 disables) and terminal on re-init / recovery-frame failure (failed).
  • Texture restores flow through the existing per-frame upload budget (TextureThrottler.processSome drains the restore queue first); cheap canvas-generated sources reload immediately, image sources stream in throttled. In-flight decodes from before the loss are discarded, never marked failed. Previously evicted textures reload on demand; destroy while lost drops all queued restores.
  • Shader programs recompile in place (stable references); executor VBOs, state-manager caches, compressed-extension handles and the render-texture pool are rebuilt; render-to-texture elements re-render via a forced type-3 update (a type-1 update would keep binding dead result textures).
  • Canvas/2D renderer explicitly out of scope (no listeners attached in c2d mode).

FWIW, L2 has no handleLoopError/emitSafe equivalent: recovery-frame render errors claimed by the stage are reported as contextRestoreFailed and swallowed (the event is the report); anything else rethrows to preserve fail-fast behavior.

Same-source texture reloads now also force a render update — without it, fresh pixels never paint in a static scene (renderFrame only renders when dirty).

Ports lightning-js/renderer#930 semantics to the Lightning 2
architecture: context state machine on Stage, suspend-all-GPU-work
while not running, throttled texture restores through the existing
per-frame upload budget, in-place shader recompiles, generation
counters for stale async work, and a single recovery frame outside
the normal loop before emitting contextRestored. Canvas2D is
explicitly out of scope.
Copilot AI balanced review requested due to automatic review settings October 15, 2026 08:28
@rdkcmf-jenkins

Copy link
Copy Markdown
Contributor

b'## Blackduck scan failure details

Summary: 0 violations, 0 files pending approval, 1 file pending identification.

  • Protex Server Path: /home/blackduck/github/Lightning/622/rdkcentral/Lightning

  • Commit: 45b546c

Report detail: gist'

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved lifecycle, rendering, and resource-restoration defects can prevent reliable recovery.

6 open findings
What changed in this PR

Adds WebGL context-loss recovery to Lightning 2, covering stage lifecycle, texture reloads, and GPU resource rebuilding.

Changes:

  • Suspends rendering during context loss and reports recovery outcomes.
  • Restores textures, shader programs, buffers, and render-to-texture state.
  • Adds recovery tests and an interactive example.
File Description
tests/​context-loss.spec.ts Adds browser recovery and pixel checks.
src/​tree/​TextureThrottler.mjs Drains queued restores within upload budgets.
src/​tree/​TextureSource.mjs Invalidates textures and discards stale loads.
src/​tree/​TextureSource.d.mts Declares source recovery APIs.
src/​tree/​TextureManager.mjs Tracks and reloads invalidated sources.
src/​tree/​TextureManager.d.mts Declares manager recovery APIs.
src/​tree/​TextureManager.contextLoss.test.mjs Tests texture recovery lifecycle.
src/​tree/​Stage.mjs Adds recovery states, events, and rendering guards.
src/​tree/​Stage.d.mts Declares recovery options and events.
src/​tree/​Stage.contextLoss.test.mjs Tests stage recovery transitions.
src/​tree/​Element.mjs Forces repaint after texture reloads.
src/​tree/​Element.contextLoss.test.mjs Tests reload-triggered repainting.
src/​tree/​core/​ElementTexturizer.mjs Rejects obsolete render textures.
src/​tree/​core/​ElementTexturizer.contextLoss.test.mjs Tests render-texture generations.
src/​tree/​core/​CoreContext.mjs Invalidates pooled render textures.
src/​renderer/​webgl/​WebGLShaderProgram.mjs Recompiles programs in place.
src/​renderer/​webgl/​WebGLShaderProgram.contextLoss.test.mjs Tests program restoration.
src/​renderer/​webgl/​WebGLShader.mjs Guards program use during suspension.
src/​renderer/​webgl/​WebGLRenderer.mjs Rebuilds renderer-owned GPU state.
src/​renderer/​webgl/​WebGLRenderer.d.mts Declares renderer restoration.
src/​renderer/​webgl/​WebGLCoreRenderExecutor.mjs Recreates rendering buffers.
src/​platforms/​browser/​WebPlatform.mjs Handles context events and recovery frames.
src/​platforms/​browser/​WebPlatform.d.mts Declares frame-loop suspension.
src/​platforms/​browser/​WebPlatform.contextLoss.test.mjs Tests browser recovery scheduling.
examples/​context-loss/​context-loss.html Demonstrates loss and restoration.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/platforms/browser/WebPlatform.mjs Resolved
Comment thread src/tree/Stage.mjs Resolved
Comment thread src/tree/TextureManager.mjs Resolved
Comment thread src/renderer/webgl/WebGLRenderer.mjs Resolved
Comment thread src/tree/Stage.mjs Resolved
Comment thread src/tree/TextureSource.mjs Resolved
Comment thread src/tree/TextureSource.mjs Fixed
Comment thread src/tree/TextureThrottler.mjs Fixed
- Keep canvas context listeners armed until destroy so a loss while
  stopped still hits preventDefault(); track explicit stop/resume
  intent so recovery never restarts a deliberately stopped loop.
- Recheck context state after texture uploads and after the scene
  update inside renderFrame; reset _updatingFrame on early exit.
- Skip texture uploads when the context is already lost and route
  upload failures through recovery only on loss (rethrow otherwise).
- Recreate CircularPushShader values textures on a new context
  generation instead of binding dead handles.
- Dirty the dummy parent on restore so the recovery frame renders
  even when nothing else does.
- Check the generation before untracking loading sources so stale
  callbacks cannot evict replacement loads.
- Standardize license headers on new test/example files.
Copilot AI balanced review requested due to automatic review settings October 15, 2026 09:05
Comment thread src/tree/TextureSource.mjs Dismissed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Shader-backed image recovery, pending recovery scheduling, and worker-backed preload teardown have unresolved failures.

2 open findings
5 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity resume() can bypass recovery completion and suppress contextRestored

src/​platforms/​browser/​WebPlatform.mjs:108

[P2] Calling resume() (with or without a preceding stop()) after browser restoration but before the recovery animation frame bypasses recovery completion. The stage is already running, so _beginLoop() cancels the recovery frame and starts the normal loop, which never calls onRecoveryFrameCompleted(). This leaves pendingRestoreEvent true and prevents contextRestored from being emitted. Schedule the pending recovery frame before starting the normal loop.

Medium severity Stage destruction crashes when cancelling preloads after worker teardown

src/​tree/​TextureManager.mjs:83

[P2] With useImageWorker enabled, destroying a stage with a pending worker-backed preload throws here. Stage.destroy() destroys the platform first, deleting ImageWorker._worker and _items. This new cancellation then reaches ImageWorker.cancel(), which dereferences _worker.postMessage(), aborting the remaining teardown. Background preloads need not belong to an active element, so detaching the application does not necessarily cancel them first. Cancel pending sources before platform teardown, or make worker cancellation safe after destruction.

🧠 Review effort: Balanced

Comment thread src/tree/TextureSource.mjs Resolved
Recovery frames queue quads whose sources have no native texture yet
while image restores decode. PixelateShader/BoxBlurShader read texture
dimensions per quad, which dereferenced the null handle and failed
recovery terminally. Skip such quads the way never-loaded textures are
skipped, and harden getTextureWidth/Height with a source-dimensions
fallback. Adds a PixelateShader element to the example so the browser
spec covers the path.
Copilot AI balanced review requested due to automatic review settings October 15, 2026 10:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved lifecycle bugs affect recovery completion, manual rendering, stale callbacks, suspended uploads, and worker teardown.

1 open finding
1 resolved since last review
Previously missed (5)

In code that hasn't changed since last review

Medium severity Manual rendering mode omits context recovery listeners

src/​platforms/​browser/​WebPlatform.mjs:88

[P2] Context listeners are attached only by startLoop(), but Stage.init() skips that call when autostart is false. This supported manual-rendering mode therefore has no listener to call preventDefault() on context loss or handle restoration, leaving it unrecoverable. Attach listeners after context initialization independently of starting the automatic loop, and preserve manual-loop intent when recovery completes.

Medium severity Recovery completion is lost when restart cancels the recovery frame

src/​platforms/​browser/​WebPlatform.mjs:108

[P2] Calling stop()/resume() after the restore event but before its recovery animation frame cancels that frame. The stage is already running, so startLoop() enters the normal loop, which never calls onRecoveryFrameCompleted(). Consequently, pendingRestoreEvent remains set and contextRestored is never emitted. Schedule the recovery frame whenever completion is still pending before starting the normal loop.

Medium severity Pending image cancellation crashes after platform teardown

src/​tree/​TextureManager.mjs:83

[P2] With useImageWorker enabled, destroying a stage with an image load still pending can throw here. Stage.destroy() destroys the platform first, which deletes ImageWorker's _worker and _items. This new cancellation then calls ImageWorker.cancel(), which dereferences _worker.postMessage, aborting the remaining teardown. Cancel pending loads before destroying the platform, or make worker cancellation safe after teardown.

Medium severity Async uploads proceed while context recovery is suspended

src/​tree/​TextureManager.mjs:246

[P2] An async text restore can complete after reinitialization or the recovery frame moves the stage to failed. Its callback generation still matches, and TextTexture uses throttle: false, so it reaches this uploader directly. Because the browser context is healthy, GPU allocation and uploads still proceed despite stage suspension. Requeue uploads when _isContextSuspended() is true, without reporting another context loss.

Medium severity Stale source callbacks survive cancellation and generation invalidation

src/​tree/​TextureSource.mjs:270

[P2] A source canceled before context loss is removed from _loadingSources, so invalidation never increments its source generation. Async text draws have no abort callback. If the source starts loading again after restoration, its old callback can pass this check, untrack the replacement load, or report a stale txError. Capture and compare the manager-wide generation too, before changing shared tracking.

🧠 Review effort: Balanced

- Attach context listeners in manual-rendering mode, and keep it
  manual after recovery unless the loop was explicitly running.
- If stop/resume cancels a pending recovery frame, run that frame
  before restarting the normal loop so contextRestored still emits.
- Cancel pending texture loads before platform teardown while keeping
  the original GPU cleanup order; make worker cancellation teardown-safe.
- Requeue async texture completions instead of uploading while the
  stage is suspended or failed.
- Compare manager and source generations so callbacks from sources
  canceled before loss cannot affect replacement loads.
Copilot AI balanced review requested due to automatic review settings October 15, 2026 14:00
@wouterlucas

Copy link
Copy Markdown
Contributor Author

Follow-up to Copilot's 10:58 re-review: all five previously-missed lifecycle findings are fixed on the current head (47d715b): (1) context listeners are armed even with autostart:false and manual mode stays manual; (2) resume reruns a canceled pending recovery frame before the normal loop; (3) pending decodes are canceled before platform/worker teardown, with worker cancellation now teardown-safe and GPU cleanup order preserved; (4) async texture completions requeue instead of uploading when state is suspended/failed; (5) callbacks compare both source and manager generations, including sources canceled before loss. Added regression tests for each path. Validation: 91 Vitest tests passed, tsc clean, Playwright context-loss spec passed. The repeated FossID match is still the same Apache-2.0 @lightningjs/core lineage.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved texture lifecycle issues and cross-cutting GPU recovery changes require fixes and representative browser/device review.

0 open findings

1 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Invalidate render-to-texture sources before recovery rendering

src/​tree/​TextureManager.mjs:174

[P2] Render-to-texture result sources are not in _uploadedTextureSources: ElementTexturizer._getTextureSource() creates them directly, and replaceNativeTexture() bypasses uploads. This branch therefore leaves their pre-loss handles intact. If a consumer is rendered before its producer during recovery, it queues the dead handle; the new null-texture guard cannot reject it. Invalidate these sources through their owning texturizers before recovery rendering, or track them separately. Add coverage with a consumer rendered before its producer.

Medium severity Clear previous load errors when starting texture retries

src/​tree/​TextureSource.mjs:263

[P2] Clear the previous load error when starting a retry. _loadError survives until a successful upload, so context loss during a pending retry cancels and queues it, but both restoration paths then discard it because source.loadError is still set. The active texture remains unloaded with no pending work. Reset the error for the new attempt and add a regression covering loss during a retry.

🧠 Review effort: Balanced

Comment thread src/tree/TextureSource.mjs Dismissed

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants