Skip to content

Softer shadows in raytracing and RTAO - #7695

Open
The-E wants to merge 18 commits into
imgui-profilerfrom
feat/softer-shadows
Open

The-E wants to merge 18 commits into
imgui-profilerfrom
feat/softer-shadows

Conversation

@The-E

@The-E The-E commented Aug 7, 2026

Copy link
Copy Markdown
Member

Recreated from #7653 as part of the stack master ← #7652 ← this ← #7671 ← #7673 (moved to a same-repo branch so gh stack link can register a real GitHub Stack). See #7653 for original review history.


Adds methods for softening RT shadows. This mechanism works by setting an apparent light size either in the mission file, a lab override, or by deriving it from the sun bitmap. The current implementation limits the apparent size to 1.5°; this is an artificial constraint based around the limits of raytraced shadows - the wider the sun is, the sparser the rays to check whether something is in shadow become, and the more grainy the shadows appear.
There are table settings and FRED/qtFRED controls to set this in missions.

And because the methods required to do soft shadows are mostly identical to what you need to do in order to do raytraced Ambient Occlusion, I threw in that effect as well - but I seriously cannot recommend anyone use it. The visual impact is only noticeable when you really look for it, and the performance cost is pretty large.

@The-E
The-E force-pushed the feat/softer-shadows branch from 01baf0d to 77aa029 Compare August 7, 2026 16:55
@The-E
The-E force-pushed the feat/softer-shadows branch from 77aa029 to 603fe2e Compare August 13, 2026 17:02
@The-E
The-E force-pushed the feat/softer-shadows branch 2 times, most recently from 61e8ab9 to da190af Compare August 20, 2026 14:35
@The-E
The-E force-pushed the feat/softer-shadows branch from da190af to efc13ac Compare August 24, 2026 15:41
@The-E The-E added enhancement A new feature or upgrade of an existing feature to add additional functionality. graphics A feature or issue related to graphics (2d and 3d) vulkan Issues and Features related to the vulkan render backend labels Aug 25, 2026
@The-E
The-E force-pushed the feat/softer-shadows branch from efc13ac to 0380235 Compare August 26, 2026 17:32
@The-E
The-E force-pushed the feat/softer-shadows branch from 0380235 to 1633eb6 Compare August 29, 2026 07:32
@The-E
The-E force-pushed the feat/softer-shadows branch from 1633eb6 to 2519074 Compare August 29, 2026 09:17
@The-E
The-E force-pushed the feat/softer-shadows branch from 2519074 to 5da5ab0 Compare September 6, 2026 06:31
@The-E
The-E force-pushed the feat/softer-shadows branch from 5da5ab0 to 8e06e96 Compare September 10, 2026 17:46
Adds ImPlot as a build dependency, to be used for the upcoming ImGui
frame profiler overlay's frametime graph and pie chart.
Replaces the old text-dump frame profile display (-profile_frame_time)
with a runtime-toggleable ImGui/ImPlot overlay showing average/median
frametime, a scrolling frametime graph, and a pie chart of the top
contributors to frame time. Toggled via a new "Frame Profiler Overlay"
option, which also seeds from -profile_frame_time for backward
compatibility and now drives frame profiling collection at runtime
instead of only at startup.
- Integrated per-backend diagnostic counters into the ImGui frame profiler overlay, triggered by `-gr_debug`.
- Consolidated and cleaned up Vulkan rendering state and descriptor management for compatibility with ImGui's Vulkan pipeline.
- Enabled uniform buffer and draw statistics reporting for enhanced performance insights during debugging.
The-E and others added 15 commits September 14, 2026 18:33
- Adjust copyright notices for extended license coverage (2025-2026).
- Modernize constants with `constexpr` replacements, improving type safety and clarity.
- Enforce `nullptr` usage over `NULL` throughout, aligning with modern C++ practices.
- Add various enhancements to ImPlot legends, markers, and axes behavior.
- Introduce additional helper methods for time handling and timestamp calculations.
- Address UI and typo improvements in comments and documentation, refining accuracy.
- Introduced `accumulate_self_times` for single-pass exclusive time computation, improving performance and accuracy.
- Updated `FrameProfiler` and overlay snapshot logic to leverage `Category`'s stable `getId` and `getCount`.
- Replaced pie chart with a stacked bar for frame budget visualization, optimizing rendering cost.
- Cached `get_tid` and `get_pid` with thread-local storage to minimize syscall overhead.
- Added support for unit testing frame profiling functionality and included new test cases.
- Improved code comments and clarified documentation.
…Vulkan

The Material and PerDraw descriptor sets were reallocated and fully rewritten on
essentially every draw call. model_draw_list::render_buffer rebinds ModelData with
a fresh uniform buffer offset for each queued draw, and that offset was part of the
Material set's memoization key, so the cache in VulkanDrawManager::applyMaterial
missed every time - taking a frame-pool allocation, a full template write, the
16-entry material texture resolution and a vkUpdateDescriptorSets with it.
GenericData/Matrices did the same to the PerDraw set. renderShadowDraw was worse
still: all three sets allocated and rewritten per shadow draw, with no memoization
at all.

A large asteroid field (Solaris m21) measured 2566 descriptor sets and 19107
descriptor writes per frame at 1606 model draws, and grew an extra descriptor pool
chunk every frame.

Declare the bindings whose offset moves per draw - MaterialBinding::ModelData,
MaterialBinding::ShadowMapData, PerDrawBinding::GenericData and
PerDrawBinding::Matrices - as eUniformBufferDynamic. DescriptorWriter::setBuffer
now splits a dynamic binding's offset out of the descriptor, writing
{buffer, 0, range} and stashing the offset for vkCmdBindDescriptorSets'
pDynamicOffsets, so an offset-only change leaves the set's contents identical and
the memoization caches start hitting. Shader sources are unchanged.

VulkanStateTracker::bindDescriptorSet now compares the dynamic offsets as well as
the set handle. This is required, not an optimization: the common case is now the
same set rebound with a new offset, and the old handle-only redundancy check would
have skipped that bind and fed every draw the previous draw's uniforms.

renderShadowDraw memoizes its three sets the way applyMaterial does. It does not
invalidate applyMaterial's caches - those record what a set contains, the shadow
pass allocates its own sets rather than overwriting theirs, and it binds through
the state tracker, so the next material cache hit still rebinds correctly.

Also switch allocateFrameSet to the non-allocating allocateDescriptorSets overload;
the vector-returning one heap-allocated once per set, thousands of times a frame.
MAX_SETS_PER_POOL stays at 1024 - the per-frame pool growth it used to log was a
symptom of the churn, and one chunk now covers the scene outright.

Measured on that mission in a Debug build, A/B on the same tree with and without
this change:

  descriptor sets/frame  @ ~1370 draws  2566   -> 250-345
  descriptor writes/frame                19107 -> ~1700-2300
  median frametime                       92.9/83.6 ms -> 72.0/66.5 ms
  Render Buffer (profiler overlay)       24.68 ms (30.8%) -> 12.73 ms (18.4%)
  Build Shadow Map                       5.44 ms -> off the top-5 entirely

Release builds already inlined most of this away and are unchanged, as expected.
Verified with the validation layers (sync validation enabled): no new VUIDs versus
the same build without the change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Consolidated dynamic descriptor offset logic into `DescriptorWriter::bindSets`, replacing repetitive `vkCmdBindDescriptorSets` patterns across Vulkan modules.
- Eliminated redundant descriptor binding operations for Material and PerDraw sets by centralizing logic in `bindSets`.
- Simplified descriptors by directly splitting offset changes from descriptor contents, avoiding unnecessary updates.
- Updated comments and documentation to align with the new approach while maintaining reference consistency.
- Applied optimizations across Vulkan modules (`VulkanPostProcessing`, `VulkanDescriptorManager`, etc.) to reduce per-draw overhead and improve readability.
- Added `topologySupportsPrimitiveRestart` to determine restart eligibility for each topology.
- Updated `primitiveRestartEnable` to be conditionally set based on topology, addressing compatibility issues with Metal/MoltenVK.
- Prevented crash scenarios on macOS by ensuring pipelines are valid for strip/fan topologies requiring primitive restart.
…dings

- Introduced `DYNAMIC_SLOT_INVALID` sentinel to replace the throw in `dynamic_offset_slot` for MSVC compatibility.
- Added `static_assert` checks to validate binding constants, ensuring compile-time validation for dynamic offsets.
- Updated comments to clarify design rationale and avoid potential misuse.
The memory panel showed no texture bytes and no geometry bytes under
OpenGL. Only the depth renderbuffers went into the render target group.
All three groups now give a value.

Textures: GL_texture_bytes_used holds the bytes of every live texture
cache slot. opengl_texture_release_bytes() removes the bytes of a slot at
each site that drops its texture data. The bitmap render targets record
the bytes of their color texture.

Geometry: the unused GL_vertex_data_in global becomes
GL_geometry_bytes_used, which counts only vertex and index buffers. This
matches the geometry group of the Vulkan backend.
gr_opengl_delete_buffer() sets the size of the buffer to 0, because
opengl_destroy_all_buffers() calls it a second time for a buffer that the
game already deleted.

Render targets: three new functions add the scene textures, the
post-processing textures, and the shadow map to the group. They sum the
bytes at the moment of the call from the textures that exist, thus they
need no bookkeeping at the create and delete sites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The category registry kept a Category pointer for each id. The comment said that all categories
are global statics, but that is not correct. The Lua API tracing_category constructs a Category
and copies it into a script-owned object. The registry then kept a pointer to the dead temporary.
build_overlay_snapshot called getName() on that pointer, made an SCP_string from the garbage
char*, and the process stopped with SIGABRT.

The registry now keeps the name by value and never keeps a Category pointer. getById() becomes
getNameById(), which is all that build_overlay_snapshot needs. A copy of a Category keeps the id
of the original, which stays correct because the id only selects a name.

Add two tests: one shows that the name stays available after the Category object is dead, and one
drives the full path that failed, from processEvent to the overlay snapshot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Updated `trace_event` to store `category_id` instead of a pointer to prevent dangling references to category objects that may already be destroyed.
- Modified all references to category names to use `Category::getNameById()` instead of dereferencing pointers.
- Expanded memory statistics in the profiler overlay with new sound, Lua, and OS-level memory usage tracking.
- Added new tests to verify proper use of category IDs in frame profiling and event tracing.
- Improved memory safety and stability across tracing, profiling, and overlay systems.
…ular size

Raytraced shadow penumbras needed a hand-authored $SunAngularSize: in
stars.tbl, which no shipped content sets -- so in practice nothing ever
got a penumbra, and the shadow-mapped path's $Shadow Smoothness Factor:
had no relation to it at all. Derive a size from the sun bitmap instead
and drive both methods from the same number.

g3_render_rect_screen_aligned_2d() sizes the sun quad so its rad is the
tangent of the half-angle it subtends, making tan(angular radius) equal
to 0.05 * the mission's +Scale: * the emitting disc's fraction of the
bitmap. That fraction is the area of the pixels at or above 90% of the
brightest one, converted to an equivalent radius. Across 90 sun bitmaps
from retail, the MediaVPs, Blue Planet and BtA it lands at a median of
0.261 (retail's chunkier art at 0.487), and the threshold acts as a
constant factor rather than a per-bitmap judgement call -- Spearman rho
0.96 between a 90% and a 50% cutoff.

Taken literally that yields suns 3.9-8.5x Sol's apparent diameter once
+Scale: is folded in (median 1.55, up to 5.0 in retail's own missions),
so the measurement is scaled by 0.25 and clamped to 1.5 degrees, putting
typical content near Sol. Derivation is the default: $SunAngularSize: 0
asks for hard shadows, an explicit value still wins, and missions can
also set +AngularSize: on an individual sun to override both, with FRED
and QtFRED UI (validation and fallback included) to author it. Bitmaps
that can't be read -- and the deliberately blank sun bitmaps mods ship
to get a light source with no visible disc -- fall back to hard shadows
rather than to the widest possible penumbra.

That angular size now sizes the raytraced penumbra cone directly
(traceShadowRayCone() in shadows.sdr), with sample count following the
same Shadow_quality tier that picks the shadow map's resolution (1 ray
at Low, up to 16 at Ultra) rather than being a separate setting. The
shadow-mapped path's tabled smoothness values were tuned by eye against
Sol-sized retail/MediaVPs art, so shadow_smoothness_scale() in
shadows.cpp scales them by the active sun's size relative to Sol,
clamped to what the shadow map can represent (shadow_clamp_smoothness()).
The two methods still disagree about occluder distance -- the raytracer
uses the real one, the shadow map's tabled values stand in for a blocker
distance pinned to the cascade's own extent -- so RT_SHADOW_SUN_SIZE_CALIBRATION
in shadows.sdr is a constant factor that makes a given $SunAngularSize:
read as roughly the same softness either way; it's the number to revisit
if the shadow map ever grows a real blocker search.

Also adds RTAO (raytraced ambient occlusion) sharing the same TLAS as
the shadow rays, with lighting-profile-driven radius/strength, and lab
controls for RT shadow quality (Low/High), the local light cap, RT
shadow samples, RTAO, and an angular-size override for comparing both
shadow methods against the same sun. All are safe to change live: they
only gate which lights are picked as shadow casters or scale existing
uniforms while filling the per-frame light data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shadow-mapped path's penumbra width was a fixed per-cascade value
(shadow_smoothness_scale()'s ceiling), so every occluder blurred its
shadow by the same amount regardless of how close it actually was to
the receiver -- unlike the raytraced path, which gets that for free
from traceShadowRayCone()'s real occluder distance. Close that gap
with PCSS: search nearby texels for occluders, then scale the filter
radius by how far away they were.

The blocker search (pcssBlockerSearch() in shadows.sdr) needs raw,
uncompared depth reads, which the existing PCF sampler can't do -- it's
bound with a compare mode for the hardware shadow-compare instructions
the filtered sampling itself relies on. So shadow maps now get a second
sampler, shadow_map_raw, bound to the same texture with compare mode
off; on OpenGL that needs a second sampler object and GL 3.3
(shadow_contact_hardening_supported()), Vulkan can always bind it.
Where the raw sampler isn't available, or the new Graphics.ShadowContactHardening
option is off, shadows_start_render() sets Shadow_penumbra_scale[cascade]
to -1 as a sentinel and pcssPenumbraRadius() (shadows.sdr) falls straight
back to the fixed radius, bit-identical to the pre-contact-hardening
behavior and skipping the extra texture reads.

When it is on, shadows_start_render() computes each cascade's penumbra
scale from its own frustum extents and the sun's tanθ; the shader turns
that plus the blocker search's average depth into a per-pixel radius,
clamped to [1 texel, the old fixed ceiling]. A blocker search that finds
nothing doesn't fall back to "no shadow" -- a sparse 8-tap search can
miss a thin occluder, so it instead runs the full filter at the hardest
(1-texel) radius, trading a touch of extra blur for not popping a real
shadow to nothing.

Drops RT_SHADOW_SUN_SIZE_CALIBRATION accordingly: that constant existed
only to fake a blocker-distance-driven penumbra on the shadow-mapped path
by pre-scaling the raytraced cone to match it, and the shadow-mapped path
now has a real blocker search of its own, so the raytraced cone goes back
to its physically correct size.
@The-E
The-E force-pushed the feat/softer-shadows branch from 8e06e96 to 53ca00e Compare September 14, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement A new feature or upgrade of an existing feature to add additional functionality. graphics A feature or issue related to graphics (2d and 3d) vulkan Issues and Features related to the vulkan render backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant