Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,21 +113,21 @@ previews, and chain in either order without extra dependencies.
`collapse` lives under **SHAPE & MOTION**. The default `seam` mode behaves like
Photoshop content-aware scaling: each frame starts at its original composition,
then its content-aware shrink grows linearly from the first frame to the clip's
end. Low-energy seams are removed from a reduced analysis frame, and the carved
coordinate map is stretched back over the original canvas, so output dimensions
never change. High-gradient regions resist shrinking. Analysis is capped at 192
pixels on the longest side and updates at most five times per second; the
displacement field is smoothed between updates. `mode=elastic` keeps the earlier
seeded, spatially nonuniform elastic warp available as an alternative.
end. Low-energy seams are removed from a reduced analysis frame. The renderer
inverts that seam-removal displacement so the content shrinks, then maps it to
the original canvas dimensions. High-gradient regions resist shrinking.
Analysis is capped at 192 pixels on the longest side and updates at most five
times per second; the displacement field is smoothed between updates.
`mode=elastic` keeps the earlier seeded, spatially nonuniform elastic warp.

| option | range / values | default |
| --- | --- | --- |
| `intensity` | 0-100%, overall scale strength (elastic warp strength in elastic mode) | 65% |
| `horizontal` | 0-100%, share of the maximum width shrink / horizontal warp | 65% |
| `vertical` | 0-100%, share of the maximum height shrink / vertical warp | 45% |
| `intensity` | 0-100%, overall scale strength (elastic warp strength in elastic mode) | 100% |
| `horizontal` | 0-100%, share of the maximum width shrink / horizontal warp | 100% |
| `vertical` | 0-100%, share of the maximum height shrink / vertical warp | 100% |
| `preserve` | 0-100%, content/edge protection | 75% |
| `mode` | seam, elastic | seam |
| `seam_strength` | 0-100%, maximum content-aware shrink by clip end | 60% |
| `seam_strength` | 0-100%, maximum content-aware shrink by clip end | 100% |
| `motion` | 0-100%, elastic motion influence | 50% |
| `smoothness` | 0-100%, spatial field smoothness | 70% |
| `speed` | 0-4 elastic cycles per second | 0.35 |
Expand Down Expand Up @@ -161,18 +161,18 @@ The report measures processor frame time and peak process RSS at 640x360 and
end-to-end playback guarantee. Seam mode also reports carving-update time.

Recorded on this Linux x86_64 host with one OpenCV CPU thread, 120 frames per
case, Python 3.14, NumPy 2.5.4 and OpenCV 5.0 (raw data in
case, Python 3.13.5, NumPy 2.2.4 and OpenCV 4.12.0 (raw data in
`benchmarks/collapse.json`):

| mode | frame size | mean ms/frame | p95 ms/frame | peak process RSS |
| --- | --- | --- | --- | --- |
| elastic | 640x360 | 4.55 | 7.02 | 72.8 MiB |
| elastic | 1280x720 | 11.57 | 12.30 | 119.1 MiB |
| seam | 640x360 | 3.43 | 6.37 | 68.1 MiB |
| seam | 1280x720 | 9.72 | 13.20 | 119.1 MiB |
| elastic | 640x360 | 4.23 | 4.57 | 63.3 MiB |
| elastic | 1280x720 | 9.79 | 10.60 | 110.1 MiB |
| seam | 640x360 | 16.59 | 89.64 | 62.6 MiB |
| seam | 1280x720 | 25.31 | 123.05 | 109.0 MiB |

Seam averages include frames reusing a cached map; carving-update frames alone
averaged 6.68 ms at 360p and 12.87 ms at 720p. Persistent state was about
averaged 89.37 ms at 360p and 109.63 ms at 720p. Persistent state was about
0.38 MiB for elastic and 0.53 MiB for seam mode in these cases. These are
measurements of this fixture on this host, not a promise for every input.

Expand Down
9 changes: 5 additions & 4 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -380,10 +380,11 @@ Shared rules for the stateful effects:
- Test linear progress against clip duration, fixed output size, content preservation, repeatability, bounded state and real CLI/menu renders.
_Done: `collapse` defaults to a content-aware seam scale that begins at the
source composition and grows linearly to its target over probed clip duration.
The low-resolution energy map removes seams from low-detail regions, then the
source-coordinate map is stretched back to the original frame dimensions.
Analysis is capped at 192px; shrink is limited to 50% per axis and refreshed
at <=5Hz with temporal smoothing. The seeded elastic warp remains selectable.
The low-resolution energy map removes seams from low-detail regions, and the
renderer inverts that displacement so salient content shrinks instead of
growing. Each output is mapped to the original frame dimensions. Analysis is
capped at 192px; shrink is limited to 50% per axis and refreshed at <=5Hz with
temporal smoothing. The seeded elastic warp remains selectable.
Duration flows through the shared stateful adapter. Tests cover exact start,
linear midpoint field, end-frame deformation, output shape/audio, resets,
preservation, optional dependencies and TUI/CLI integration. Optional headless
Expand Down
112 changes: 56 additions & 56 deletions benchmarks/collapse.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"recorded_utc": "2026-10-23T22:29:41.163465+00:00",
"recorded_utc": "2026-10-25T14:16:27.765814+00:00",
"platform": "Linux-7.1.13+parrot7-amd64-x86_64-with-glibc2.41",
"logical_cpus": 22,
"scope": "synthetic moving RGB; process() only; fresh process per mode/size; not end-to-end render FPS",
Expand All @@ -10,34 +10,34 @@
"height": 360,
"frames": 120,
"config": {
"intensity": 65,
"horizontal": 65,
"vertical": 45,
"intensity": 100,
"horizontal": 100,
"vertical": 100,
"preserve": 75,
"mode": "elastic",
"seam_strength": 60,
"seam_strength": 100,
"motion": 50,
"smoothness": 70,
"speed": 0.35,
"coherence": 80,
"seed": 17
},
"mean_ms": 4.550746899894875,
"p50_ms": 4.366609497083118,
"p95_ms": 7.016759949146943,
"max_ms": 11.28179499937687,
"processor_fps": 219.74414793824297,
"mean_ms": 4.230598408328963,
"p50_ms": 4.365750500255672,
"p95_ms": 4.567256800373798,
"max_ms": 5.016768999666965,
"processor_fps": 236.37318021754476,
"seam_updates": 0,
"seam_update_mean_ms": null,
"peak_rss_mib": 72.765625,
"peak_rss_mib": 63.31640625,
"persistent_state_mib": 0.375762939453125,
"work_shape": [
108,
192
],
"python": "3.14.0",
"numpy": "2.5.4",
"opencv": "5.0.0",
"python": "3.13.5",
"numpy": "2.2.4",
"opencv": "4.12.0",
"opencv_threads": 1
},
{
Expand All @@ -46,34 +46,34 @@
"height": 720,
"frames": 120,
"config": {
"intensity": 65,
"horizontal": 65,
"vertical": 45,
"intensity": 100,
"horizontal": 100,
"vertical": 100,
"preserve": 75,
"mode": "elastic",
"seam_strength": 60,
"seam_strength": 100,
"motion": 50,
"smoothness": 70,
"speed": 0.35,
"coherence": 80,
"seed": 17
},
"mean_ms": 11.565306316727705,
"p50_ms": 11.400867499105516,
"p95_ms": 12.299060149962315,
"max_ms": 18.616312998346984,
"processor_fps": 86.46550057680967,
"mean_ms": 9.785860641697278,
"p50_ms": 9.651802000007592,
"p95_ms": 10.604852450251201,
"max_ms": 14.228853000531672,
"processor_fps": 102.18825268561746,
"seam_updates": 0,
"seam_update_mean_ms": null,
"peak_rss_mib": 119.12890625,
"peak_rss_mib": 110.08984375,
"persistent_state_mib": 0.375762939453125,
"work_shape": [
108,
192
],
"python": "3.14.0",
"numpy": "2.5.4",
"opencv": "5.0.0",
"python": "3.13.5",
"numpy": "2.2.4",
"opencv": "4.12.0",
"opencv_threads": 1
},
{
Expand All @@ -82,34 +82,34 @@
"height": 360,
"frames": 120,
"config": {
"intensity": 65,
"horizontal": 65,
"vertical": 45,
"intensity": 100,
"horizontal": 100,
"vertical": 100,
"preserve": 75,
"mode": "seam",
"seam_strength": 60,
"seam_strength": 100,
"motion": 50,
"smoothness": 70,
"speed": 0.35,
"coherence": 80,
"seed": 17
},
"mean_ms": 3.429766949830082,
"p50_ms": 2.686837002329412,
"p95_ms": 6.3660287009042795,
"max_ms": 10.528957998758415,
"processor_fps": 291.56499978797166,
"mean_ms": 16.585925233357557,
"p50_ms": 2.023097500568838,
"p95_ms": 89.63626609997846,
"max_ms": 92.01955000025919,
"processor_fps": 60.29208415752432,
"seam_updates": 20,
"seam_update_mean_ms": 6.684701200538257,
"peak_rss_mib": 68.12109375,
"seam_update_mean_ms": 89.37203475011302,
"peak_rss_mib": 62.61328125,
"persistent_state_mib": 0.533966064453125,
"work_shape": [
108,
192
],
"python": "3.14.0",
"numpy": "2.5.4",
"opencv": "5.0.0",
"python": "3.13.5",
"numpy": "2.2.4",
"opencv": "4.12.0",
"opencv_threads": 1
},
{
Expand All @@ -118,35 +118,35 @@
"height": 720,
"frames": 120,
"config": {
"intensity": 65,
"horizontal": 65,
"vertical": 45,
"intensity": 100,
"horizontal": 100,
"vertical": 100,
"preserve": 75,
"mode": "seam",
"seam_strength": 60,
"seam_strength": 100,
"motion": 50,
"smoothness": 70,
"speed": 0.35,
"coherence": 80,
"seed": 17
},
"mean_ms": 9.723339424726873,
"p50_ms": 9.247966498151072,
"p95_ms": 13.197447299717169,
"max_ms": 15.983984998456435,
"processor_fps": 102.84532466869939,
"mean_ms": 25.305225183334795,
"p50_ms": 9.182545500152628,
"p95_ms": 123.04726984962144,
"max_ms": 130.75589300024149,
"processor_fps": 39.51753018418377,
"seam_updates": 20,
"seam_update_mean_ms": 12.868869349949819,
"peak_rss_mib": 119.125,
"seam_update_mean_ms": 109.627773550028,
"peak_rss_mib": 109.03125,
"persistent_state_mib": 0.533966064453125,
"work_shape": [
108,
192
],
"python": "3.14.0",
"numpy": "2.5.4",
"opencv": "5.0.0",
"python": "3.13.5",
"numpy": "2.2.4",
"opencv": "4.12.0",
"opencv_threads": 1
}
]
}
}
8 changes: 4 additions & 4 deletions effex.py
Original file line number Diff line number Diff line change
Expand Up @@ -1918,11 +1918,11 @@ def build_temporal_sediment(ctx: EffectContext) -> Plan:

COLLAPSE_PARAMS = (
Param("intensity", "intensity", "how strongly the displacement is applied",
"percent", 65, minimum=0, maximum=100, step=5),
"percent", 100, minimum=0, maximum=100, step=5),
Param("horizontal", "horizontal scale", "horizontal deformation amount",
"percent", 65, minimum=0, maximum=100, step=5),
"percent", 100, minimum=0, maximum=100, step=5),
Param("vertical", "vertical scale", "vertical deformation amount",
"percent", 45, minimum=0, maximum=100, step=5),
"percent", 100, minimum=0, maximum=100, step=5),
Param("preserve", "content preservation", "protect strong edges from collapse",
"percent", 75, minimum=0, maximum=100, step=5),
Param("mode", "mode", "progressive content-aware scaling or elastic warping",
Expand All @@ -1932,7 +1932,7 @@ def build_temporal_sediment(ctx: EffectContext) -> Plan:
("elastic", "elastic warp", "smooth content-weighted CPU remapping"),
)),
Param("seam_strength", "shrink amount", "maximum content-aware shrink by the end of the clip",
"percent", 60, minimum=0, maximum=100, step=5),
"percent", 100, minimum=0, maximum=100, step=5),
Param("motion", "motion influence", "amplify elastic deformation in moving regions",
"percent", 50, minimum=0, maximum=100, step=5),
Param("smoothness", "field smoothness", "larger, smoother deformation regions",
Expand Down
9 changes: 5 additions & 4 deletions stateful_effects.py
Original file line number Diff line number Diff line change
Expand Up @@ -293,8 +293,8 @@ class ElasticCollapse:
MAX_WORK_SIDE = 192
MAX_STATE_BYTES = 4 * 1024 * 1024
DEFAULTS = {
"intensity": 65, "horizontal": 65, "vertical": 45, "preserve": 75,
"mode": "seam", "seam_strength": 60, "motion": 50,
"intensity": 100, "horizontal": 100, "vertical": 100, "preserve": 75,
"mode": "seam", "seam_strength": 100, "motion": 50,
"smoothness": 70, "speed": 0.35, "coherence": 80, "seed": 0,
}

Expand Down Expand Up @@ -383,7 +383,7 @@ def displacement(self) -> Any:
"""A copy of the applied normalized (dx, dy) field for diagnostics."""
if self._field is None:
return None
amount = (self._progress if self._config["mode"] == "seam"
amount = (-self._progress if self._config["mode"] == "seam"
else self._config["intensity"] / 100.0)
return self._field * amount

Expand Down Expand Up @@ -501,7 +501,8 @@ def process(self, frame, frame_index: int, timestamp: Optional[float],
or not (self._config["horizontal"] or self._config["vertical"])):
return frame.copy()
full = cv.resize(self._field, (width, height), interpolation=cv.INTER_LINEAR)
amount = (self._progress if self._config["mode"] == "seam"
# Seam removal's resize map expands the retained content; invert it to shrink.
amount = (-self._progress if self._config["mode"] == "seam"
else intensity)
map_x = full[:, :, 0] * (amount * max(0, width - 1))
map_y = full[:, :, 1] * (amount * max(0, height - 1))
Expand Down
18 changes: 17 additions & 1 deletion tests/collapse_regression.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,11 @@ def test_collapse_is_registered_with_all_controls(self):
effect = effex.EFFECT_BY_KEY["collapse"]
self.assertEqual({p.key for p in effect.params}, set(stateful_effects.ElasticCollapse.DEFAULTS))
self.assertIs(effect.frame_processor, stateful_effects.ElasticCollapse)
self.assertEqual(effex.default_opts(effect)["mode"], "seam")
defaults = effex.default_opts(effect)
self.assertEqual(defaults["mode"], "seam")
self.assertEqual([defaults[key] for key in
("intensity", "horizontal", "vertical", "seam_strength")],
[100, 100, 100, 100])

def test_seam_mode_is_distinct_and_zero_strength_disables_it(self):
frame = self.frame()
Expand Down Expand Up @@ -205,6 +209,18 @@ def test_content_aware_scale_grows_linearly_over_clip_duration(self):
self.assertEqual(final.shape, frame.shape)
self.assertGreater(float(np.abs(final.astype(float) - frame).mean()), 4)

def test_content_aware_scale_shrinks_the_salient_subject(self):
frame = np.zeros((64, 96, 3), dtype=np.uint8)
frame[20:44, 30:66] = 255
processor = self.processor(
mode="seam", intensity=100, horizontal=100, vertical=0,
preserve=100, seam_strength=100, coherence=0, _clip_duration=1,
)
out = processor.process(frame, 30, 1.0, {})
ys, xs = np.where(out.mean(axis=2) > 128)
self.assertEqual(out.shape, frame.shape)
self.assertLess(int(np.ptp(xs)) + 1, 36)

def test_seam_updates_are_rate_limited_and_deterministic(self):
frame = self.frame()
a, b = self.processor(mode="seam"), self.processor(mode="seam")
Expand Down
5 changes: 3 additions & 2 deletions tests/menu_smoke.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
UP = b"\x1bOA"
DOWN = b"\x1bOB"
RIGHT = b"\x1bOC"
LEFT = b"\x1bOD"
ENTER = b"\r"


Expand Down Expand Up @@ -860,8 +861,8 @@ def scenario_collapse(c: Checker):
press_and_wait(d, UP)
c.check("collapse elastic: legacy warp mode can be selected",
"elastic warp" in last_frame(d))
press_and_wait(d, RIGHT)
c.check(f"collapse {mode}: intensity is editable", "70%" in last_frame(d))
press_and_wait(d, LEFT)
c.check(f"collapse {mode}: intensity is editable", "95%" in last_frame(d))
if mode == "seam":
for _ in range(4):
press_and_wait(d, DOWN)
Expand Down
Loading