Football analyzer pipeline by zoq · Pull Request #19 · collabora/gst-python-ml

zoq · 2026-04-28T13:19:39Z

Support player detection, referee, and ball detection, within a single model
Support camera movement compensation
Custom drawing, of the results

zoq · 2026-04-28T13:20:55Z

This one requires a custom trained model, do we make it part of the project, or do we upload it to HF?

aaron-boxer · 2026-04-28T19:50:10Z

Thanks for the PR! let's make the model a part of the project for now, if not too big.

boxerab · 2026-05-01T22:11:49Z

one request - is it possible to move all the module level imports to the method level instead ? This will speed up pipeline startup as gstreamer scans all python scripts for elements at startup, and module level imports are executed, so we want to avoid these imports. The other elements in the framework all avoid module level imports. 🙏

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

…ine with ONNX/YOLO class-name and FP16 handling, plus the football models. Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

…a Yocto layer for the on-board gst-python-ml stack. Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

boxerab · 2026-06-16T09:51:49Z

Looks good, thanks! I think it would be good to have an extern directory at the top level of the project for external dependencies, and to place rzv2h folder in extern. Then LGTM.

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

…on the original-resolution frame, expose the detector's confidence/NMS thresholds, and harden the overlay's kit-colour referee detection, box de-duplication, missed-detection bridging and circle smoothing. Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

gst elements are just thin shells around gst base classes

Replace the inline MuxedBufferProcessor with frameio.read_frames (the shared backend frame I/O the sibling video leaves already use). Behavior is identical: the gst frameio wraps the same MuxedBufferProcessor call. Removes a direct gst-utility dependency from the element shell.

Replace inline MuxedBufferProcessor with frameio.read_frames and return the backend's FlowReturn instead of Gst.FlowReturn in the transform body (Gst is still used for element registration). Behavior-identical.

Replace inline MuxedBufferProcessor with frameio.read_frames. Behavior- identical (gst frameio wraps the same call). Not pipeline-smoke-tested here (no caption model downloaded); import + lint verified.

Replace inline MuxedBufferProcessor with frameio.read_frames. Behavior- identical; import + lint verified (no LLM model downloaded to smoke-test).

The source framerate was hardcoded to 30/1; expose it as a 'num/denom' property (threaded into frame I/O for muxed-stream frame timing). Defaults to 30/1, so existing pipelines are unchanged.

confluent_kafka was imported in the same guarded block as gi, so when the optional kafka client was absent the block failed and Gst was never imported, making the unguarded 'class KafkaSink(Gst.Element)' raise NameError during the plugin scan. Import Producer lazily in initialize_producer (where it is used) instead, so a missing kafka client no longer breaks the scan and only matters when a producer is created.

Split the object-detector per-frame path into a backend-agnostic process_frames(frames, num_sources, fmt, target) hook (inference + metadata, identical on any backend) and a backend driver that does frame extraction + error mapping. The gst driver moves to VideoTransform's do_transform_ip (extract via frameio, run process_frames, map to FlowReturn), so every gst video element that supplies process_frames is driven uniformly; base_objectdetector now supplies only process_frames. This is the seam the g2g backend's g2g_process already calls, so the same detector logic runs on either backend. Validated on gst: pyml_yolo (Torch/CUDA + NVDEC + tracking, 20-23 persons/frame) and pyml_objectdetector (ONNX/CPU) both run clean through the hoisted driver.

Import the base class at module top level and register the GStreamer factory only under GSTML_BACKEND=gst, so the leaf imports cleanly under a non-gst backend (where gi is present but the class is built on shims).

Add the g2g element backend skeleton selectable via GSTML_BACKEND=g2g, with aggregator, analytics, frame I/O, transform/video-transform shims, and accompanying tests. Load the model on the first frame even when the device pre-created the engine, and convert RGBA->RGB before inference.

g2g element backend

registration moves behind backend.register_gst_element, called only under the gst backend; GObject now comes from backend wherever the g2g shim can stand in for it.

the five audio and text families (transcribe, translate, llm, separate, tts) share one process_payload hook that both backends drive, and each states its pad caps once instead of spelling them for gst and for g2g separately. no element builds a Gst object at import time any more, which segfaulted the interpreter under PYML_BACKEND=g2g and took the caption and vlm families down with it.

the ml elements no longer import gi unless the gst backend is selected, and captions get a process_frames seam so they run hosted too. the g2g aggregator takes the same process_frames shape as the transform, and the ml property declarations move into one ml_property_namespace so the two backends cannot drift. emitted payload frames inherit the anchor's meta and count sequences on from the input's, a detection staged on a stream with no pixels is dropped rather than becoming a zero bbox, and the frameio and analytics bindings are per-thread since the host runs one thread per element. pyml_launch stops applying its defaulted-property table to hosted elements, which was eating pyml_kafkasink's sync.

the engine import sat ahead of the base class in one try, so a missing optional dependency skipped the base too and the class body raised NameError instead of degrading. the import test now looks for the shadowed gi's message rather than any stderr at all, so an unrelated degradation warning does not read as an element reaching for GStreamer.

torch moves 2.11 to 2.13 and torchvision follows it, so the ml path changed. verified by the unit tests and lint only, not the gpu pipeline suites. diskcache has an advisory with no fix, so it stays.

the old floor let uv keep pytest 8.4.2 on python below 3.14. executorch moves 1.2 to 1.4 to go with it, which drops the test plugins it used to pull in and nothing here uses.

the run only exited early on gst, where a leaky queue was inserted after filesrc and cut the input to 65 of 5554 frames. g2g got no such queue, ran the whole video, and hit the timeout, so the two backends were never compared on the same work. both now run to the cap and pass if the log is clean. the gap counts recorded here came from that comparison, so they go until the suite is run again.

resolve the yolo and base_objectdetector conflicts on master's split task/engine layout, register the football elements through the backend shims, move cv2/numpy/ultralytics imports into the methods, move rzv2h under extern, and fetch the models from the hugging face hub instead of git lfs.

boxerab · 2026-09-14T20:01:06Z

Thanks again for the PR!

boxerab force-pushed the master branch 2 times, most recently from 0be9a68 to 44e42dd Compare May 3, 2026 16:26

improved football anylyzer pipeline.

59d1e51

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

zoq force-pushed the soccer-analyzer branch 2 times, most recently from f44c413 to 59d1e51 Compare June 16, 2026 01:54

zoq added 4 commits June 16, 2026 02:05

Add streaming football overlay (pyml_football_overlay) and DRP-AI eng…

cfbb60d

…ine with ONNX/YOLO class-name and FP16 handling, plus the football models. Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

Add RZ/V2H DRP-AI tooling: YOLO INT8 compile, C++ runtime pybind and …

bb0e589

…a Yocto layer for the on-board gst-python-ml stack. Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

Fix split semicolon-joined statements style.

82e3c49

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

Apply black formatting to football_overlay.py and football_analyzer.py.

d090e87

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

zoq and others added 17 commits June 17, 2026 03:18

Improve overlay settings.

809951a

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

Add buffer to combat slow inference.

f43da7d

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

Add README with some settings.

32e846c

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

Switch from tracking overlay to detection overlay and apply ema filter.

bea1428

Signed-off-by: Marcus Edel <marcus.edel@collabora.com>

refactor: separate out gst-independant code

ab40010

gst elements are just thin shells around gst base classes

refactor buffer I/O

927d46f

inference: extract frames through backend frame I/O

67b0f3a

Replace inline MuxedBufferProcessor with frameio.read_frames and return the backend's FlowReturn instead of Gst.FlowReturn in the transform body (Gst is still used for element registration). Behavior-identical.

caption: extract frames through backend frame I/O

0d49bf2

Replace inline MuxedBufferProcessor with frameio.read_frames. Behavior- identical (gst frameio wraps the same call). Not pipeline-smoke-tested here (no caption model downloaded); import + lint verified.

llm_stream_filter: extract frames through backend frame I/O

a5f20c3

Replace inline MuxedBufferProcessor with frameio.read_frames. Behavior- identical; import + lint verified (no LLM model downloaded to smoke-test).

objectdetector: expose framerate property

5dc0f5a

The source framerate was hardcoded to 30/1; expose it as a 'num/denom' property (threaded into frame I/O for muxed-stream frame timing). Defaults to 30/1, so existing pipelines are unchanged.

objectdetector: gate gst factory registration on the active backend

206ca29

Import the base class at module top level and register the GStreamer factory only under GSTML_BACKEND=gst, so the leaf imports cleanly under a non-gst backend (where gi is present but the class is built on shims).

g2g: drop torch dep from backend test

59be4f4

aaron-boxer and others added 25 commits August 11, 2026 10:24

ci: run g2g backend tests

f0888ab

Merge pull request collabora#20 from collabora/g2g

0a0f358

g2g element backend

readme: drop redundant torch install in uv setup

790f04a

g2g: stage tracking ids and relations

d65906b

yolo: gate gst registration on the backend

5e3ee9d

don't unwrap analytics handles in logs

f884a6a

gate gst registration in every element

fc44cd5

registration moves behind backend.register_gst_element, called only under the gst backend; GObject now comes from backend wherever the g2g shim can stand in for it.

stop failing pipeline tests on benign warnings

25df04c

point remote llm examples at the /v1 endpoint

7133d44

register the serial marker, allow a longer pipeline timeout

37c605f

publish the class-name table to the host

f63f4b6

add pyml-launch, one pipeline on either backend

8022c94

keep the formatting black wants

ebc8d06

skip the import checks where pygobject is absent

80e28c9

record the open g2g backend gaps

028edab

record the measured g2g pipeline gaps

a2e21c4

bump the packages with open advisories

6a3b3e4

torch moves 2.11 to 2.13 and torchvision follows it, so the ml path changed. verified by the unit tests and lint only, not the gpu pipeline suites. diskcache has an advisory with no fix, so it stays.

let renovate open the dependency updates

e9044ad

raise the pytest floor past its advisory

8d19f85

the old floor let uv keep pytest 8.4.2 on python below 3.14. executorch moves 1.2 to 1.4 to go with it, which drops the test plugins it used to pull in and nothing here uses.

skip football weights already on disk

b633652

boxerab merged commit c89ae87 into collabora:master Sep 14, 2026
1 check passed

Navigation Menu

Football analyzer pipeline - #19

Football analyzer pipeline#19
boxerab merged 47 commits into
collabora:masterfrom
zoq:soccer-analyzer

zoq commented Apr 28, 2026

Uh oh!

zoq commented Apr 28, 2026

Uh oh!

aaron-boxer commented Apr 28, 2026 •

edited

Loading

Uh oh!

boxerab commented May 1, 2026

Uh oh!

boxerab commented Jun 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

boxerab commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Conversation

zoq commented Apr 28, 2026

Uh oh!

zoq commented Apr 28, 2026

Uh oh!

aaron-boxer commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boxerab commented May 1, 2026

Uh oh!

boxerab commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

boxerab commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

aaron-boxer commented Apr 28, 2026 •

edited

Loading

boxerab commented Jun 16, 2026 •

edited

Loading