Football analyzer pipeline

.github/workflows/ci.yml

@@ -26,6 +26,13 @@ jobs:
       - name: Run black check
         run: black --check plugins/ tests/ utils/
 
+      # Scoped to this file on purpose: test_pipelines.py raises on import when
+      # gst-launch-1.0 is missing, which would fail collection for the whole dir.
+      - name: Run g2g backend tests
+        run: |
+          pip install pytest numpy
+          pytest tests/test_g2g_backend.py -q
+
   build:
     needs: lint
     strategy:

CONTRIBUTING.md

@@ -97,7 +97,7 @@ For pipeline testing:
 
 ```bash
 gst-inspect-1.0 python  # Verify all elements load
-gst-launch-1.0 filesrc location=data/people.mp4 num-buffers=5 \
+python pyml-launch.py filesrc location=data/people.mp4 num-buffers=5 \
   ! decodebin ! videoconvert ! videoscale \
   ! "video/x-raw,format=RGB,width=640,height=640" \
   ! pyml_objectdetector engine-name=onnx model-name=yolo11n.onnx device=cpu \

DESIGN_TODO.md

@@ -0,0 +1,53 @@
+# DESIGN_TODO
+
+A terse catalogue of open tasks only. Gaps on the host side of the glass2glass
+Python-element host are tracked in that repo's `DESIGN_TODO.md`, under
+"Python-element host", not here.
+
+## g2g backend coverage
+
+- **How many README pipelines run under `PYML_BACKEND=g2g` needs measuring.**
+  Run `tests/test_pipelines.py` under each backend and compare: one that passes
+  on gst and fails on g2g is a gap, one that fails on both is the environment.
+  Only the error categories count as gaps. `pipeline error: Hardware(Other)` is
+  how g2g reports a hosted element raising, so each needs its log in
+  `tests/logs` read to name the cause. Known so far: `pyml_kafkasink` calls
+  `Gst.Pad` APIs directly and dies on `Gst.init`, `demo_soccer`'s engine raises
+  `TypeError: MLEngine.__init__() got an unexpected keyword argument 'device'`,
+  and `pyml_streammux` is refused with `pyelement: more than one input links
+  here, but it is not a registered muxer`. The suite wants the GPU for about 20
+  minutes per backend, so run one backend at a time on a 6 GB card and leave the
+  machine otherwise idle, including between backends: a model still resident
+  from the previous run fails the next one at preroll.
+
+- **Eleven elements have no per-frame seam, so they cannot run on g2g at all.**
+  `alert`, `tracker`, `vad`, `clap`, `overlay_counter`, `kafkasink`,
+  `streammux`, `streamdemux`, `coalescehistory` and `llm_remote` subclass a
+  GStreamer base directly. `stablediffusion` is hosted but fills in neither
+  `process_frames` nor `process_payload`. Reparenting a family onto one of those
+  two seams in `backend/core.py` is what makes its pipelines runnable.
+  `overlay_counter` inherits `overlay`, which the launcher rewrites to g2g's
+  native `analyticsoverlay`, so the plain overlay line works regardless.
+
+- **A hosted element's properties are only checked once its class loads.** The
+  g2g host takes any name it does not read itself and hands it to the Python
+  class, which is the only thing that knows the real set, so a typo fails at
+  pipeline start rather than at parse. `gst-inspect` on `pyelement` lists the
+  host's own properties and says the rest come from the class.
+
+## Elements
+
+- **`WhisperSpeechTTS.do_generate_speech` returns a `(1, n)` array**, which
+  `soundfile` rejects with `LibsndfileError: Format not recognised`, so the
+  element emits no audio. `CoquiTTS` returns 1-D and is fine. Pre-existing on
+  both backends.
+
+- **`AnomalyEngine._transform` is assigned only in `do_load_model`**, so
+  `_get_transform` raises `AttributeError` on an engine whose model never
+  loaded. Pre-existing on both backends.
+
+- **An engine that fails to load its model keeps running with `model=None`**,
+  so the first frame raises somewhere further on instead of naming what went
+  wrong. The README caption line wants `gptqmodel` for its AWQ model; without it
+  `CaptionQwen` logs the load failure, then dies on `captioning returned None`.
+  Failing at load time would name the missing package.

demo/football/README.md

@@ -0,0 +1,53 @@
+# Football demo
+
+Real-time football broadcast overlay: **detection → tracking → overlay**
+(`pyml_yolo`/`pyml_objectdetector` -> `pyml_tracker` -> `pyml_football_overlay`).
+
+The overlay draws a foot ellipse per player coloured by team (red/blue, voted
+from jersey hue), a gold ellipse for referees, motion trails (off by default),
+and a focal-player HUD with headshot, ball contacts, and distance travelled.
+Players whose team isn't decided yet (and unclassifiable kits, e.g. the
+goalkeeper) are left unmarked rather than drawn in a placeholder colour. The
+ball is tracked for contact counting but its marker is off by default.
+
+## Models
+
+The detector weights (`football.pt`, `football.onnx`, `football_fp16.onnx`,
+`football_int8.onnx`) are hosted on the Hugging Face Hub at
+`collabora/gst-python-ml-football`, not in git. `run.sh` downloads the one its
+`BACKEND` needs into `models/football/` on first use. To fetch by hand:
+
+```bash
+python demo/football/fetch_models.py          # pt + fp16
+python demo/football/fetch_models.py all
+```
+
+## Run
+
+```bash
+# file -> annotated MP4
+demo/football/run.sh
+demo/football/run.sh 08fd33_4.mp4 demo/football/out.mp4 1280x720
+
+# file -> live on-screen
+demo/football/run.sh display
+demo/football/run.sh display 08fd33_4.mp4 1280x720
+
+# live camera -> on-screen
+demo/football/run.sh camera /dev/video0
+```
+
+## Environment knobs
+
+| Var        | Default | Meaning |
+|------------|---------|---------|
+| `BACKEND`  | `pt`    | `pt` = PyTorch `pyml_yolo`; `fp16` = ONNX FP16 via `pyml_objectdetector` (CUDA). |
+| `INTERVAL` | `3`     | Run detection every Nth frame; the tracker/overlay still update every frame, so it stays smooth at ~N× less inference cost. The main real-time lever. |
+
+```bash
+BACKEND=fp16 demo/football/run.sh display     # faster inference path
+INTERVAL=5   demo/football/run.sh display     # detect every 5th frame
+INTERVAL=1   demo/football/run.sh             # detect every frame (max accuracy)
+```
+
+

demo/football/fetch_models.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+# Football demo model download
+# Copyright (C) 2026 Collabora Ltd.
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Library General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Library General Public License for more details.
+#
+# You should have received a copy of the GNU Library General Public
+# License along with this library; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+#
+# Download the football detector weights from the Hugging Face Hub into
+# models/football/ (gitignored). Usage:
+#   python demo/football/fetch_models.py            # pt + fp16, what run.sh uses
+#   python demo/football/fetch_models.py int8 onnx  # named variants
+#   python demo/football/fetch_models.py all
+
+import os
+import sys
+
+from huggingface_hub import hf_hub_download
+
+REPO_ID = "collabora/gst-python-ml-football"
+LOCAL_DIR = os.path.join(
+    os.path.dirname(os.path.abspath(__file__)), "..", "..", "models", "football"
+)
+# run.sh's BACKEND value -> the file it loads
+VARIANTS = {
+    "pt": "football.pt",
+    "fp16": "football_fp16.onnx",
+    "onnx": "football.onnx",
+    "int8": "football_int8.onnx",
+}
+
+
+def main(argv):
+    wanted = argv[1:] or ["pt", "fp16"]
+    if wanted == ["all"]:
+        wanted = list(VARIANTS)
+    for variant in wanted:
+        if variant not in VARIANTS:
+            sys.exit(
+                f"unknown model variant {variant!r}; "
+                f"choose from {', '.join(VARIANTS)} or all"
+            )
+        local = os.path.join(LOCAL_DIR, VARIANTS[variant])
+        if os.path.isfile(local):
+            print(local)
+            continue
+        path = hf_hub_download(
+            repo_id=REPO_ID, filename=VARIANTS[variant], local_dir=LOCAL_DIR
+        )
+        print(path)
+
+
+if __name__ == "__main__":
+    main(sys.argv)

demo/football/onnx_loop.py

@@ -0,0 +1,166 @@
+#!/usr/bin/env python3
+# Run a video through the ONNX (fp16) football pipeline.
+#
+#   detector (onnx) -> pyml_tracker -> pyml_football_overlay
+#
+# Usage:
+#   python demo/football/onnx_loop.py INPUT.mp4             # live display, looping
+#   python demo/football/onnx_loop.py INPUT.mp4 OUTPUT.mp4  # write annotated mp4
+# (self-contained: finds the repo venv + plugins and re-execs into them)
+import os
+import subprocess
+import sys
+import glob
+
+REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+VENV = os.path.join(REPO, ".venv")
+MODEL = os.path.join(REPO, "models/football/football_fp16.onnx")
+os.environ["GST_PLUGIN_PATH"] = (
+    os.path.join(REPO, "plugins") + os.pathsep + os.environ.get("GST_PLUGIN_PATH", "")
+)
+if not os.environ.get("_ONNX_LOOP_REEXEC") and os.path.isdir(VENV):
+    os.environ["VIRTUAL_ENV"] = VENV
+    os.environ["PATH"] = (
+        os.path.join(VENV, "bin") + os.pathsep + os.environ.get("PATH", "")
+    )
+    libs = sorted(
+        set(
+            glob.glob(
+                os.path.join(
+                    VENV, "lib", "python*", "site-packages", "nvidia", "*", "lib"
+                )
+            )
+        )
+    )
+    if libs:
+        os.environ["LD_LIBRARY_PATH"] = os.pathsep.join(
+            [*libs, os.environ.get("LD_LIBRARY_PATH", "")]
+        )
+    os.environ["_ONNX_LOOP_REEXEC"] = "1"
+    pybin = os.path.join(VENV, "bin", "python")
+    exe = pybin if os.path.exists(pybin) else sys.executable
+    os.execv(exe, [exe, *sys.argv])
+
+if not os.path.isfile(MODEL):
+    subprocess.check_call(
+        [
+            sys.executable,
+            os.path.join(REPO, "demo", "football", "fetch_models.py"),
+            "fp16",
+        ]
+    )
+
+import gi  # noqa: E402
+
+gi.require_version("Gst", "1.0")
+from gi.repository import Gst, GLib  # noqa: E402
+
+Gst.init(None)
+
+
+def on_message(bus, message, loop, pipeline, do_loop):
+    t = message.type
+
+    if t == Gst.MessageType.EOS:
+        if do_loop:
+            # Display mode: seek back to the start to loop the clip.
+            print("Looping...")
+            if not pipeline.seek_simple(
+                Gst.Format.TIME, Gst.SeekFlags.FLUSH | Gst.SeekFlags.KEY_UNIT, 0
+            ):
+                print("Failed to seek back to start", file=sys.stderr)
+                loop.quit()
+        else:
+            # mp4 mode: end of file, the muxer has finalized the file.
+            loop.quit()
+
+    elif t == Gst.MessageType.ERROR:
+        err, debug = message.parse_error()
+        print(f"ERROR: {err}", file=sys.stderr)
+        if debug:
+            print(f"DEBUG: {debug}", file=sys.stderr)
+        loop.quit()
+
+
+def main():
+    if len(sys.argv) < 2:
+        print(f"usage: {sys.argv[0]} INPUT.mp4 [OUTPUT.mp4]", file=sys.stderr)
+        print(
+            "  no OUTPUT -> live display (looping); OUTPUT -> write annotated mp4",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+    video = os.path.abspath(sys.argv[1])
+    out = os.path.abspath(sys.argv[2]) if len(sys.argv) > 2 else None
+
+    # Shared detection + overlay chain. Feed the ORIGINAL resolution:
+    # pyml_objectdetector letterboxes to the model's 640 internally for
+    # inference and maps boxes back, so the overlay stays full-res.
+    chain = (
+        f"filesrc location={video} ! "
+        "decodebin ! videoconvert ! video/x-raw,format=RGB ! "
+        "queue max-size-buffers=8 max-size-time=0 max-size-bytes=0 ! "
+        "pyml_objectdetector engine-name=onnx "
+        f"  model-name={MODEL} device=cuda:0 "
+        "  input-format=nchw post-process=anchor_free interval=1 "
+        "  confidence=0.1 nms-iou=0.7 ! "
+        "queue max-size-buffers=8 max-size-time=0 max-size-bytes=0 ! "
+        "pyml_tracker tracker-type=bytetrack new-track-confidence=0.25 ! "
+        "videoconvert ! video/x-raw,format=RGBA ! "
+        "queue max-size-buffers=8 max-size-time=0 max-size-bytes=0 ! "
+        "pyml_football_overlay class-names=ball,goalkeeper,player,referee "
+        "  team-colors=true trails=false show-ids=false show-labels=false "
+        "  draw-from-detections=true min-confidence=0 merge-iou=0.5 "
+        "  position-smoothing=0.7 highlight-focal=false ! "
+    )
+    if out:
+        pipeline_description = (
+            chain + "queue max-size-buffers=8 max-size-time=0 max-size-bytes=0 ! "
+            "videoconvert ! openh264enc ! h264parse ! mp4mux ! "
+            f"filesink location={out}"
+        )
+        do_loop = False
+    else:
+        # Pre-roll buffer absorbs inference jitter for smooth real-time display.
+        pipeline_description = (
+            chain + "queue max-size-buffers=600 max-size-time=0 max-size-bytes=0 "
+            "  min-threshold-buffers=30 ! "
+            "videoconvert ! autovideosink sync=true"
+        )
+        do_loop = True
+
+    print(pipeline_description)
+    print(f"writing -> {out}" if out else "live display (looping)")
+
+    try:
+        pipeline = Gst.parse_launch(pipeline_description)
+    except GLib.Error as e:
+        print(f"Failed to create pipeline: {e}", file=sys.stderr)
+        sys.exit(1)
+
+    loop = GLib.MainLoop()
+
+    bus = pipeline.get_bus()
+    bus.add_signal_watch()
+    bus.connect("message", on_message, loop, pipeline, do_loop)
+
+    pipeline.set_state(Gst.State.PLAYING)
+
+    try:
+        loop.run()
+    except KeyboardInterrupt:
+        if out:
+            # Finalize the mp4 on Ctrl-C: send EOS and wait for the muxer to
+            # flush its trailer, otherwise the file is left unplayable.
+            pipeline.send_event(Gst.Event.new_eos())
+            bus.timed_pop_filtered(
+                5 * Gst.SECOND, Gst.MessageType.EOS | Gst.MessageType.ERROR
+            )
+    finally:
+        pipeline.set_state(Gst.State.NULL)
+    if out:
+        print(f"Done: {out}")
+
+
+if __name__ == "__main__":
+    main()