Conversation
…window Clarify how the effective target FPS is resolved for each camera: TARGET_FPS overrides every stream, then the camera's targetFps, then its fps, and finally the 14.95 default. Add the MEASUREMENT_WINDOW_SECONDS and INIT_DURATION settle-time controls, including their defaults and the per-window pass/fail step logic, and note that the resolved values are recorded in stream_density.log. Signed-off-by: sumanaga <sumana.gargi.m@intel.com>
…window Update the loss-prevention benchmarking docs to match the new stream-density behavior. Explain that the effective target FPS resolves in order: the TARGET_FPS env override for every stream, the camera's targetFps, its fps, then the 14.95 default, and note that the resolved value and source are logged to stream_density.log. Add the MEASUREMENT_WINDOW_SECONDS and INIT_DURATION settle-time controls, including their defaults and the per-window pass/fail step logic, to both performance.md and lp-benchmarking.md. Also tidy table formatting and whitespace in lp-benchmarking.md. Signed-off-by: sumanaga <sumana.gargi.m@intel.com>
jcork-intel
left a comment
There was a problem hiding this comment.
Thanks Sumana, and you're right about benchmark.md: it isn't published, so my Sep 25th note was wrong. It was commented out of the nav in "Update mkdocs.yml" (#116) on Jul 13th, when the LP Benchmark page was added. Was that meant to replace it? If so, the benchmark.py reference has no home on the site. Should it come back, or move into the LP page?
A terms table. We've agreed one vocabulary with the benchmarking team, and it isn't written down anywhere public yet. Could you add a short "Terms" section to the benchmarking page (benchmark.md if it comes back, otherwise the LP Benchmark page)? The LP README and the performance-tools report can then link to it rather than each defining the words. If performance-tools is moving to its own docs page, that's the better home. Suggested text:
| Term | Meaning |
|---|---|
| Sweep | The whole search for the maximum lane count, made of many benchmark runs |
| Benchmark run | One complete attempt at one lane count: bring up, warm up, measure, tear down |
| Warmup period | The first 10 seconds of a run, dropped while streams settle |
| Measurement interval | The part of a run that counts, at least 100 seconds |
| Sampling interval | How long one sample covers: one second |
| FPS sample | One second of frames for one stream |
| Throughput threshold | 95% of a camera's own frame rate; 14.25 fps for a 15 fps camera |
| Per-frame latency and its threshold | From a frame arriving to its results leaving the pipeline, ready for the solution to use. Threshold: p95 at or under 500 ms (proposed) |
| Run acceptance criterion | Two consecutive runs agree before a count is declared or a lane given up |
| Use case density | Lanes running the full workload at once |
| Stream density | Camera streams decoded at that load: lanes times streams per lane |
For docs_src/use-cases/loss-prevention/performance.md:
- Line 138, "Default behavior": still says a stream is judged against targetFps or 14.95. Please match the precedence you wrote at lines 152 to 155 (TARGET_FPS if passed, then targetFps, then fps, then the default), and say the threshold is 95% of that target.
- Line 185, TARGET_FPS row: it's now an override that's unset by default, not a "minimum FPS threshold" defaulting to 14.95.
- Line 187, INIT_DURATION row: please add the 10 s default, as in the MEASUREMENT_WINDOW_SECONDS row above it.
- Lines 146 and 155: the last-resort default is 14.95 today. I've asked on #254 for it to become 15, so please update these once that lands.
- Terminology, lines 169 to 173: please use the agreed terms. "Measurement Window & Settle Time" becomes "Measurement interval and warmup period", "settle time" becomes "warmup period", "window" becomes "measurement interval", and "two passing windows in a row" becomes "run acceptance criterion (two consecutive passes)". Variable names stay as they are.
For docs/user-guide/loss-prevention/lp-benchmarking.md: please keep only the content fixes (the camera_to_workload_full.json references) and revert the formatting changes. Rishika cleaned that file on Jul 29th; this version brings back four emoji, drops about 25 blank lines, and repeats the make clean-all line (lines 216 and 218).
Once #254 lands, a short description of the new per-stream report (average, p10, p90, seconds below the throughput threshold) would help readers. Fine as a follow-up.
One idea that's worked well for us: before opening a PR, run a "ship it" checklist or agent skill that compares the code changes against the docs and lists the pages that need updating. It may have caught the TARGET_FPS and INIT_DURATION rows here.
PR Checklist
What requirement is this design document for?
#250 , #247
Anything the reviewer should know when reviewing this PR?
docs_src/performance-tools/benchmark.md is not built and published
If there are any other design Pull Requests or requirements, please link them here (i.e. intel-retail/automated-self-checkout )
intel-retail/performance-tools#254
intel-retail/loss-prevention#359