Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
__pycache__/
.venv/
.pytest_cache/
exports/*
!exports/README.txt
results/*
!results/README.txt
dist/
VERSION
requirements.txt
45 changes: 23 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# 📊 Evolve Charts

**Turn Evolve deep analysis exports into publication-ready charts**
**Turn Evolve benchmark runs into publication-ready charts**

[![Banner](./docs/images/banner.png)](https://traverseresearch.nl)

Expand All @@ -29,29 +29,28 @@ your web browser, and **everything stays on your computer**. Nothing is uploaded
the libraries the tool uses (about 150 MB). This needs an internet connection, takes a few minutes and
only happens once. Keep the window that opens running while you use the tool; close it when you're
done.
3. **Export your results from Evolve.** After a benchmark run, click **Export** on the score screen. Do
this on every GPU or setting you want to compare, using the same benchmark on each. Scores and
per-frame data are always included; deep analysis (per-pass timings) needs an Evolve Professional
licence.
4. **Add your results.** Your browser opens Evolve Charts. Drag the export files onto *Add Evolve
exports*, or put them in the `exports` folder so they load automatically every time.
3. **Run your benchmarks in Evolve.** Evolve saves every run as a `.evolve` file, in
`Documents\Evolve\results` on Windows. Run on every GPU or setting you want to compare, using the
same benchmark on each.
4. **Add your results.** Your browser opens Evolve Charts. Drag the `.evolve` files onto *Add Evolve
runs*, or put them in the `results` folder so they load automatically every time.
5. **Name your results** on their cards (for example "RTX 4070 Ti" or "Gen 1"). These names appear in
every chart.

To update, download the zip again and replace the old folder (keep your `exports` folder if you use it).
To update, download the zip again and replace the old folder (keep your `results` folder if you use it).

## 🗺️ What's in the tool

The tool reads every Evolve export: the **Export** zip from the score screen, and the files Evolve's
command line writes (`evolve_results.csv`, `scores.csv`, the per-frame CSV and `deep_analysis.json`).
It has a page per question, in the bar at the top:
The tool reads the `.evolve` file Evolve saves for every run: the measurements of every frame and pass,
the GPU sensor readings and the system information. It calculates the scores from those the way Evolve
does. It has a page per question, in the bar at the top:

| Page | What it shows | Needs |
|---|---|---|
| **Results** | Load and name exports; GPU, driver, OS, Evolve version and what each file contains | |
| **Results** | Load and name runs; GPU, driver, OS, Evolve version, benchmark and what each file contains | |
| **Scores** | Evolve's scores side by side, with the workload category each one scores, and the differences | scores |
| **GPU time** | Where the GPU spends its time each frame, per workload category; then one workload category over the run and the passes it's made of | per-frame data; the passes need deep analysis |
| **Passes** | Every render pass in one searchable table; tick passes to plot them over the run | deep analysis |
| **GPU time** | Where the GPU spends its time each frame, per workload category; then one workload category over the run and the passes it's made of | per-frame data; the passes need pass timings |
| **Passes** | Every render pass in one searchable table; tick passes to plot them over the run | pass timings |
| **GPU sensors** | Clocks, power, temperature and more during the run, to spot throttling | per-frame data |
| **Export** | CSV files for Excel, Google Sheets, Datawrapper and Flourish | |

Expand All @@ -78,13 +77,13 @@ drivers and Evolve version. **Help** explains how the numbers are made.
because a GPU can idle between passes or run passes at the same time.
- **Whiskers** on bar charts show the range covering 90% of frames (5th to 95th percentile).
- Only compare results from the **same Evolve version and benchmark settings**. The tool warns you
when the Evolve versions or graphics APIs differ.
when the benchmarks, Evolve versions or graphics APIs differ.

## ❓ Troubleshooting

- **A result is missing from *Passes***: its export has no deep analysis, which
needs an Evolve Professional licence. Its card shows ✗ Deep analysis; scores and workload
categories still work.
- **A result is missing from *Passes***: its run has no pass timings. Its card shows ✗ Passes; scores
and workload categories still work.
- **"was saved by a newer Evolve"**: download the latest Evolve Charts.
- **The browser doesn't open**: open <http://localhost:8501> yourself while the black window is open.
- **Anything else**: send a screenshot of the black window to Traverse Research.

Expand All @@ -93,16 +92,18 @@ drivers and Evolve version. **Help** explains how the numbers are made.
- Run without the launcher: `uv run evolve-charts.py`. Dependencies are declared inline
([PEP 723](https://peps.python.org/pep-0723/)), so no virtual environment is needed.
- Code lives in `evolve_charts/`:
- `loader.py` reads export zips into tidy tables.
- `loader.py` reads `.evolve` files (a zip of Parquet tables and a JSON document) into tidy tables.
- `scores.py` calculates Evolve's scores from those tables.
- `categories.py` describes Evolve's workload categories and which pass belongs to which.
- `stats.py` computes the summaries.
- `charts.py` builds the Plotly figures.
- `ui.py` is the Streamlit page.
- The look follows Evolve's `crates/evolve/src/ui/styling.rs`: theme colours and the Spartan font are set in
`.streamlit/config.toml`, and the fonts, logo and icon live in `static/`. Chart colours are in `charts.py`.
- Which pass belongs to which workload category comes from Evolve's Rust source. After pass names
change in Evolve, regenerate the list with `python tools/sync_categories.py` (it reads
`../evolve/crates/evolve-frame-bucketing/src/lib.rs` by default).
- Which pass belongs to which workload category, and the score multipliers, come from Evolve's Rust
source. After pass names or score calibrations change in Evolve, regenerate them with
`python tools/sync_categories.py` and `python tools/sync_scores.py` (both read `../evolve` by default),
then `uvx ruff format evolve_charts`.
- Dependencies are locked in `evolve-charts.py.lock`; refresh it with `uv lock --script evolve-charts.py --upgrade`.
- Tests: `uv export --script evolve-charts.py --no-hashes > requirements.txt`, then
`uv venv && uv pip install -r requirements.txt pytest && uv run --no-project pytest`. CI runs the same.
Expand Down
28 changes: 12 additions & 16 deletions docs/Android.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,17 @@
## Automating `evolve` benchmark run on Android
## Automating `evolve` benchmark runs on Android

1. Make sure the app is installed.
2. Use relative paths in `--export*` parameters, so that `evolve` has access to write them relative to `/sdcard/Android/data/nl.traverse_research.evolve/files/`. Subfolders are supported, these will automatically created.
- It is recommended to use subfolders with the current date and time, as `evolve` will refuse to start and overwrite any export files that aleady exist.
3. Use the `python ./scripts/run_on_android.py` script in this folder to start Evolve, like a desktop run. Common parameters to export all data from a `run-custom` are:
2. Use the `python ./scripts/run_on_android.py` script in this folder to start Evolve with command-line
arguments, like a desktop run:
```sh
ts=$(date '+%F-%T')
python ./scripts/run_on_android.py run-custom --export-scores $ts/scores.csv --export-per-frame $ts/frames.csv --export-deep-analysis $ts/deep-analysis.json
python ./scripts/run_on_android.py run-custom --benchmark ray-tracing-inline
```
4. When the benchmark finishes, run the following commands to pull the resulting files to your desktop:
```sh
adb pull /sdcard/Android/data/nl.traverse_research.evolve/files/$ts/deep-analysis.json
adb pull /sdcard/Android/data/nl.traverse_research.evolve/files/$ts/scores.csv
adb pull /sdcard/Android/data/nl.traverse_research.evolve/files/$ts/frames.csv
```
Alternatively, pull the entire timestamped folder:
```sh
adb pull /sdcard/Android/data/nl.traverse_research.evolve/files/$ts
3. Evolve saves every run as a `.evolve` file in
`/sdcard/Android/data/nl.traverse_research.evolve/files/Documents/results/`. When the benchmark finishes,
pull them to your desktop:
```sh
adb pull /sdcard/Android/data/nl.traverse_research.evolve/files/Documents/results evolve-results
```
These paths can also be found directly on the phone by navigating its internal storage using MTP (when `USB for file transfer` is enabled via the notification on the phone).
The folder can also be found on the phone by browsing its internal storage over MTP (when
`USB for file transfer` is enabled via the notification on the phone).
4. Load the `.evolve` files into Evolve Charts.
3 changes: 2 additions & 1 deletion evolve-charts.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
# "streamlit>=1.65",
# "plotly>=6",
# "pandas>=2.2",
# "pyarrow>=18",
# ]
# ///
"""Evolve Charts: turn Evolve deep analysis exports into charts.
"""Evolve Charts: turn Evolve benchmark runs (.evolve files) into charts.

Start it by double-clicking `Start Evolve Charts.bat` (Windows) or `start-evolve-charts.sh`
(macOS/Linux), or run `uv run evolve-charts.py`. A page opens in your browser; nothing leaves
Expand Down
5 changes: 3 additions & 2 deletions evolve-charts.py.lock
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,18 @@ resolution-markers = [
"python_full_version >= '3.14' and sys_platform == 'emscripten'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'",
"python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'win32'",
"python_full_version < '3.12' and sys_platform == 'win32'",
"python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'emscripten'",
"python_full_version < '3.12' and sys_platform == 'emscripten'",
"python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'",
"python_full_version < '3.12' and sys_platform == 'win32'",
"python_full_version < '3.12' and sys_platform == 'emscripten'",
"python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'",
]

[manifest]
requirements = [
{ name = "pandas", specifier = ">=2.2" },
{ name = "plotly", specifier = ">=6" },
{ name = "pyarrow", specifier = ">=18" },
{ name = "streamlit", specifier = ">=1.65" },
]

Expand Down
5 changes: 5 additions & 0 deletions evolve_charts/categories.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,11 @@
WORKLOAD_ORDER = list(WORKLOAD_DESCRIPTIONS)


def workload_variants(scope: str) -> list[str]:
"""The `WorkloadCategory` variants the pass counts towards, as named in Evolve's source."""
return WORKLOADS.get(scope, [])


def workloads(scope: str) -> list[str]:
"""The workload categories the pass is part of, in Evolve's order; empty if Evolve leaves it uncategorized.

Expand Down
Loading
Loading