Find out how many GPU-hours you're paying for and not using.
Read-only. Runs offline. Never touches your cluster.
$ gpuwaste synth --out ./export # synthetic data with known faults
$ gpuwaste analyze --data ./export
CLUSTER prod-us-east WINDOW 2026-08-01 → 2026-08-07
GPUs 12 × NVIDIA A100-SXM4-80GB
ALLOCATED 2,016 GPU-hours
ACTUALLY USED 461 GPU-hours 22.9%
──────────────────────────────────────────────
UNUSED 1,555 GPU-hours
ESTIMATED WASTE $ 21,754 / month
(cloud rates, default $2.40/GPU-hour)
TOP FINDINGS
1 ml-training / trainer-c04e NEVER INITIALIZED $9,400/mo
GPU allocated, CUDA context never created
2 research / notebook-51ab FRACTIONAL USER $3,409/mo
2 GPUs requested, 1 showing activity
also: NO SCALE-TO-ZERO — idle 50% of samples, allocation constant
3 ml-training / featurizer-3b81 MEMORY-PARKED $2,322/mo
83% of framebuffer resident, SM active 1.2% — model loaded, no traffic
4 data-eng / embedder-88f1 CRASHLOOPING ALLOCATION $2,232/mo
41 restarts while holding the allocation
5 research / batch-scorer-0e7d NO SCALE-TO-ZERO $1,558/mo
idle 46% of samples, allocation constant
6 ml-serving / vision-api-9c2a GHOST WORK $288/mo
engine 94% active, tensor cores 0.3% — busy, but not doing ML
Dropped 1 column(s) not in schema: internal_cost_center
The run above is from
gpuwaste synth, which generates data with deliberately injected faults. Try it yourself in 30 seconds — no cluster required.
Because DCGM_FI_DEV_GPU_UTIL doesn't measure what you think it does.
It reports that a kernel is resident on the device — not that the device is doing useful work. A process that pins the GPU with a trivial kernel reads 100% utilized while computing nothing. Every dashboard built on that metric will tell you your cluster is busy.
gpuwaste reads DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE instead, and reports the gap between them. High engine activity with idle tensor cores is a GPU that's occupied but not serving models — the failure mode that costs the most and shows up in no dashboard.
Most tools in this space want a live endpoint. In real production that means network policies, an mTLS service mesh, custom RBAC, and a security review — so most people never get past ingress.
gpuwaste ingests CSV exports you already know how to pull. It works in air-gapped environments, on a laptop, from a jumpbox, or from a Grafana export someone emailed you. The value is in the analysis, not the extraction.
git clone https://github.com/Dgotlieb/gpuwaste
cd gpuwaste && pip install -e .
# See it work with no cluster at all — no GPU required
gpuwaste synth --out ./export
gpuwaste analyze --data ./export
# Then with your own data
gpuwaste init-pricing > pricing.yaml # edit to your rates
gpuwaste analyze --data ./my-export --pricing pricing.yaml --step 60
# Before sharing an export with anyone
gpuwaste anonymize --data ./my-export --out ./share
gpuwaste inspect --data ./shareNo Prometheus access from your machine? The PromQL is in docs/queries.md — hand it to whoever has it, or export from Grafana. gpuwaste analyze only needs the CSVs.
On EKS, GKE or a GPU cloud, use your effective rate — on-demand, committed-use or spot, whatever you actually pay:
by_model:
"NVIDIA A100-SXM4-80GB": 3.20
"NVIDIA H100 80GB HBM3": 5.10Running your own hardware? Derive the rate from capex — the case cloud-native cost tools don't handle:
amortization:
capex_usd: 240000
useful_life_months: 36
gpu_count: 8
overhead_multiplier: 1.35 # power, cooling, space, support| Finding | What it means |
|---|---|
| Ghost work | Engine active, tensor cores idle — occupied, not serving |
| Memory-parked | Weights resident, no compute — loaded and forgotten |
| Never initialized | GPU allocated, CUDA context never created |
| Crashlooping allocation | Restarts climbing, GPU stays reserved |
| Fractional user | N GPUs requested, one in use |
| Oversized MIG | Partition far larger than observed peak |
| No scale-to-zero | Strong diurnal idle, flat allocation |
Anonymization is on by default; --no-anonymize is the flag you have to type.
- Namespace, pod, container, node and GPU UUID are replaced with salted HMAC tokens
- The salt is generated locally, stored at
~/.gpuwaste/salt, and never included in any export - Only columns defined in the schema are emitted — everything else is dropped unread, so a custom label can't leak through a filter it wasn't written for
gpuwaste inspect <file>shows exactly what a file contains before you send it anywhere
Tokens stay consistent within one export so the report reads normally, and mean nothing across exports so nothing cross-references.
What this is not: a guarantee of zero information disclosure. The numbers themselves still carry fleet size, GPU mix, and rough scale. If you're sharing an export outside your organization, check with your team first.
You don't need NVIDIA hardware to work on this. dev/ runs a fake
dcgm-exporter behind a real Prometheus, using the label spellings a live
cluster produces:
cd dev && docker compose up -d && sleep 300
gpuwaste extract --prometheus http://localhost:9090 --days 1 --step 15 --out ./local
gpuwaste analyze --data ./local --step 15Time is compressed — 10 real seconds is one simulated hour, so a full day/night
cycle passes every four minutes and idle-overnight patterns appear in a short
run. See dev/README.md.
v0.1 — offline analysis only. Live cluster connection (as an in-cluster Job) is planned for v0.2.
Bug reports and label mappings from cluster setups it doesn't handle yet are the most useful contributions right now.
Built by Daniel Gotlieb — Director of AI Engineering & Infra running production inference on Kubernetes, on-prem and cloud. The failure taxonomy comes from finding these in real clusters, including several that cost more than anyone expected.
MIT licensed.