Independent AI Researcher | Energy Efficiency & Sustainable Computing
I measure how much energy LLM inference actually costs — GPU package power, NVML, CV < 2% — and publish every number with its basis. No FLOPs math, no vendor specs.
NF4 crossover ranges from ~1.8B (Blackwell, current stack) to ~3.7B (Ada). Below the crossover, quantization costs more energy; above it, quantization saves. The crossover belongs to a card+stack combination, not to an architecture. — Measured on RTX 4090D, RTX 5090, A800 with NVML power sampling.
This finding challenges the default assumption that "quantize everything = green." All benchmark data is open and reproducible.
Key Findings:
- NF4 crossover: ~2.1B (Turing T4) → ~3.7B (Ada 4090) → ~4.8B (Blackwell 5090 v1.1.0); ~1.8B on 5090 with a newer stack
- INT8 (
load_in_8bit=True): default increases energy +17–147% vs FP16; setllm_int8_threshold=0.0to recover +79% throughput and −36% energy - Below crossover: NF4 adds 11–29% energy (Blackwell, small models)
- Above crossover: NF4 saves 8–35% energy (Ada, models ≥6B)
| # | What | Where |
|---|---|---|
| 1 | Run the measurement container — reproduce every number on your own GPU (MLCube, NVML, MLCommons-style energy reporting) | ecocompute-mlcube → |
| 2 | Use the web estimator — compare precision choices for your GPU × model size, every answer states its basis and confidence | quantenergy.tech → |
| 3 | Ask your AI assistant — EcoCompute is listed in the official MCP Registry (io.github.hongping-zh/ecocompute-estimator, 4 tools; Claude Desktop / Cursor setup in the announcement post) |
MCP endpoint → |
Also: an interactive Accuracy × Cost × Carbon dashboard →, and a GitHub Action → that lints CI for LLM energy waste.
| Achievement | Details |
|---|---|
| 🤗 HuggingFace Optimum | Quantization energy findings cited in the official documentation (optimum PR #2410) |
| 🏛️ MLCommons | Invited discussion on quantization energy metrics for MLPerf Inference (inference #2558) |
| 🔥 torchao maintainers | Independently confirmed the RTX 5090 FP8 energy anomaly we reported (ao #4094) |
| ⬆️ Upstream contributions | bitsandbytes PR #1882 · transformers PR #44407 |
| 📊 Open dataset | 360+ configurations across 4 GPU architectures (Turing, Ampere, Ada, Blackwell), CV < 2% |
| 🌍 Zenodo (CC BY 4.0) | 10.5281/zenodo.19647290 (main, v1.1.0) · 10.5281/zenodo.22037483 (RTX 4090 deep dive) |
- @gkgoing — first independent replication (2026-08): TinyLlama-1.1B in NF4 on an RTX 3050 Ti Laptop. The perplexity matched our RTX 4090 measurement to four decimals; the energy number did not — and that contrast is the reason this project exists. Submission →
- @amarjithkumar2007-wq — AMD GPU (ROCm) detection and fallback (PR #20); context_length applied in measurement path (PR #22).
- @yesheng-oss — backend contribution guide (PR #21): documents how to add llama.cpp, vLLM, or TensorRT-LLM as inference engines.
- ✅ HuggingFace integration published · ✅ MLCommons engagement · ✅ MCP Registry listing
- 🔄 arXiv: "When Does Quantization Save Energy?" — seeking endorsement
- 🛡️ VS Code extension: real-time energy linting before code merges
- 🤝 Enterprise pilots: carbon-aware CI/CD design partners
| Action | Link |
|---|---|
| ⭐ Star / run the container | ecocompute-mlcube — an independent replication is the most valuable contribution |
| 📊 Submit a measurement | Contribution guide → — your GPU + model + precision, CC BY 4.0 into the dataset |
| 📧 arXiv endorsement | Email me → |
| 🤝 Design partner / pilot | Email me → |
- Website & estimator: quantenergy.tech · MCP registry listing
- Dataset: GitHub (container + replication) | HuggingFace card | Zenodo DOI
- Paper: "When Does Quantization Save Energy? Empirical Analysis of the Energy-Efficiency Crossover Effect Across GPU Generations" (arXiv in progress)
🌍 Making AI development more sustainable, one measured number at a time.