Skip to content

study.md

Latest commit

 

History

History
318 lines (282 loc) · 21.3 KB

study.md

File metadata and controls

318 lines (282 loc) · 21.3 KB
Raw

The study

Is run time a good stand-in for energy, and how do C, Python and Node.js compare?

Developers rarely measure energy. They measure time and assume that faster code uses less energy. That is plausible for one program on one processor, but it is an assumption. This study tests it on a small, fully controlled workload: three sorting algorithms, implemented identically in three languages, sorting identical inputs.

Status. The harness, the workers, the statistics and the report are built and tested. The results, and whether they include energy yet, are in results.md, which the report script generates from a recorded run. Energy can only be measured on Linux running directly on hardware (how); until the study has run there, results.md comes from a run in Docker with energy off (ADR 20). It gives time and exact operation counts and marks every energy, power and carbon figure as pending.

Questions

  1. Time as a proxy. Does energy follow time? If every configuration drew the same average power while sorting, energy per sort would be proportional to time per sort, and time would be a perfect stand-in for energy.
  2. Languages. How much energy does one sort take in C, Python and Node.js, for the same algorithm and input? Is the energy ratio between two languages the same as their time ratio?
  3. Input order. Bubble sort does the same number of comparisons on sorted and random input, and this quicksort does far more on sorted input than on random input. Does energy follow those differences in the same way that time does?

What would count as an answer

Time is a good stand-in for energy to the extent that the power drawn while sorting (energy per sort divided by time per sort) is the same in every configuration. The report tests this in three ways, from the most direct to the weakest:

  • Power ratios. For each pair of languages on the same algorithm and input, and for each input order against random input in the same language, the report gives the time ratio, the energy ratio and their quotient, the power ratio, each with a 95% confidence interval. Each power ratio gets one of four readings against a margin of 10% either way (from 1/1.1, about 0.909, to 1.1), which is set here, before any energy has been measured:
    • same power within 10%: the whole interval lies inside the margin, so the time ratio predicts the energy ratio to within 10%;
    • differs: the interval excludes 1, so one side draws more power, and time alone misjudges the energy difference by that factor;
    • inconclusive: the interval includes 1 but reaches beyond the margin. Failing to find a difference is not evidence that there is none, so this is never read as the same power;
    • undefined: energy or time per sort is lost in the noise.
  • Net power above idle for every configuration, and a chart of energy per sort against time per sort, in which constant power is a straight line.
  • Rank agreement. Spearman's rank correlation between median time per sort and median energy per sort across configurations. This is weaker: configurations differ in cost by large factors, so rho stays near 1 unless power differs enough to reorder them.

The margin is a judgement: a difference in power of less than 10% would rarely change a decision made from time alone. The study profile gives 144 power ratios between languages and 108 between input orders. If most of them read "same power within 10%", and the ones that differ are no more than chance would produce (see Analysis), the answer to the first question is "yes". Power ratios that differ by more than the margin, consistently across sizes and orders, answer the second: for those languages or inputs, comparing time is not comparing energy. Many inconclusive readings would mean that the measurements were too noisy to decide, and the report would say so.

Method

Workload

  • Algorithms. Bubble sort, insertion sort and quicksort (Lomuto partition, last-element pivot, recursing into the smaller side), carried over unchanged from Algorithm-Energy-C.
  • Inputs. Permutations of 0..n-1 in four orders: random (Fisher-Yates), sorted, reversed, and nearly sorted (5% of positions swapped with random partners). Sizes 250, 500, 1,000 and 2,000 in the study profile.
  • Generator. xorshift32 (shifts 13, 17, 5) from seed 2463534242, reimplemented in each language. Every run of every language sorts exactly the same array.
  • Languages. C (compiled with -O2), Python and Node.js, each as a worker program that generates the input, sorts a fresh copy of it a set number of times, checks the result and prints one line with the counts, its own loop time and a hash of its input.

The three implementations are line-for-line ports. They count comparisons and array writes in the same places, and each uses its language's everyday array type (C int arrays, Python lists, JavaScript arrays). None uses a built-in sort.

Checks that the comparison is fair

  • The parity tests compare the first 10,000 xorshift32 outputs for four seeds, whole inputs of every order at eight sizes up to 100,000, and the comparison and write counts of every algorithm on every order.
  • During a run, the harness compares every worker's input hash and counts with those of the other languages for the same input, and stops if they differ. The report checks again.
  • The C sorts still produce the exact counts that Algorithm-Energy-C published (for example 55,119 comparisons and 56,984 writes for quicksort on 4,000 random integers), which the unit tests check in all three languages.

Measurement

  • One process per measured run. The harness (build/greenbench, in C) reads the RAPL counters and CLOCK_MONOTONIC_RAW immediately before starting a worker and immediately after it exits. C is measured the same way as Python and Node.js.
  • Energy. Package and DRAM zones from /sys/class/powercap/intel-rapl:*, summed over sockets, with one counter wrap handled using max_energy_range_uj. If the counters are missing, unreadable or frozen (present but never changing), the harness records "energy unavailable", never a zero. A run stops if a package counter stands still for 100 ms or more, or if the counters rise by more than 1 kW per package over an interval, as they can when a suspend resets them.
  • Enough work per process. RAPL counters update about once a millisecond (Hähnel et al., 2012). During warm-up the harness sets the number of sorts per process so that each worker's sort loop lasts at least 200 ms: it extrapolates from the last warm-up run and aims 20% higher, at 240 ms, so that ordinary run-to-run variation stays above 200 ms. A measured loop can still fall short on a busy machine; the report counts the loops that did.
  • Clocks. Every reported time comes from the harness's CLOCK_MONOTONIC_RAW, read around the whole process. Each worker also times its own sort loop (C with CLOCK_MONOTONIC_RAW, Python with time.perf_counter_ns and Node.js with process.hrtime, both CLOCK_MONOTONIC on Linux); the harness uses these loop times only to calibrate the number of sorts, and the report only to count loops that fell short of the target.
  • Idle baseline. After every run the harness sleeps for the same duration and measures the idle energy, which is subtracted, scaled to the run's exact length.
  • Startup cost. For each language, order and size, a startup trial runs a worker that generates the input and exits without sorting. A single start can be as short as the RAPL update interval, so each measured startup run launches the worker back to back until the batch lasts about as long as a sorting run, and is divided by the number of launches. The median cost per process is subtracted before dividing by the number of sorts, giving time and energy per sort.
  • Repetitions and order. Three warm-up runs per configuration, then 30 measured runs. All measured runs of all configurations are shuffled into one random order from a recorded seed.
  • Pinning. The study script pins the harness and all workers to two CPUs on different physical cores of one processor (never CPU 0 or a CPU that shares its core, and only performance cores on a processor that also has efficiency cores), so the measured processes stay on known cores of one kind. Two rather than one, because Node.js runs its compiler and garbage collector on helper threads (ADR 15).
  • Records. runs.csv holds one row per measured run (tidy data: one variable per column). meta.json holds the settings, the RAPL zones and their wrap values, the CPU model, the kernel, the scaling governor, the turbo setting, the language versions and the exact command. environment.txt adds lscpu output and the git commit.

Analysis

  • Central value. The median of the 30 runs of each configuration. Per-sort time and energy are the whole-process values minus the median startup cost per process, divided by the number of sorts. Energy is net of the idle baseline.
  • Uncertainty. 95% percentile-bootstrap confidence intervals from 10,000 resamples, seeded so that the same data always gives the same report. Each configuration's runs are resampled once, keeping each run's time and energy together, and every estimate for that configuration comes from the same resamples. The startup medians that are subtracted are treated as fixed.
  • Net power above idle. Median net energy per sort divided by median time per sort, in watts: the power the sort draws on top of an idle machine, not the processor's total power. The proxy question is decided on net power, because the idle power is drawn whether the sort runs or not (see threats to validity).
  • Ratios. Between languages (Python against C, Node.js against C, Python against Node.js) for the same algorithm and input, and between each input order and random input for the same language and algorithm: the time ratio, the energy ratio and the power ratio (energy ratio divided by time ratio). The two configurations are resampled independently. Each power ratio is read against the 10% margin as described above; --equivalence-margin changes it, which a published result should state and justify. A power ratio also counts as undefined when a time or energy per sort of either configuration falls to zero or below in more than 2.5% of the resamples, because the interval from the remaining resamples would look narrower than it is.
  • Many comparisons. The intervals are separate 95% intervals, not adjusted for being many. The report counts how many exclude 1 and sets that against the number that chance alone would give if every configuration drew the same power: 5% of them. It does not apply a Bonferroni or Holm correction, which would widen every interval and push more readings to "inconclusive"; the question is about the overall pattern, not about any single pair.
  • Rank agreement. Spearman's rho between median time per sort and median energy per sort across configurations: at each input size, for each language, for each language and algorithm across input orders and sizes, and over all of them, with an interval from 2,000 of the resamples.
  • Run-to-run coupling. Within each configuration, Spearman's rho between the process time and process energy of its 30 runs; the report gives the median per language and algorithm. With steady power, a run that takes longer uses more energy, so a positive value is expected almost by construction. It is a check on measurement noise, not an answer to the question.
  • Carbon. Energy converted to grams of CO2-equivalent with annual grid intensities for the UAE and the UK (Ember, with major processing by Our World in Data, CC BY 4.0; see data/README.md). Per-sort figures are given twice: net of idle, which is what the sort adds, and including the idle power drawn while it runs. Other regions or a custom intensity can be passed to the report.
  • Checks on the data. The report refuses a run that did not finish unless told otherwise (--allow-partial), refuses energy readings that are all zero, and warns when a configuration has fewer than 30 runs.

Threats to validity

Measurement

  • RAPL is the processor's own estimate. Studies of recent Intel processors found package readings that track external power meters closely (Khan et al., 2018), but accuracy varies between processor generations and vendors. This study does not calibrate against an external meter.
  • Package, not system. The counters cover the processor package and, where present, DRAM. They miss the disk, screen, fans and power-supply losses, so the energy at the wall is higher.
  • Everything on the package counts. RAPL cannot separate processes. Background activity during a run is counted as the worker's. The idle baseline removes steady background power but not bursts; randomising the run order spreads bursts across configurations rather than removing them.
  • Idle after work is not idle before work. The idle window follows each run, when the processor is still warm and may be leaving a turbo state, so idle power can be slightly higher than true idle, and net energy slightly low.
  • Counter security filtering. After the PLATYPUS power side-channel attack (Lipp et al., 2021), some Intel systems add noise to RAPL readings. Measured intervals of about 200 ms and 30 repetitions reduce, but do not remove, its effect.
  • Frequency scaling and heat. Turbo and the scaling governor are left as the system sets them and recorded. Long runs heat the processor, which raises its power at the same work; the random order keeps this from favouring one configuration.

Statistics

  • Net rather than total power. The power ratios use energy net of idle: what the sort adds. Idle power is drawn whether or not the sort runs, so adding it to both sides of a ratio pulls every ratio towards 1. With a package that idles at 5 W, sorts that draw 10 W and 12 W above idle have a power ratio of 1.2, but the package's total power (15 W against 17 W) gives 1.13. Judged on total power, time would look like a better stand-in for energy than it is for the work itself. The carbon section gives energy both ways.
  • Many comparisons. The report prints 252 intervals for the study profile. About 13 would exclude 1 by chance even if every configuration drew the same power, so a single "differs" reading proves little on its own. The report states how many exclude 1 against that expectation, and does not adjust the intervals.
  • The margin. Whether a 10% difference in power matters is a judgement, fixed before measuring. A reader who prefers another margin can rerun the report with --equivalence-margin; the intervals do not change, only their readings.

Workload and implementations

  • Instrumentation. Counting comparisons and writes adds work in every language, and proportionally more in Python than in C.

  • Just-in-time compilation and garbage collection. Node.js compiles hot code while it runs and both interpreters collect garbage. A process that sorts once measures more unoptimised code than one that sorts thousands of times. This is what running the program costs, but it makes per-sort figures depend on the number of sorts per process.

  • Helper threads and pinning. C and Python run one thread; a Node.js worker runs seven, because V8 compiles and collects garbage on helper threads. The study pins every worker to the same two CPUs, so these threads run beside the sorting thread, and their energy counts because RAPL measures the whole package. On a machine with more free cores, or with a different choice of CPUs, Node.js could be somewhat faster. Pinned and unpinned runs have not been compared.

  • Startup subtraction. Per-sort figures subtract a median startup cost measured in separate processes. Noise in that median shifts every per-sort value of the configuration, and the confidence intervals do not include it. Startup is measured over batches of processes started back to back; after the first, the files each process reads are in the page cache, so the cost per process describes a warm start. The sorting runs start warm too, because warm-up runs come first.

  • Calibration. Sorts per process are set from the warm-up runs, aiming 20% above the 200 ms target, and can still leave a measured loop shorter than the target on a busy machine. Shorter loops are still far longer than the counter's update interval; the report says how many there were and gives the shortest.

  • Language versions. Results describe the recorded compiler and interpreter versions, not the languages in general. The study script uses the system Python (3.12 on Ubuntu 24.04) and the Node.js 24 LTS that bare-metal.md installs from nodejs.org, and warns if an older Node.js is on the path.

  • Compiler code generation. GCC 14.2 at -O2 compiles the C bubble sort's swap into one 8-byte load and one 8-byte store that cover both neighbouring elements (SLP vectorisation). The next comparison loads 8 bytes that half overlap that store, which the processor cannot forward from its store buffer, so each swap waits for the store to reach the cache. The C bubble sort is therefore slow wherever it swaps often, and in the time run in Docker it was slower than Node.js on random and reversed input. A check on 2026-10-03, in the same setup as that run (Docker Desktop on a Windows 11 laptop with an AMD Ryzen 7 6800H, a container on CPUs 2 and 4 with 2 GiB of memory), measured C bubble sort at 2,000 elements with the harness, built as in the study and with -fno-tree-slp-vectorize added: 6.12 ms per sort on random input and 11.3 ms on reversed input, against 1.34 ms and 1.40 ms (medians of 30 runs, each 95% interval within 1.5% of its median). The commands ran inside docker run --rm --network none --cpuset-cpus 2,4 --memory 2g --memory-swap 2g greenbench:

    make all && make BUILD=build/noslp CFLAGS="-O2 -fno-tree-slp-vectorize" all
build/greenbench run --profile study --languages c --algorithms bubble --sizes 2000 \
    --energy off --cpus 2,4 --out results/o2
build/noslp/greenbench run --profile study --languages c --algorithms bubble --sizes 2000 \
    --energy off --cpus 2,4 --out results/noslp

    The study keeps the plain -O2 build: it measures what a default optimised build of this code does, and C bubble sort results describe that build and compiler.

Scope

  • One machine at a time, single-threaded sorts (Node.js adds its runtime's helper threads), small integer arrays, no input or output. The results say nothing about parallel code, large memory footprints or I/O-bound programs.
  • Carbon estimates use annual, location-based averages that include lifecycle emissions. They do not describe the marginal emissions of running the benchmark at a particular hour. Per-sort carbon net of idle counts only the extra power the sort draws; including idle, it also counts the idle power drawn while the sort runs. Neither includes the rest of the machine, so both understate the electricity used.

Reproducing

make && make test                       # build and unit tests, on any Linux
make setup                              # a virtual environment with the analysis packages
make smoke                              # a short run of all three languages with energy off
./scripts/run-docker-timing.sh          # with Docker: time for the study profile, energy off
./scripts/run-study.sh --check          # on bare-metal Linux: can this machine measure energy?
./scripts/run-study.sh                  # the study itself; writes docs/results.md

Related work

This study asks a narrower question than Pereira et al. (2017), who compared energy, time and memory across 27 languages on benchmark programs. Here the workload is small enough to check line by line, the inputs are provably identical across languages, and each language is measured by the same external harness with an idle baseline and confidence intervals.

References

  • Hähnel, M., Döbel, B., Völp, M. and Härtig, H. (2012). Measuring energy consumption for short code paths using RAPL. ACM SIGMETRICS Performance Evaluation Review, 40(3).
  • Khan, K. N., Hirki, M., Niemi, T., Nurminen, J. K. and Ou, Z. (2018). RAPL in action: experiences in using RAPL for power measurements. ACM Transactions on Modeling and Performance Evaluation of Computing Systems, 3(2).
  • Lipp, M., Kogler, A., Oswald, D., Schwarz, M., Easdon, C., Canella, C. and Gruss, D. (2021). PLATYPUS: software-based power side-channel attacks on x86. IEEE Symposium on Security and Privacy.
  • Marsaglia, G. (2003). Xorshift RNGs. Journal of Statistical Software, 8(14).
  • Pereira, R., Couto, M., Ribeiro, F., Rua, R., Cunha, J., Fernandes, J. P. and Saraiva, J. (2017). Energy efficiency across programming languages: how do energy, time, and memory relate? Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineering.
  • Ember (2026). Yearly electricity data. https://ember-energy.org/data/yearly-electricity-data/ (CC BY 4.0), via Our World in Data, https://ourworldindata.org/grapher/carbon-intensity-electricity.