From 40f2bf192b5095ac9aac47f6d959a338bdea88c8 Mon Sep 17 00:00:00 2001 From: Ayush Pathak <31178268+Raphaeal19@users.noreply.github.com> Date: Sat, 17 Oct 2026 01:04:51 -0400 Subject: [PATCH] android: add an on-device benchmark that prints the community submission block Add BenchTest, an instrumented test that measures decode and prefill throughput and peak memory on a real device and prints the submission block from the Hardware Benchmarks discussion (runtime: android). It runs one cold warmup and two measured runs over a fixed ~3.3k-token prompt, each decoding exactly 200 tokens, and fails the run if it leaves that shape. The block carries the mean; the notes list the cold run and every run, each with its own peak memory. tools/bench/run_bench.sh runs the test through adb and prints only the block; the test can also run as a Gradle connected test. README section 1.7 documents both, the protocol, the sampling settings the engine does and does not apply, and troubleshooting. The device model, SoC, RAM and Android version are collected automatically, and engine_commit comes from vendor.llama.pin through a new BuildConfig field. Refs #141 --- android/README.md | 225 +++++++++++++- android/app/build.gradle.kts | 5 + .../java/dev/edge0/runtime/app/BenchTest.kt | 88 ++++++ .../java/dev/edge0/runtime/app/bench/Bench.kt | 185 ++++++++++++ .../dev/edge0/runtime/app/bench/BenchBlock.kt | 79 +++++ .../edge0/runtime/app/bench/DeviceProbe.kt | 22 ++ .../edge0/runtime/app/bench/BenchBlockTest.kt | 100 +++++++ .../runtime/app/bench/BenchProtocolTest.kt | 281 ++++++++++++++++++ android/tools/bench/run_bench.sh | 61 ++++ 9 files changed, 1045 insertions(+), 1 deletion(-) create mode 100644 android/app/src/androidTest/java/dev/edge0/runtime/app/BenchTest.kt create mode 100644 android/app/src/main/java/dev/edge0/runtime/app/bench/Bench.kt create mode 100644 android/app/src/main/java/dev/edge0/runtime/app/bench/BenchBlock.kt create mode 100644 android/app/src/main/java/dev/edge0/runtime/app/bench/DeviceProbe.kt create mode 100644 android/app/src/test/java/dev/edge0/runtime/app/bench/BenchBlockTest.kt create mode 100644 android/app/src/test/java/dev/edge0/runtime/app/bench/BenchProtocolTest.kt create mode 100644 android/tools/bench/run_bench.sh diff --git a/android/README.md b/android/README.md index 5b78cf1..673b314 100644 --- a/android/README.md +++ b/android/README.md @@ -111,7 +111,229 @@ adb shell am instrument -w -e class dev.edge0.runtime.app.LlamaRuntimeTest \ Coverage: 8B/35B smoke, 35B↔8B in-process switching, prefix-reuse fidelity, thinking on/off × system-prompt quadrants (8B gate + 35B off leak probe), and identity retention across multi-turn rendering. Host-side logic tests: -`./gradlew :app:testDebugUnitTest` (32 tests, no device needed). +`./gradlew :app:testDebugUnitTest` (no device needed). + +### 1.7 Benchmark (community submission block) + +`BenchTest` measures decode speed, prefill speed and peak memory of a model on a real phone and +prints the result block used by the +[Hardware Benchmarks](https://github.com/Edge0-AI/Edge0/discussions/142) Discussion +(`runtime: android`). Besides the speed and memory numbers, the block records the device model, SoC, +RAM and Android version and the engine commit, so it is ready to paste. + +#### What you need + +- A phone connected over USB with USB debugging on (`adb devices` lists it as `device`). +- The app and the test APK installed (§1.3) and the model staged on the phone (§1.4). +- A tier that fits the phone: `8b` runs on any 8 GB+ phone, `35b` needs 12 GB+. + +#### Quick start + +```bash +./gradlew :app:installDebug :app:installDebugAndroidTest +bash tools/model/push_models.sh --8b # or --35b; restage if the app's data was cleared +bash tools/bench/run_bench.sh --8b --reporter @your-handle +``` + +The last command prints the block on stdout and nothing else, so it can be redirected: +`bash tools/bench/run_bench.sh --8b > block.txt`. + +#### The runner script: `tools/bench/run_bench.sh` + +``` +bash tools/bench/run_bench.sh --8b | --35b [--reporter @handle] [--notes "text"] [--temperature 0.2] +``` + +| Option | Required | Meaning | +|---|---|---| +| `--8b` / `--35b` | yes | The tier to run. If both are given, the last one wins. | +| `--reporter @handle` | no | Your handle, written to the `reporter` field. The `@` is added if missing. | +| `--notes "text"` | no | Free text appended to the `notes` field (for example `case on, charging`). Quotes and spaces are safe. | +| `--temperature 0.2` | no | Sampling temperature, 0 (greedy) to 2. The default, 0.2, is the app's own default. See "Sampling". | + +| Environment | Meaning | +|---|---| +| `ADB_SERIAL` | Which device to use. Required when more than one is attached (`adb devices` shows the serials). | + +What it does, in order: + +1. Checks that `adb` sees a usable device. If not, it stops with exit code 1 and tells you to check + `adb devices` or set `ADB_SERIAL`. +2. Starts the instrumented test with `adb shell am instrument -w`, passing the tier, reporter and + notes. It prints one line to stderr, then stays silent until the run ends (see "Following + progress"). +3. Reads the block out of the instrumentation output and prints it to stdout. +4. If no block was produced, or the test failed, it prints the full instrumentation output to + stderr and exits with code 1. The failure text from the test is in that output. + +Exit codes: `0` block printed, `1` no device or no block (including a failed test), `2` bad +arguments. + +The script does not check beforehand that the APKs and the model are on the phone. The test +reports it: a missing model fails with `edge0-.gguf not found under files/models`, and a +missing test APK fails with `INSTRUMENTATION_FAILED` and "Unable to find instrumentation info". + +Without the script, the same run is: + +```bash +adb shell am instrument -w -e class dev.edge0.runtime.app.BenchTest \ + -e tier 8b -e reporter @your-handle \ + dev.edge0.runtime.app.test/androidx.test.runner.AndroidJUnitRunner +``` + +Besides the instrumentation output, the test logs the block under the tag `E0Bench` and saves a +copy to `/sdcard/Android/data/dev.edge0.runtime.app/files/bench-edge0-.txt` (`adb pull`). +`BenchTest` does nothing unless `-e tier` is given, so other instrumented runs are unaffected. + +#### Running through Gradle + +The same test can be run as a Gradle connected test. Install the app and stage the model first, then +name the test class and pass the arguments as Gradle properties: + +```bash +./gradlew :app:installDebug # the app must exist before the model can be staged (§1.4) +./gradlew :app:connectedDebugAndroidTest \ + -Pandroid.testInstrumentationRunnerArguments.class=dev.edge0.runtime.app.BenchTest \ + -Pandroid.testInstrumentationRunnerArguments.tier=8b \ + -Pandroid.testInstrumentationRunnerArguments.reporter=@your-handle +``` + +- Always pass `class=`: without it Gradle runs every instrumented test, including + `LlamaRuntimeTest`, which needs both models. Without `tier=`, `BenchTest` skips itself. +- Gradle reinstalls the app in place, so the model you staged is still there for the run. When the run + ends it **uninstalls both APKs, which removes the staged model**, so stage it again for the next run. +- The console only shows pass or fail. The block is recorded in + `app/build/outputs/androidTest-results/connected/debug//` (`test-result.textproto`, key + `submission_block`) and in the logcat attachment next to it, which prints it with: + + ```bash + sed -n 's/.*E0Bench *: //p' app/build/outputs/androidTest-results/connected/debug/*/logcat-*BenchTest*.txt + ``` + +- Gradle warns that runner arguments given on the command line are not compatible with its + configuration cache. The arguments still reach the test; the run just does not use the cache. + +#### What the benchmark does + +- **Prompt:** one fixed prompt of about 3.3k tokens, built from the `BENCH_LONG=1` paragraphs of + `python/examples/bench.py` plus an instruction that asks for a long answer. Thinking is off. +- **Protocol:** one cold warmup run, then 2 measured runs with a 5 s pause between them. Each run + starts from scratch, so every run pays a full prefill. The block reports the mean of the 2 + measured runs. The warmup is not averaged, but its numbers and each measured run's are listed in + `notes`. +- **Decode:** exactly 200 new tokens, so every run measures an extended decode, sampled at + temperature 0.2 by default (the app's own default; `--temperature 0` is greedy). See "Sampling". + The engine cannot be told to ignore end-of-sequence, so a run where the model stops earlier fails + instead of reporting a shorter window. +- **Timing:** prefill and decode speeds come from the engine's own timers, as in the app's + metrics line. +- **Peak memory:** the process's peak resident set (`VmHWM`). The benchmark resets that counter + before every run (by writing `5` to `/proc/self/clear_refs`), so each run reports its own peak, and + the block shows the highest of the measured runs. If the phone refuses the reset, the number is the + whole-process maximum and `notes` says so. It includes memory-mapped model pages that are resident + at the time, so it is not the model's size and not comparable to MLX's allocator peak. +- **Comparable runs only:** a run fails instead of reporting if the prompt is outside 2.8k-3.8k + tokens, the prefix cache was reused, or decode did not produce exactly 200 tokens. + +Differences from `python/examples/bench.py`: the Android run reports a mean of 2 runs after a cold +warmup (the Python bench reports its cold and warm runs) and has no 10-token untimed decode warmup +inside a run; the model may end its answer early, which fails the run (the Python bench always +decodes 200 tokens); only the temperature is applied when sampling (next section); and peak memory +is the process's resident peak, not MLX's allocator peak. + +#### Sampling + +| Setting | `python/examples/bench.py` | Android benchmark | +|---|---|---| +| Temperature | `BENCH_TEMP`, default 0.7 | `--temperature`, default 0.2 (the app's default); 0 is greedy | +| Top-k | model config, default 64 | Not applied. With temperature above 0 the engine always uses top-k 40. | +| Top-p | model config, default 0.95 | Not applied. The engine has no top-p. | +| Repetition penalty | model config, default off | Not applied. | +| Seed | `BENCH_SEED`, optional | Fixed inside the engine. | + +The app's Settings screen has top-k and top-p sliders, but the llama path forwards only the +temperature to the engine (`LlamaRuntime.kt`), so they have no effect there either. Applying them +needs a change to the engine's decode loop (`llama_chat.c`), which the benchmark does not touch. + +#### The block + +``` +tier: edge0-8b +runtime: android +device: Samsung SM-S911U +soc: Snapdragon 8 Gen 2 +ram_gb: 8 +os: Android 16 (API 36) +decode_tok_s: 1.14 +prefill_tok_s: 10.3 +peak_active_gib: 3.40 +engine_commit: 7ab4ee7baad2d920464cbacfad4f4b07cf111fd2 +reporter: @your-handle +notes: cold run: decode 1.80 tok/s, prefill 10.4 tok/s, peak 3.54 GiB; run 1: decode 1.56 tok/s, prefill 10.0 tok/s, peak 3.33 GiB; run 2: decode 0.72 tok/s, prefill 10.7 tok/s, peak 3.40 GiB; 3301-token prompt, 200 decoded, mean of 2 reps, temperature 0.2 +``` + +The block above is the output of one run of the 8B tier on an 8 GB phone, where the 5 GB model does +not fit in free memory (see below). The screen turned off during run 2, which decoded at less than +half the speed of run 1; the notes list every run so that an outlier like this stays visible. + +| Field | Who fills it | +|---|---| +| `tier`, `runtime` | The benchmark. | +| `decode_tok_s`, `prefill_tok_s`, `peak_active_gib` | The benchmark (means of the 2 measured runs; peak is the highest of the two). | +| `engine_commit` | The benchmark: the pinned engine commit from `vendor.llama.pin`. | +| `reporter` | `--reporter`. | +| `notes` | The benchmark adds the cold run and each measured run (decode, prefill and, when the peak could be reset per run, peak), the prompt length, tokens decoded, run count and temperature; `--notes` is appended. | +| `device`, `soc`, `ram_gb`, `os` | The benchmark, from the Android framework. `device` is manufacturer and model. `soc` is the chip: known Qualcomm codes are mapped to marketing names (`SM8550` becomes Snapdragon 8 Gen 2), any other chip is shown as vendor and model code. `ram_gb` is total memory rounded up to the marketed size (a phone that reports 7.4 GiB shows 8). `os` is the Android release and API level. | + +#### Following progress + +The runner prints nothing until the run finishes. To follow progress, watch the engine log in a +second terminal: + +```bash +adb logcat -s E0LLAMA +``` + +The benchmark sends three requests: the warmup, then runs 1 and 2. Each logs a `REND` line when it +starts, `PRE fill ...` when its prefill is done, and `OK gen=... tps=... prefill_ms=...` when it +finishes, so the third `OK gen=` line means the run is nearly over. + +#### Getting representative numbers + +The model file is memory-mapped. When it is larger than the free memory on the phone, the weights +are read back from flash again and again, and speed drops sharply (an 8 GB phone running the 5 GB +8B model reaches only a few tokens per second, and the warmup alone can take several minutes). For a +representative number, reboot, close other apps, keep the screen on for the whole run, and note in +`--notes` whether the phone was charging and how much memory was free. The notes list every run, so a +run that was disturbed (for example the screen turning off partway) shows up as an outlier instead of +being hidden in the mean. + +#### Troubleshooting + +| Message | Meaning and fix | +|---|---| +| `no usable device` | `adb devices` shows nothing usable. Unlock the phone, accept the USB debugging prompt, or set `ADB_SERIAL`. | +| `INSTRUMENTATION_FAILED` / `Unable to find instrumentation info` | The test APK is not installed. Run `./gradlew :app:installDebugAndroidTest`. | +| `edge0-.gguf not found under files/models` | The model is not staged, or the app's data was cleared. Stage it again (§1.4). | +| `load failed: ...` | The engine could not load the model. The text after it is the engine's reason. | +| `prefix cache reused N tokens` | The run did not pay a full prefill, so the prefill speed would be wrong. Re-run. | +| `prompt is N tokens, expected 2800..3800` | The tokenizer produced a different prompt length than the benchmark was designed for. Report it. | +| `decode stopped after N tokens, need exactly 200` | The model ended its answer early, or the engine hit a decode error (for example a full KV cache). The engine reports such an error as a normal, shorter answer, so the two look the same. Re-run; if it repeats, report it with the tier and the number of tokens. | + +#### Code layout + +| File | Role | +|---|---| +| `app/src/main/java/dev/edge0/runtime/app/bench/Bench.kt` | Tiers, instrumentation arguments, the fixed prompt, the measurement protocol and its results. | +| `app/src/main/java/dev/edge0/runtime/app/bench/BenchBlock.kt` | Renders the submission block and the per-run notes; holds the device-name, SoC and RAM helpers. | +| `app/src/main/java/dev/edge0/runtime/app/bench/DeviceProbe.kt` | Reads the device model, SoC, RAM and Android version from the Android framework, and resets the peak-memory counter. | +| `app/src/androidTest/java/dev/edge0/runtime/app/BenchTest.kt` | The instrumented test: reads the arguments, loads the model, runs the protocol, prints the block. | +| `app/src/test/java/dev/edge0/runtime/app/bench/` | JVM unit tests for the protocol and the block (no device needed). | +| `tools/bench/run_bench.sh` | The runner script described above. | + +The protocol only depends on the app's `Runtime` interface, so it is unit-tested with a scripted +runtime. `app/build.gradle.kts` adds a `BuildConfig.ENGINE_COMMIT` field read from +`../vendor.llama.pin`. --- @@ -179,6 +401,7 @@ app/ Android app: Compose UI (src/main/java), JNI shell (src instrumented + unit tests (src/androidTest, src/test) tools/llama/ build_vendor_libs.sh — engine rebuild from the pinned tree + bands tools/model/ push_models.sh — md5-gated model staging to devices +tools/bench/ run_bench.sh — on-device benchmark runner (§1.7) ../vendor/llama.cpp/ materialized by the build scripts from vendor.llama.pin (gitignored; never patched in place) ../patches/llama.cpp/ common(6) + android(14) hook-point bands + ledger README ../windows/ desktop companion — hosts the MLX→GGUF converter used in §1.4 diff --git a/android/app/build.gradle.kts b/android/app/build.gradle.kts index 37876ca..deab95e 100644 --- a/android/app/build.gradle.kts +++ b/android/app/build.gradle.kts @@ -32,6 +32,11 @@ android { testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner" buildConfigField("String", "GIT_SHA", "\"${System.getenv("E0_GIT_SHA") ?: "unknown"}\"") + // engine provenance for the benchmark submission block (repo-root vendor.llama.pin) + val enginePin = providers.fileContents( + rootProject.layout.projectDirectory.dir("..").file("vendor.llama.pin") + ).asText.map { it.trim().substringBefore(' ') }.orElse("unknown") + buildConfigField("String", "ENGINE_COMMIT", "\"${enginePin.get()}\"") } buildTypes { debug { isMinifyEnabled = false } diff --git a/android/app/src/androidTest/java/dev/edge0/runtime/app/BenchTest.kt b/android/app/src/androidTest/java/dev/edge0/runtime/app/BenchTest.kt new file mode 100644 index 0000000..fe572f0 --- /dev/null +++ b/android/app/src/androidTest/java/dev/edge0/runtime/app/BenchTest.kt @@ -0,0 +1,88 @@ +// BenchTest - on-device benchmark that prints the community submission block. +// Skipped unless `-e tier 8b|35b` is given, so other instrumented runs are unaffected. +// Usage and model staging: android/README.md, "Benchmark". +package dev.edge0.runtime.app + +import android.content.Context +import android.os.Bundle +import android.util.Log +import androidx.test.platform.app.InstrumentationRegistry +import dev.edge0.runtime.app.bench.BenchArgs +import dev.edge0.runtime.app.bench.BenchConfig +import dev.edge0.runtime.app.bench.BenchRunner +import dev.edge0.runtime.app.bench.buildPrompt +import dev.edge0.runtime.app.bench.describeRuns +import dev.edge0.runtime.app.bench.findModel +import dev.edge0.runtime.app.bench.formatSubmissionBlock +import dev.edge0.runtime.app.bench.readDeviceInfo +import dev.edge0.runtime.app.bench.resetPeakMemory +import dev.edge0.runtime.app.runtime.LlamaRuntime +import java.io.File +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertNotNull +import org.junit.Assert.assertTrue +import org.junit.Assume.assumeTrue +import org.junit.Test + +class BenchTest { + + @Test + fun submissionBlock() = runBlocking { + val instrumentation = InstrumentationRegistry.getInstrumentation() + val raw = InstrumentationRegistry.getArguments() + val rawArgs = raw.keySet().associateWith { raw.getString(it) } + assumeTrue("pass -e ${BenchArgs.KEY_TIER} 8b|35b to run the benchmark", + BenchArgs.KEY_TIER in rawArgs) + val args = BenchArgs.parse(rawArgs) + val context = instrumentation.targetContext + + val model = findModel(modelRoots(context), args.tier) + assertNotNull("${args.tier.modelFile} not found under files/models (see README, Benchmark)", model) + + val runtime = LlamaRuntime() + runtime.ensureLoaded(model!!.absolutePath) + assertTrue("load failed: ${runtime.state.value}", runtime.activeModelDir != null) + + val summary = try { + BenchRunner(runtime, BenchConfig(temperature = args.temperature), resetPeak = ::resetPeakMemory) + .run(buildPrompt()) + } finally { + runtime.unloadActive() + } + + val shape = summary.reps.first() + val block = formatSubmissionBlock( + tier = args.tier, + device = readDeviceInfo(context), + summary = summary, + engineCommit = BuildConfig.ENGINE_COMMIT, + reporter = args.reporter, + notes = describeRuns(summary) + listOf( + "${shape.promptTokens}-token prompt, ${shape.newTokens} decoded, mean of ${summary.reps.size} reps, " + + "temperature ${args.temperature}", + args.notes, + ), + ) + publish(context.getExternalFilesDir(null) ?: context.filesDir, args.tier.id, block) + } + + private fun modelRoots(context: Context): List = listOf( + File(context.filesDir, "models"), + context.getDir("models", 0), + File(context.cacheDir, "models"), + context.getExternalFilesDir(null)?.let { File(it, "models") }, + ) + + /** Prints through `am instrument` and logcat, then saves a pullable copy; the copy is best effort. */ + private fun publish(outputDir: File, tierId: String, block: String) { + Log.i(TAG, "\n$block") + val instrumentation = InstrumentationRegistry.getInstrumentation() + instrumentation.sendStatus(0, Bundle().apply { putString("submission_block", "\n$block") }) + // "stream" is the standard key for a test's console output, which Gradle's connected-test report shows + instrumentation.sendStatus(0, Bundle().apply { putString("stream", "\n$block\n") }) + runCatching { File(outputDir, "bench-$tierId.txt").writeText(block + "\n") } + .onFailure { Log.w(TAG, "could not save the block to $outputDir", it) } + } + + private companion object { const val TAG = "E0Bench" } +} diff --git a/android/app/src/main/java/dev/edge0/runtime/app/bench/Bench.kt b/android/app/src/main/java/dev/edge0/runtime/app/bench/Bench.kt new file mode 100644 index 0000000..8228ae4 --- /dev/null +++ b/android/app/src/main/java/dev/edge0/runtime/app/bench/Bench.kt @@ -0,0 +1,185 @@ +// bench/Bench.kt - the benchmark protocol: tiers, instrumentation arguments, the fixed prompt, +// the measurement runner and its results, and model lookup. Depends only on the Runtime contract. +package dev.edge0.runtime.app.bench + +import dev.edge0.runtime.app.runtime.GenEvent +import dev.edge0.runtime.app.runtime.RequestMetrics +import dev.edge0.runtime.app.runtime.Runtime +import dev.edge0.runtime.engine.ChatMessage +import dev.edge0.runtime.engine.GenParams +import java.io.File +import kotlinx.coroutines.delay + +enum class BenchTier(val id: String, val modelFile: String) { + EDGE0_8B("edge0-8b", "edge0-8b.gguf"), + EDGE0_35B("edge0-35b", "edge0-35b.gguf"); + + companion object { + /** Accepts "8b", "35b" or the full "edge0-" id, case-insensitive. */ + fun parse(raw: String?): BenchTier? { + val key = raw?.trim()?.lowercase()?.removePrefix("edge0-") ?: return null + return entries.firstOrNull { it.id.removePrefix("edge0-") == key } + } + } +} + +/** Instrumentation arguments (`am instrument -e `). */ +data class BenchArgs(val tier: BenchTier, val reporter: String, val notes: String, val temperature: Float) { + companion object { + const val KEY_TIER = "tier" + const val KEY_REPORTER = "reporter" + const val KEY_NOTES = "notes" + const val KEY_TEMPERATURE = "temperature" + + /** The app's own default (Settings), so the benchmark samples like a normal chat turn. */ + const val DEFAULT_TEMPERATURE = 0.2f + private const val MAX_TEMPERATURE = 2f + + /** @throws IllegalArgumentException when `tier` is missing or invalid, or `temperature` is out of range. */ + fun parse(args: Map): BenchArgs { + val tier = BenchTier.parse(args[KEY_TIER]) + ?: throw IllegalArgumentException( + "-e $KEY_TIER must be 8b or 35b (got '${args[KEY_TIER].orEmpty()}')") + val reporter = args[KEY_REPORTER].orEmpty().trim() + val rawTemperature = args[KEY_TEMPERATURE].orEmpty().trim() + val temperature = if (rawTemperature.isEmpty()) DEFAULT_TEMPERATURE else + rawTemperature.toFloatOrNull()?.takeIf { it in 0f..MAX_TEMPERATURE } + ?: throw IllegalArgumentException( + "-e $KEY_TEMPERATURE must be a number from 0 (greedy) to $MAX_TEMPERATURE (got '$rawTemperature')") + return BenchArgs( + tier = tier, + reporter = if (reporter.isEmpty() || reporter.startsWith("@")) reporter else "@$reporter", + notes = args[KEY_NOTES].orEmpty().trim(), + temperature = temperature, + ) + } + } +} + +// Fixed long prompt so prefill numbers are comparable across devices. Paragraphs match +// python/examples/bench.py (BENCH_LONG). The instruction asks for a long answer because the +// engine stops at end-of-sequence and the decode window needs enough tokens. +private const val PARAGRAPH_COUNT = 40 +private const val INSTRUCTION = + "请仔细阅读以下材料，并写一篇不少于八百字的详细总结，分五个部分，每个部分至少一个完整段落：" + +internal const val PARAGRAPH_ZH = + "流式推理框架的设计要点在于把磁盘带宽、缓存层级与计算核心三者重叠" + + "起来。混合专家模型每一层只在少数专家上激活，路由器给出的稀疏选择" + + "恰好为按需加载提供了天然的批粒度。若能在前一步的隐状态基础上预判" + + "下一步的专家集合，固态硬盘的读取延迟就能被计算完全掩盖，这对边缘" + + "设备上的大模型部署具有直接意义。" + +internal const val PARAGRAPH_EN = + "The design of a streaming inference framework hinges on overlapping " + + "disk bandwidth, cache hierarchy, and compute. A mixture-of-experts " + + "layer activates only a handful of experts per token, and the router " + + "sparsity provides a natural granularity for on-demand loading. If " + + "the expert set of the next step can be predicted from the previous " + + "hidden state, SSD read latency hides behind compute entirely, which " + + "matters for large-model deployment on edge hardware." + +fun buildPrompt(): String = INSTRUCTION + + List(PARAGRAPH_COUNT) { if (it % 2 == 0) PARAGRAPH_ZH else PARAGRAPH_EN }.joinToString(" ") + +/** Finds the staged GGUF for a tier among candidate model roots; null entries are skipped. */ +fun findModel(roots: List, tier: BenchTier): File? = + roots.filterNotNull().map { File(it, tier.modelFile) }.firstOrNull { it.isFile } + +data class BenchConfig( + val reps: Int = 2, + val maxNewTokens: Int = 200, + val promptTokens: IntRange = 2_800..3_800, + val pauseBetweenRepsMs: Long = 5_000, + val temperature: Float = BenchArgs.DEFAULT_TEMPERATURE, +) + +// The measurement protocol: one cold warmup (kept for the notes, not averaged), then N measured +// reps, each a fresh request (the runtime rebuilds the KV on a forked history, so every rep pays +// a full prefill). resetPeak clears the process peak-memory counter before each run so every +// run reports its own peak; it returns false when the platform refuses. +class BenchRunner( + private val runtime: Runtime, + private val config: BenchConfig = BenchConfig(), + private val pause: suspend (Long) -> Unit = { delay(it) }, + private val resetPeak: () -> Boolean = { false }, +) { + /** @throws IllegalStateException when a turn fails or leaves the comparable measurement shape. */ + suspend fun run(prompt: String): BenchSummary { + var peakPerRun = resetPeak() + val cold = measure(prompt) + val reps = List(config.reps) { + pause(config.pauseBetweenRepsMs) + peakPerRun = resetPeak() && peakPerRun + measure(prompt) + } + return BenchSummary(reps, cold, peakPerRun) + } + + private suspend fun measure(prompt: String): RepResult { + var metrics: RequestMetrics? = null + var failure: String? = null + runtime.submit( + THREAD_ID, listOf(ChatMessage("user", prompt)), thinkingOn = false, + params = GenParams(temperature = config.temperature, maxNewTokens = config.maxNewTokens), + ).collect { event -> + when (event) { + is GenEvent.Done -> metrics = event.metrics + is GenEvent.Failed -> failure = event.diag + is GenEvent.Cancelled -> failure = "cancelled" + else -> Unit + } + } + check(failure == null) { "generation failed: $failure" } + return validated(checkNotNull(metrics) { "engine returned no metrics" }) + } + + private fun validated(metrics: RequestMetrics): RepResult { + check(metrics.prefix_reused_tokens == 0) { + "prefix cache reused ${metrics.prefix_reused_tokens} tokens; prefill would not be measured in full" + } + check(metrics.promptTokens in config.promptTokens) { + "prompt is ${metrics.promptTokens} tokens, expected ${config.promptTokens}" + } + // every run must decode the full length: it measures an extended run, and the engine reports a + // failed decode as a normal, shorter answer, which this check also catches + check(metrics.newTokens == config.maxNewTokens) { + "decode stopped after ${metrics.newTokens} tokens, need exactly ${config.maxNewTokens}" + } + return RepResult( + promptTokens = metrics.promptTokens, + newTokens = metrics.newTokens, + prefillTokS = metrics.prefillTokS, + decodeTokS = metrics.decodeTokS, + peakRssBytes = metrics.peakRssBytes, + ) + } + + private companion object { const val THREAD_ID = "bench" } +} + +private const val BYTES_PER_GIB = (1L shl 30).toDouble() + +data class RepResult( + val promptTokens: Int, + val newTokens: Int, + val prefillTokS: Double, + val decodeTokS: Double, + val peakRssBytes: Long, +) { + val peakGib: Double get() = peakRssBytes / BYTES_PER_GIB +} + +/** + * The block reports the mean of [reps]; [cold] is the discarded first run. [peakPerRun] is true when the + * peak-memory counter was reset before every run; otherwise each peak is the whole-process maximum so far. + */ +data class BenchSummary(val reps: List, val cold: RepResult? = null, val peakPerRun: Boolean = false) { + init { require(reps.isNotEmpty()) { "no measured reps" } } + + val prefillTokS: Double get() = reps.map { it.prefillTokS }.average() + val decodeTokS: Double get() = reps.map { it.decodeTokS }.average() + + /** Peak resident set (VmHWM) in GiB, the highest of the measured runs. */ + val peakActiveGib: Double get() = reps.maxOf { it.peakGib } +} diff --git a/android/app/src/main/java/dev/edge0/runtime/app/bench/BenchBlock.kt b/android/app/src/main/java/dev/edge0/runtime/app/bench/BenchBlock.kt new file mode 100644 index 0000000..03931c1 --- /dev/null +++ b/android/app/src/main/java/dev/edge0/runtime/app/bench/BenchBlock.kt @@ -0,0 +1,79 @@ +// bench/BenchBlock.kt - the community submission block (field names as in the "Hardware +// Benchmarks" Discussion template) and the device metadata it reports; helpers are pure and unit-tested. +package dev.edge0.runtime.app.bench + +import java.util.Locale + +private const val BLOCK_KEY_WIDTH = 17 + +fun formatSubmissionBlock( + tier: BenchTier, + device: DeviceInfo, + summary: BenchSummary, + engineCommit: String, + reporter: String, + notes: List, +): String = listOf( + "tier" to tier.id, + "runtime" to "android", + "device" to device.name, + "soc" to device.soc, + "ram_gb" to device.ramGb.toString(), + "os" to device.os, + "decode_tok_s" to formatNumber("%.2f", summary.decodeTokS), + "prefill_tok_s" to formatNumber("%.1f", summary.prefillTokS), + "peak_active_gib" to formatNumber("%.2f", summary.peakActiveGib), + "engine_commit" to engineCommit, + "reporter" to reporter, + "notes" to notes.filter { it.isNotBlank() }.joinToString("; "), +).joinToString("\n") { (key, value) -> ("$key:".padEnd(BLOCK_KEY_WIDTH) + value).trimEnd() } + +private fun formatNumber(pattern: String, value: Double) = String.format(Locale.ROOT, pattern, value) + +/** One note per run (cold run first), so the block carries every run and not only the mean. */ +fun describeRuns(summary: BenchSummary): List { + fun line(label: String, run: RepResult) = buildString { + append("$label: decode ${formatNumber("%.2f", run.decodeTokS)} tok/s") + append(", prefill ${formatNumber("%.1f", run.prefillTokS)} tok/s") + if (summary.peakPerRun) append(", peak ${formatNumber("%.2f", run.peakGib)} GiB") + } + val runs = listOfNotNull(summary.cold?.let { line("cold run", it) }) + + summary.reps.mapIndexed { index, run -> line("run ${index + 1}", run) } + return if (summary.peakPerRun) runs else runs + "peak is the whole-process maximum (per-run reset unavailable)" +} + +data class DeviceInfo(val name: String, val soc: String, val ramGb: Int, val os: String) { + companion object { + private const val BYTES_PER_GIB = 1L shl 30 + + // totalMem excludes kernel-reserved memory, so it reads below the marketed size. + private val MARKETED_RAM_GB = listOf(2, 3, 4, 6, 8, 10, 12, 16, 18, 24, 32) + + private val QUALCOMM_SOC_NAMES = mapOf( + "SM8350" to "Snapdragon 888", + "SM8450" to "Snapdragon 8 Gen 1", + "SM8475" to "Snapdragon 8+ Gen 1", + "SM8550" to "Snapdragon 8 Gen 2", + "SM8650" to "Snapdragon 8 Gen 3", + "SM8635" to "Snapdragon 8s Gen 3", + "SM8735" to "Snapdragon 8s Gen 4", + "SM8750" to "Snapdragon 8 Elite", + "SM8850" to "Snapdragon 8 Elite Gen 5", + ) + + fun marketedRamGb(totalBytes: Long): Int { + val gib = totalBytes.toDouble() / BYTES_PER_GIB + return MARKETED_RAM_GB.firstOrNull { it >= gib } ?: Math.ceil(gib).toInt() + } + + fun socName(manufacturer: String, model: String): String = + QUALCOMM_SOC_NAMES[model.trim().uppercase(Locale.ROOT)] + ?: listOf(manufacturer, model).map { it.trim() }.filter { it.isNotEmpty() } + .joinToString(" ").ifEmpty { "unknown" } + + fun deviceName(manufacturer: String, model: String): String { + val brand = manufacturer.trim().replaceFirstChar { it.uppercase() } + return if (model.startsWith(brand, ignoreCase = true)) model.trim() else "$brand ${model.trim()}".trim() + } + } +} diff --git a/android/app/src/main/java/dev/edge0/runtime/app/bench/DeviceProbe.kt b/android/app/src/main/java/dev/edge0/runtime/app/bench/DeviceProbe.kt new file mode 100644 index 0000000..3ddc90b --- /dev/null +++ b/android/app/src/main/java/dev/edge0/runtime/app/bench/DeviceProbe.kt @@ -0,0 +1,22 @@ +// bench/DeviceProbe.kt - reads the device metadata for the submission block and resets the process +// peak-memory counter; both need the Android framework or the Linux /proc interface. +package dev.edge0.runtime.app.bench + +import android.app.ActivityManager +import android.content.Context +import android.os.Build +import java.io.File + +fun readDeviceInfo(context: Context): DeviceInfo { + val memory = ActivityManager.MemoryInfo() + context.getSystemService(ActivityManager::class.java).getMemoryInfo(memory) + return DeviceInfo( + name = DeviceInfo.deviceName(Build.MANUFACTURER, Build.MODEL), + soc = DeviceInfo.socName(Build.SOC_MANUFACTURER, Build.SOC_MODEL), + ramGb = DeviceInfo.marketedRamGb(memory.totalMem), + os = "Android ${Build.VERSION.RELEASE} (API ${Build.VERSION.SDK_INT})", + ) +} + +/** Resets the kernel's peak-RSS counter (VmHWM) to the current RSS; false when the kernel refuses. */ +fun resetPeakMemory(): Boolean = runCatching { File("/proc/self/clear_refs").writeText("5") }.isSuccess diff --git a/android/app/src/test/java/dev/edge0/runtime/app/bench/BenchBlockTest.kt b/android/app/src/test/java/dev/edge0/runtime/app/bench/BenchBlockTest.kt new file mode 100644 index 0000000..7309b42 --- /dev/null +++ b/android/app/src/test/java/dev/edge0/runtime/app/bench/BenchBlockTest.kt @@ -0,0 +1,100 @@ +// BenchBlockTest - submission block field order and formatting, plus the device metadata helpers. +package dev.edge0.runtime.app.bench + +import org.junit.Assert.assertEquals +import org.junit.Test + +class BenchBlockTest { + + private val device = DeviceInfo( + name = "Samsung SM-S911B", soc = "Snapdragon 8 Gen 2", ramGb = 8, os = "Android 15 (API 35)") + private val summary = BenchSummary(listOf( + RepResult(3300, 200, prefillTokS = 100.04, decodeTokS = 10.123, peakRssBytes = 3L shl 30))) + + private fun gib(value: Double) = (value * (1L shl 30)).toLong() + + @Test fun rendersTemplateFieldsInOrder() { + val block = formatSubmissionBlock( + tier = BenchTier.EDGE0_8B, device = device, summary = summary, + engineCommit = "7ab4ee7", reporter = "@handle", notes = listOf("thermal NONE->LIGHT", "case on")) + val expected = """ + tier: edge0-8b + runtime: android + device: Samsung SM-S911B + soc: Snapdragon 8 Gen 2 + ram_gb: 8 + os: Android 15 (API 35) + decode_tok_s: 10.12 + prefill_tok_s: 100.0 + peak_active_gib: 3.00 + engine_commit: 7ab4ee7 + reporter: @handle + notes: thermal NONE->LIGHT; case on + """.trimIndent() + assertEquals(expected, block) + } + + @Test fun emptyOptionalFieldsLeaveNoTrailingSpaces() { + val block = formatSubmissionBlock( + tier = BenchTier.EDGE0_35B, device = device, summary = summary, + engineCommit = "7ab4ee7", reporter = "", notes = emptyList()) + assertEquals(listOf("reporter:", "notes:"), block.lines().takeLast(2)) + } + + @Test fun blankNotesAreDropped() { + val block = formatSubmissionBlock( + tier = BenchTier.EDGE0_8B, device = device, summary = summary, + engineCommit = "x", reporter = "@a", notes = listOf("", "only")) + assertEquals("notes: only", block.lines().last()) + } + + private fun run(decode: Double, prefill: Double, peakGib: Long) = + RepResult(3300, 200, prefillTokS = prefill, decodeTokS = decode, peakRssBytes = peakGib shl 30) + + @Test fun describesEveryRunWithPerRunPeaks() { + val runs = BenchSummary( + reps = listOf(run(1.5, 10.0, 3), run(1.25, 9.5, 2)), cold = run(1.0, 8.0, 4), peakPerRun = true) + assertEquals(listOf( + "cold run: decode 1.00 tok/s, prefill 8.0 tok/s, peak 4.00 GiB", + "run 1: decode 1.50 tok/s, prefill 10.0 tok/s, peak 3.00 GiB", + "run 2: decode 1.25 tok/s, prefill 9.5 tok/s, peak 2.00 GiB", + ), describeRuns(runs)) + } + + @Test fun omitsPerRunPeaksWhenTheResetWasUnavailable() { + val runs = BenchSummary(reps = listOf(run(1.5, 10.0, 3)), cold = null, peakPerRun = false) + assertEquals(listOf( + "run 1: decode 1.50 tok/s, prefill 10.0 tok/s", + "peak is the whole-process maximum (per-run reset unavailable)", + ), describeRuns(runs)) + } + + @Test fun ramRoundsUpToTheMarketedSize() { + assertEquals(8, DeviceInfo.marketedRamGb(gib(7.4))) + assertEquals(12, DeviceInfo.marketedRamGb(gib(11.2))) + assertEquals(16, DeviceInfo.marketedRamGb(gib(14.9))) + assertEquals(6, DeviceInfo.marketedRamGb(gib(5.6))) + } + + @Test fun ramBeyondTheTableRoundsUpToWholeGib() { + assertEquals(33, DeviceInfo.marketedRamGb(gib(32.4))) + } + + @Test fun knownQualcommCodesMapToMarketingNames() { + assertEquals("Snapdragon 8 Gen 2", DeviceInfo.socName("Qualcomm", "SM8550")) + assertEquals("Snapdragon 8 Elite", DeviceInfo.socName("QTI", "sm8750")) + assertEquals("Snapdragon 8s Gen 3", DeviceInfo.socName("Qualcomm", "SM8635")) + assertEquals("Snapdragon 8s Gen 4", DeviceInfo.socName("Qualcomm", "SM8735")) + assertEquals("Snapdragon 8 Elite Gen 5", DeviceInfo.socName("Qualcomm", "SM8850")) + } + + @Test fun unknownSocFallsBackToVendorAndModel() { + assertEquals("Google Tensor G3", DeviceInfo.socName("Google", "Tensor G3")) + assertEquals("unknown", DeviceInfo.socName("", "")) + } + + @Test fun deviceNameAvoidsRepeatingTheManufacturer() { + assertEquals("Samsung SM-S911B", DeviceInfo.deviceName("samsung", "SM-S911B")) + assertEquals("Google Pixel 9", DeviceInfo.deviceName("Google", "Google Pixel 9")) + } +} diff --git a/android/app/src/test/java/dev/edge0/runtime/app/bench/BenchProtocolTest.kt b/android/app/src/test/java/dev/edge0/runtime/app/bench/BenchProtocolTest.kt new file mode 100644 index 0000000..21979e9 --- /dev/null +++ b/android/app/src/test/java/dev/edge0/runtime/app/bench/BenchProtocolTest.kt @@ -0,0 +1,281 @@ +// BenchProtocolTest - argument and tier parsing, the fixed prompt (including fidelity to +// python/examples/bench.py), result aggregation, the measurement protocol and model lookup. +package dev.edge0.runtime.app.bench + +import dev.edge0.runtime.app.runtime.GenEvent +import dev.edge0.runtime.app.runtime.GenState +import dev.edge0.runtime.app.runtime.RequestMetrics +import dev.edge0.runtime.app.runtime.Runtime +import dev.edge0.runtime.engine.ChatMessage +import dev.edge0.runtime.engine.GenParams +import dev.edge0.runtime.engine.Status +import java.io.File +import kotlinx.coroutines.flow.Flow +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.flowOf +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Assume.assumeTrue +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder + +class BenchProtocolTest { + + @get:Rule val tmp = TemporaryFolder() + + // unit tests run with the module directory (android/app) as working directory + private val pythonBench = File("../../python/examples/bench.py") + + private fun pythonParagraph(source: String, name: String): String { + val body = Regex("$name = \\((.*?)\\)\\n", RegexOption.DOT_MATCHES_ALL) + .find(source)!!.groupValues[1] + return Regex("\"((?:[^\"\\\\]|\\\\.)*)\"").findAll(body).joinToString("") { it.groupValues[1] } + } + + private fun rep(prefill: Double, decode: Double, peakBytes: Long) = + RepResult(promptTokens = 3300, newTokens = 200, prefillTokS = prefill, + decodeTokS = decode, peakRssBytes = peakBytes) + + private class ScriptedRuntime(private val turns: List) : Runtime { + override val state = MutableStateFlow(GenState.Idle) + override val lastMetrics = MutableStateFlow(null) + override val activeModelDir: String? = "model" + val submitted = mutableListOf, Boolean, GenParams>>() + + override suspend fun ensureLoaded(path: String) = Unit + override suspend fun unloadActive() = Unit + override suspend fun trim(ratio: Float) = true + override fun cancelActive() = Unit + override fun submit(threadId: String, messages: List, + thinkingOn: Boolean, params: GenParams): Flow { + submitted += Triple(messages, thinkingOn, params) + return flowOf(turns[submitted.size - 1]) + } + } + + private fun metrics( + prompt: Int = 3300, reused: Int = 0, generated: Int = 200, + prefill: Double = 100.0, decode: Double = 10.0, peak: Long = 1L shl 30, + ) = RequestMetrics( + promptTokens = prompt, prefix_reused_tokens = reused, newTokens = generated, + prefillTokS = prefill, decodeTokS = decode, peakRssBytes = peak) + + private fun done(m: RequestMetrics = metrics()) = GenEvent.Done(m) + + private fun runner(runtime: Runtime, pauses: MutableList = mutableListOf()) = + BenchRunner(runtime, BenchConfig(), pause = { pauses += it }) + + @Test fun tierAcceptsShortAndFullNames() { + assertEquals(BenchTier.EDGE0_8B, BenchTier.parse("8b")) + assertEquals(BenchTier.EDGE0_8B, BenchTier.parse("edge0-8b")) + assertEquals(BenchTier.EDGE0_35B, BenchTier.parse(" 35B ")) + assertEquals(BenchTier.EDGE0_35B, BenchTier.parse("Edge0-35b")) + } + + @Test fun tierRejectsUnknownValues() { + assertNull(BenchTier.parse("70b")) + assertNull(BenchTier.parse("")) + assertNull(BenchTier.parse(null)) + } + + @Test fun parsesAllArguments() { + val args = BenchArgs.parse(mapOf("tier" to "35b", "reporter" to "@handle", "notes" to "case on")) + assertEquals(BenchTier.EDGE0_35B, args.tier) + assertEquals("@handle", args.reporter) + assertEquals("case on", args.notes) + } + + @Test fun reporterGetsHandlePrefix() { + assertEquals("@handle", BenchArgs.parse(mapOf("tier" to "8b", "reporter" to "handle")).reporter) + } + + @Test fun optionalArgumentsDefaultToEmpty() { + val args = BenchArgs.parse(mapOf("tier" to "8b")) + assertEquals("", args.reporter) + assertEquals("", args.notes) + } + + @Test fun invalidTierNamesTheAcceptedValues() { + val error = assertThrows(IllegalArgumentException::class.java) { + BenchArgs.parse(mapOf("tier" to "70b")) + } + assertEquals(true, error.message!!.contains("8b") && error.message!!.contains("35b")) + } + + @Test fun missingTierIsRejected() { + assertThrows(IllegalArgumentException::class.java) { BenchArgs.parse(emptyMap()) } + } + + @Test fun temperatureDefaultsToTheAppSetting() { + assertEquals(BenchArgs.DEFAULT_TEMPERATURE, BenchArgs.parse(mapOf("tier" to "8b")).temperature) + } + + @Test fun parsesTemperature() { + assertEquals(0f, BenchArgs.parse(mapOf("tier" to "8b", "temperature" to "0")).temperature) + assertEquals(0.7f, BenchArgs.parse(mapOf("tier" to "8b", "temperature" to " 0.7 ")).temperature) + } + + @Test fun rejectsAnOutOfRangeOrNonNumericTemperature() { + listOf("abc", "-0.1", "2.5", "NaN").forEach { bad -> + assertThrows(IllegalArgumentException::class.java) { + BenchArgs.parse(mapOf("tier" to "8b", "temperature" to bad)) + } + } + } + + @Test fun promptIsDeterministic() { + assertEquals(buildPrompt(), buildPrompt()) + } + + @Test fun promptAlternatesChineseAndEnglishParagraphs() { + val prompt = buildPrompt() + assertEquals(20, Regex("流式推理框架").findAll(prompt).count()) + assertEquals(20, Regex("The design of a streaming").findAll(prompt).count()) + } + + @Test fun paragraphsMatchThePythonBench() { + assumeTrue("python/examples/bench.py not reachable from the test working directory", pythonBench.isFile) + val source = pythonBench.readText() + assertEquals(pythonParagraph(source, "_PARA_ZH"), PARAGRAPH_ZH) + assertEquals(pythonParagraph(source, "_PARA_EN"), PARAGRAPH_EN) + } + + @Test fun summaryAveragesThroughput() { + val summary = BenchSummary(listOf(rep(100.0, 10.0, 1L shl 30), rep(120.0, 14.0, 1L shl 30))) + assertEquals(110.0, summary.prefillTokS, 1e-9) + assertEquals(12.0, summary.decodeTokS, 1e-9) + } + + @Test fun summaryReportsLargestPeakInGib() { + val summary = BenchSummary(listOf(rep(1.0, 1.0, 2L shl 30), rep(1.0, 1.0, 3L shl 30))) + assertEquals(3.0, summary.peakActiveGib, 1e-9) + } + + @Test fun emptySummaryIsRejected() { + assertThrows(IllegalArgumentException::class.java) { BenchSummary(emptyList()) } + } + + @Test fun discardsWarmupAndAveragesTheMeasuredReps() = runBlocking { + val runtime = ScriptedRuntime(listOf( + done(metrics(decode = 99.0)), done(metrics(decode = 10.0)), done(metrics(decode = 14.0)))) + val summary = runner(runtime).run("prompt") + assertEquals(2, summary.reps.size) + assertEquals(12.0, summary.decodeTokS, 1e-9) + } + + @Test fun everyTurnIsAFreshThinkingOffRequest() = runBlocking { + val runtime = ScriptedRuntime(List(3) { done() }) + runner(runtime).run("prompt") + assertEquals(3, runtime.submitted.size) + runtime.submitted.forEach { (messages, thinking, params) -> + assertEquals(listOf(ChatMessage("user", "prompt")), messages) + assertEquals(false, thinking) + assertEquals(BenchArgs.DEFAULT_TEMPERATURE, params.temperature) + assertEquals(200, params.maxNewTokens) + } + } + + @Test fun usesTheConfiguredTemperature() = runBlocking { + val runtime = ScriptedRuntime(List(3) { done() }) + BenchRunner(runtime, BenchConfig(temperature = 0f), pause = {}).run("prompt") + assertEquals(setOf(0f), runtime.submitted.map { it.third.temperature }.toSet()) + } + + @Test fun keepsTheColdRunApartFromTheMean() = runBlocking { + val runtime = ScriptedRuntime(listOf(done(metrics(decode = 99.0)), done(), done())) + val summary = runner(runtime).run("prompt") + assertEquals(99.0, summary.cold!!.decodeTokS, 1e-9) + assertEquals(10.0, summary.decodeTokS, 1e-9) + } + + @Test fun resetsThePeakBeforeEveryRun() = runBlocking { + var resets = 0 + val summary = BenchRunner(ScriptedRuntime(List(3) { done() }), BenchConfig(), pause = {}, + resetPeak = { resets++; true }).run("prompt") + assertEquals(3, resets) + assertEquals(true, summary.peakPerRun) + } + + @Test fun fallsBackToTheWholeProcessPeakWhenTheResetIsRefused() = runBlocking { + val summary = BenchRunner(ScriptedRuntime(List(3) { done() }), BenchConfig(), pause = {}, + resetPeak = { false }).run("prompt") + assertEquals(false, summary.peakPerRun) + } + + @Test fun pausesBeforeEachMeasuredRep() = runBlocking { + val pauses = mutableListOf() + runner(ScriptedRuntime(List(3) { done() }), pauses).run("prompt") + assertEquals(listOf(5_000L, 5_000L), pauses) + } + + @Test fun rejectsAReusedPrefix() { + val runtime = ScriptedRuntime(listOf(done(), done(metrics(reused = 3000)))) + val error = assertThrows(IllegalStateException::class.java) { + runBlocking { runner(runtime).run("prompt") } + } + assertTrue(error.message!!.contains("prefix")) + } + + @Test fun rejectsAPromptOutsideTheComparableRange() { + val runtime = ScriptedRuntime(listOf(done(metrics(prompt = 900)))) + val error = assertThrows(IllegalStateException::class.java) { + runBlocking { runner(runtime).run("prompt") } + } + assertTrue(error.message!!.contains("900")) + } + + @Test fun rejectsADecodeThatStoppedEarly() { + val runtime = ScriptedRuntime(listOf(done(metrics(generated = 40)))) + val error = assertThrows(IllegalStateException::class.java) { + runBlocking { runner(runtime).run("prompt") } + } + assertTrue(error.message!!.contains("40") && error.message!!.contains("exactly 200")) + } + + @Test fun rejectsADecodeOneTokenShort() { + val runtime = ScriptedRuntime(listOf(done(), done(metrics(generated = 199)))) + assertThrows(IllegalStateException::class.java) { + runBlocking { runner(runtime).run("prompt") } + } + } + + @Test fun surfacesEngineFailures() { + val runtime = ScriptedRuntime(listOf(GenEvent.Failed(Status.IO, "ERR prefill@512"))) + val error = assertThrows(IllegalStateException::class.java) { + runBlocking { runner(runtime).run("prompt") } + } + assertTrue(error.message!!.contains("ERR prefill@512")) + } + + @Test fun surfacesCancellation() { + val runtime = ScriptedRuntime(listOf(GenEvent.Cancelled(null))) + val error = assertThrows(IllegalStateException::class.java) { + runBlocking { runner(runtime).run("prompt") } + } + assertTrue(error.message!!.contains("cancelled")) + } + + @Test fun rejectsATurnWithoutMetrics() { + val runtime = ScriptedRuntime(listOf(GenEvent.Done(null))) + assertThrows(IllegalStateException::class.java) { + runBlocking { runner(runtime).run("prompt") } + } + } + + @Test fun findsTheTierFileInTheFirstRootThatHasIt() { + val first = tmp.newFolder("first") + val second = tmp.newFolder("second") + val model = File(second, "edge0-35b.gguf").apply { writeText("x") } + assertEquals(model, findModel(listOf(first, null, second), BenchTier.EDGE0_35B)) + } + + @Test fun ignoresOtherTiersAndMissingFiles() { + val root = tmp.newFolder("root") + File(root, "edge0-8b.gguf").writeText("x") + assertNull(findModel(listOf(root), BenchTier.EDGE0_35B)) + } +} diff --git a/android/tools/bench/run_bench.sh b/android/tools/bench/run_bench.sh new file mode 100644 index 0000000..d7ba039 --- /dev/null +++ b/android/tools/bench/run_bench.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# run_bench.sh - run the on-device benchmark and print the community submission block. +# Usage: bash tools/bench/run_bench.sh --8b | --35b [--reporter @handle] [--notes "text"] [--temperature 0.2] +# ADB_SERIAL selects the device; required when more than one is attached. +# Prerequisites: app and test APK installed and the model staged (android/README.md, "Benchmark"). +set -euo pipefail + +TEST_CLASS=dev.edge0.runtime.app.BenchTest +RUNNER=dev.edge0.runtime.app.test/androidx.test.runner.AndroidJUnitRunner + +usage() { + echo 'usage: run_bench.sh --8b | --35b [--reporter @handle] [--notes "text"] [--temperature 0.2]' >&2 + exit 2 +} + +# single-quote a value for the device shell +shell_quote() { printf "'%s'" "$(printf '%s' "$1" | sed "s/'/'\\\\''/g")"; } + +TIER="" +REPORTER="" +NOTES="" +TEMPERATURE="" +while [ $# -gt 0 ]; do + case "$1" in + --8b) TIER=8b ;; + --35b) TIER=35b ;; + --reporter) [ $# -ge 2 ] || usage; REPORTER=$2; shift ;; + --notes) [ $# -ge 2 ] || usage; NOTES=$2; shift ;; + --temperature) [ $# -ge 2 ] || usage; TEMPERATURE=$2; shift ;; + *) usage ;; + esac + shift +done +[ -n "$TIER" ] || usage + +ADB=(adb ${ADB_SERIAL:+-s "$ADB_SERIAL"}) +"${ADB[@]}" get-state >/dev/null || { + echo "ABORT: no usable device (see 'adb devices'; set ADB_SERIAL when several are attached)" >&2 + exit 1 +} + +command="am instrument -w -e class $TEST_CLASS -e tier $TIER" +if [ -n "$REPORTER" ]; then command="$command -e reporter $(shell_quote "$REPORTER")"; fi +if [ -n "$NOTES" ]; then command="$command -e notes $(shell_quote "$NOTES")"; fi +if [ -n "$TEMPERATURE" ]; then command="$command -e temperature $(shell_quote "$TEMPERATURE")"; fi + +echo "Running the $TIER benchmark (a warmup and two timed reps; this can take several minutes)..." >&2 +output=$("${ADB[@]}" shell "$command $RUNNER" | tr -d '\r') || true + +# the block is a multi-line status value; its start marker follows the test class name on one line +block=$(printf '%s\n' "$output" | sed -n '/INSTRUMENTATION_STATUS: submission_block=/,/^INSTRUMENTATION_STATUS_CODE/{ + /INSTRUMENTATION_STATUS/d + p +}') + +if [ -z "$block" ] || printf '%s\n' "$output" | grep -q 'FAILURES!!!'; then + echo "ABORT: benchmark did not produce a submission block. Instrumentation output:" >&2 + printf '%s\n' "$output" >&2 + exit 1 +fi +printf '%s\n' "$block"