Skip to content

feat(ios): on-device scoring with the model pack, as on Android - #18

Open
IgnacioLD wants to merge 2 commits into
mainfrom
feat/ios-on-device-scoring
Open

IgnacioLD wants to merge 2 commits into
mainfrom
feat/ios-on-device-scoring

Conversation

@IgnacioLD

Copy link
Copy Markdown
Owner

The iOS app now scores takes on the phone when the model pack is installed, with the same pipeline and the same score as Android and the server. This replaces NoOnDeviceScoring.

What changed

  • IosOnDeviceScorer (shared/src/iosMain) is the iOS twin of Android's OnDeviceScorer:
    • The pack is stored in Application Support/scoring, flagged with NSURLIsExcludedFromBackupKey.
    • It downloads through the app's queue (ForegroundDownloader), checked against the digest the server publishes.
    • The model loads on Dispatchers.IO and is released on UIApplicationDidEnterBackgroundNotification and on memory warnings.
    • A background URLSession would only have to replace DownloadFile.fetch.
  • Shared code instead of duplicates.
    • The scoring pipeline (AccentScoring with variants, PhoneticProfile, StressAnalyzer, FeedbackAssembler) is now scoreOnDevice in shared commonMain. Android's scoreLoaded calls it, and the code moved verbatim.
    • ModelPackImport and the pinned/published digests also moved to shared common code, ported from java.io to okio.
  • Import from Files: rememberFilePicker on iOS uses UIDocumentPickerViewController, opened in place with security-scoped access. The 358 MB model is not copied into the app inbox first.
  • Screens: now that supported is true, onboarding, Settings, the offline drawer and the practice screen offer the model on iOS. When a take is offline and there is no pack, the app offers the download instead of an error.
  • C shim memory options: DisableCpuMemArena and session.use_device_allocator_for_initializers=1, as on Android since build: AGP 9.3 with R8 optimized resource shrinking, and lower on-device scoring memory #16. This is a separate commit.
  • Docs: docs/IOS-ENGINE.md and docs/IOS-DEV.md are updated.

Deployment target is still iOS 16.0.

Evidence (iPhone 17 simulator, iOS 27, Debug)

  • Parity gate (tools/score-parity/run-ios-parity.sh): PASS before and after the shim change, 49/49 identical final scores, and the 4 fixtures match the server golden. The iOS logits are bit-identical before and after the memory options (47/47 clips, all 49 records identical).

  • Memory from the parity run (phys_footprint, 77 clips):

    Model loaded After the 77 clips
    Before 400 MB (peak 538 MB) 666 MB (peak 706 MB)
    After 399 MB (peak 537 MB) 468 MB (peak 592 MB)
  • Memory in the app (footprint -p):

    • idle: 105 MB
    • after an on-device take: ~360-490 MB (peak ~500 MB)
    • after going to the background: 126 MB (the model is released)
    • after resuming, the next take reloads the model and scores the same.
  • End to end against a local server (:server:run with the deployed assets):

    • Download from onboarding: progress shown, files land in Application Support with the backup-exclusion xattr, sha256 matches the pinned digest.
    • Import from Files: renamed copies (model-copy.onnx, vocab-copy.json) install under the right names with the right digests.
  • Same score as the cloud. I replayed the exact ScoreRequest the app built (captured with a temporary hook, not committed) against /v1/score with the same WAV, for "Three people thank the firefighters." (a drill with content variants):

    • score, accuracy, completeness, level and transcription are identical: 94 / 94 / 100 / ALMOST.
    • The remaining numeric differences (timings, quality, medianVowelMs) come from the ORT version (server 1.20 vs device 1.30), already documented in IOS-ENGINE.md.
    • acoustics and fluency are server-only extras that Android on-device also leaves out.
  • Tests:

    • :engine:jvmTest: 463 passed
    • :shared:testAndroidHostTest: 146 passed
    • :androidApp:testFdroidDebugUnitTest / testPlaystoreDebugUnitTest: 296 / 295 passed (includes the updated ModelPackImportTest)
    • :shared:iosSimulatorArm64Test: 84 passed
    • :androidApp:assembleFdroidDebug and assemblePlaystoreDebug build, and :androidApp:lintFdroidDebug passes.

Known gap, not introduced here (owner decision)

Dialogue turns in "A Call Home" ("I'm fine, thanks, Mum. And you?") scored 88 on-device vs 91 in the cloud, with the same transcription:

  • Why: the cloud widens "and" with CMUdict's strong form /ænd/ (CLOSE). The device only gets the content's ipaVariants. Dialogue turns ship none, and the recall-card path calls engine.toggle(text, ipa, accentIpa = ...) with no variants, so /ənd/ vs heard /a/ is MISSED.
  • Scope: this is shared code, so Android on-device has the same gap today.
  • Fix: bake variants into dialogue turns, or pass them in LessonViewModel. That moves scores, so it is not in this PR.

Not verified

  • Real iPhone (XS Max) load time, inference time and memory.
  • The memory-warning path (same code as the background path).
  • A background URLSession, which is a separate task.

Risks

  • On iOS the import hashes with okio's Kotlin SHA-256, which took about 10 s for the model in the simulator.
  • Android's ModelPackImport now goes through okio instead of java.io. Its unit test still passes.

https://claude.ai/code/session_01FcVpZoyr8VdRLp1QecX1Ng

…Android

The C shim now disables the CPU memory arena and sets
session.use_device_allocator_for_initializers=1, as the JVM/Android
recognizer does since #16. Over the 77-clip parity run the footprint
after scoring drops from 666 MB to 468 MB (peak 706 -> 592 MB); the iOS
logits are bit-identical with and without the options.

Claude-Session: https://claude.ai/code/session_01FcVpZoyr8VdRLp1QecX1Ng
- IosOnDeviceScorer replaces NoOnDeviceScoring: the pack lives in
  Application Support/scoring, excluded from the iCloud backup, downloads
  through the app's queue with the published digest, loads off the main
  thread and is released on background and memory warnings.
- The scoring pipeline (scoreOnDevice) and ModelPackImport move to shared
  common code, so Android and iOS run the same code; Android behaviour
  is unchanged.
- Import from Files with the document picker, opened in place.
- The screens now offer the model on iOS, since the scorer is supported.

Claude-Session: https://claude.ai/code/session_01FcVpZoyr8VdRLp1QecX1Ng
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant