Whisper wouldn’t run on iPhone: CoreML error -14 and the Metal way out

Context

Hi everyone. I’m going to tell the story of a problem that cost me a few days and had a much less obvious solution than I expected: the same Whisper model ran perfectly on the Mac and flat out refused to run on the iPhone. The spoiler is that the culprit was neither the model nor the device. It was the path I was using to run the model.

A bit of context first. I’m building Voxfloy, a speech-to-text app that runs on-device, meaning the audio never leaves the phone. That constraint is intentional and it rules out the easy way out of sending the audio to a server to transcribe. So I need the model to actually run on the phone.

The obvious choice on the Apple platform is WhisperKit. It’s open source, well maintained, has a Swift API that’s pleasant to use, runs Whisper via CoreML and lets the system distribute the layers across Neural Engine, GPU and CPU. On the Mac, first try, it worked beautifully. I pinned large-v3-turbo, quality came out great, latency low. I filed the topic as solved in my head.

Then I went to run it on the iPhone. And it didn’t run.

Where it broke (in two different ways)

The interesting part is that it wasn’t a “ran slower” or “the transcription came out worse”. It just didn’t run, in two different ways, depending on where I pushed the model.

The Neural Engine can’t compile the encoder

With large-v3-turbo pinned, CoreML tries to build the model on the Neural Engine. On the iPhone, the log said this:

MILCompilerForANE error: ANECCompile() FAILED
... std::bad_alloc
Failed to build the model execution plan ... error code: -14

Translation: the Neural Engine compiler runs out of memory trying to build the AudioEncoder’s execution plan. That std::bad_alloc is an out-of-memory inside the compiler itself (not in my app), and -14 is CoreML throwing in the towel. On the Mac the same model compiles fine, because there’s plenty of ANE and memory headroom. On the phone, there isn’t.

Forcing the GPU, the app dies

The natural reaction was: if the ANE can’t handle it, send it to the GPU. CoreML lets you pick the compute units (MLComputeUnits.cpuAndGPU). And that was worse. Instead of an error you can handle, a fatal assertion fired deep inside MetalPerformanceShadersGraph:

MPSGraphComputePackage ... failed assertion 'cannot open input file ... .mpsgraph'

This is not an error you catch. It’s an abort(). The whole process goes down. No try/catch, no timeout, no fallback can save you: once the assertion fires, the app is already gone. In other words, my plan B was worse than the original problem, because it turned a recoverable error into a crash in the user’s face.

And here’s a point I want to make clear, to be fair: this is not a WhisperKit bug. It’s CoreML hitting hardware and compiler limits trying to build a large encoder on a phone. WhisperKit is excellent where CoreML fits. This model, on this device, did not fit.

The turning point: switch the runtime, not the model

This is where it clicked. I was stubbornly tweaking the model and the CoreML settings, when what needed to change was the engine.

Whisper isn’t married to CoreML. whisper.cpp has a completely different runtime: GGML with a Metal backend. It doesn’t go through CoreML, doesn’t go through the Neural Engine compiler, doesn’t go through MPSGraph. The two failure modes above simply don’t exist on that path, because the path is different. Same Whisper weights, just a different engine.

flowchart TD
  A["Recorded audio (16kHz mono)"] --> B{"Chosen runtime"}
  B -->|"CoreML / Neural Engine"| C["std::bad_alloc · CoreML error -14"]
  B -->|"CoreML / GPU"| D["fatal assert in MPSGraph · abort()"]
  B -->|"whisper.cpp"| E["GGML + Metal · no CoreML/ANE/MPSGraph"]
  E --> F["transcription in ~1.15s"]

I switched the iOS engine to whisper.cpp on Metal, using large-v3-turbo-q5_0 (the quantized version, ~574 MB). Same iPhone 17 Pro Max (A19 Pro), same test sentence:

[WhisperMetal] audio IN: 16.8s
[WhisperMetal] audio OUT: 168 chars in 1147ms

16.8 seconds of audio transcribed in 1.15 seconds. And no crash. Turbo helps a lot here, because it has only 4 decoder layers, so decoding flies. And since there’s no CoreML in the path, there’s no -14 or MPSGraph to go wrong.

The numbers we saw

All measured on the same iPhone 17 Pro Max (A19 Pro), with whisper.cpp on Metal. Two details surprised me: turbo is “large-v3” but has only 4 decoder layers (against 24 for medium), so it’s faster despite being bigger; and the first run compiles the embedded Metal shaders (~6.2s), but that gets cached and drops to ~0.03s afterwards.

Technical jargon spoken (in Portuguese) in the same sentence, and how each model heard it:

Turbo fixes almost all the jargon on its own. What slips through (brand spelling, like “Supabase”) is exactly the job of vocabulary boosting: injecting the terms as a decode prompt and doing an exact replacement in post-processing.

What I found ready (and what I didn’t)

Solving the engine was half the problem. The other half was discovering that, in 2026, there is no maintained SPM package that delivers whisper.cpp’s GGML/Metal runtime behind a decent Swift API. I checked carefully before complaining, and here’s the landscape:

• whisper.cpp itself now publishes an official Metal-enabled xcframework on its releases. Great, but it’s raw C: you import it and call the functions by hand.
• The classic SPM wrappers (whisper.spm, SwiftWhisper) have been stale since 2024 and ship without Metal (CPU only). whisper.spm’s Package.swift has, for two years, a TODO admitting they couldn’t build the Metal shaders in SPM.
• WhisperKit gives you the nice Swift API, but it’s CoreML, i.e. exactly the path that had failed.

So what was missing wasn’t the Metal binary (upstream already provides that). What was missing was the Swift developer-experience layer on top of that runtime. I packaged it and open-sourced it: WhisperMetalKit (Apache 2.0). It’s the Metal whisper.xcframework wrapped as a binaryTarget, plus an async WhisperModel (actor), an AVFoundation audio resampler and a model downloader. One-line install:

.package(url: "https://github.com/carloshpdoc/WhisperMetalKit.git", from: "0.1.0")

let model = try await WhisperModel(modelPath: url)            // ggml / gguf
let samples = try WhisperAudio.samples(fromFile: recording)  // any file -> 16kHz mono
let result = try await model.transcribe(samples: samples, options: .init(language: "pt"))
print(result.text)

In Voxfloy, the iOS engine starts on the Apple engine (which loads instantly) and switches to Whisper as soon as the model is ready, without the rest of the app needing to know which one is running. The user just notices the transcription got better.

flowchart LR
  A["Recording"] --> B{"Whisper model ready?"}
  B -->|"no (at boot)"| C["Apple SpeechAnalyzer · instant"]
  B -->|"yes"| D["whisper.cpp + Metal · large-v3-turbo-q5"]
  C --> E["Text"]
  D --> E["Text"]

What I took away

The bigger lesson here isn’t even about Whisper. It’s about separating the model from the runtime in your head.

When something “doesn’t run”, the instinct is to mess with the model or switch devices. But with on-device ML there’s a third, almost invisible variable: the execution path. CoreML, GGML/Metal and whatever else comes along are different ways of running the same weights, and each has very different compiler and memory limits. Switching the path was cheaper and more robust than fighting the path that was failing.

The second lesson is more mundane: it’s worth checking the real state of the ecosystem before assuming something exists. I almost assumed “of course there’s a well-maintained whisper.cpp Metal via SPM”. There wasn’t. What existed was stale or raw C. The gap was real, just narrower than it looked, and describing that honestly is worth more than selling a miracle.

To be honest (the limits)

• I didn’t invent the Metal binary. whisper.cpp already publishes the Metal xcframework. My part is the Swift layer on top and the context for why to choose GGML/Metal over CoreML on iPhone. If you want, you can point the binaryTarget straight at whisper.cpp’s official release.
• The benchmark is one device and one audio clip. iPhone 17 Pro Max, A19 Pro, one sentence. On older devices large-v3-turbo will weigh more, and the right answer is probably to drop to medium or small.
• WhisperKit is still a great choice where CoreML fits. Smaller models compile fine on the ANE and get acceleration the GGML path doesn’t have. The decision isn’t “one is better than the other”, it’s “which runtime can handle your model on your device”.
• The model is downloaded at runtime, it doesn’t ship inside the app. It’s ~574 MB downloaded once and cached (and I keep it out of the iCloud backup). Embedding it in the bundle would blow up the app size.

Conclusion

Whisper wouldn’t run on the iPhone because CoreML couldn’t compile that large encoder on that hardware, and forcing the GPU only traded a recoverable error for a crash. The fix was to stop fighting CoreML and switch the runtime to whisper.cpp’s GGML/Metal, which ignores that whole path and runs the same model in about a second. And the useful residue of this for the community became a package: the Swift API that was missing on top of a runtime that already existed.

CTA

If you ship on-device transcription in an Apple app, take a look at WhisperMetalKit and tell me if it saved you a day. And if you’ve already been burned by CoreML’s -14 with Whisper, reply and tell me which model and which device. I want to map where the CoreML frontier really is today.

References

• WhisperMetalKit on GitHub
• whisper.cpp (releases with the official Metal xcframework)
• WhisperKit / Argmax OSS
• Apple Developer: Core ML compute units