Getting a 270M-parameter model to do multi-step function calling, entirely on an iPhone
Siri + Gemini is genuinely good now — but it routes to Apple's servers for anything non-trivial. That's a privacy tradeoff I wasn't willing to make for my own reminders, and it doesn't work in flight mode.
A fully offline pipeline. No API keys, no auth, no server bills. The model, the tool registry, and the system APIs all live on the device. Open the app with no signal — it still works.
Rounded up it'll ship to A-series iPhones and fit in memory next to the OS. Bigger models (Llama, Mistral) need 4–7GB and a beefy GPU — not an option on a phone.
Google pre-trained it on JSON schema/function-calling data. So I didn't have to teach it what a function call is — just how to do my tools, my schemas, my multi-turn pattern.
Google ships LiteRT-LM as an XCFramework. I wrapped it in a Swift Actor so concurrent callers don't race the model. Tokenization, the forward pass, the Metal shader — all handled by the framework.
During streaming, if the model emits a tool call I pause generation, route the JSON to my ToolRegistry (which dispatches to WeatherTool, CalendarTool, etc.), run the Swift code, and inject the result back into the conversation.
LiteRT runs via WebGPU, so I prototyped in the browser first — faster iteration than rebuilding in Xcode every time.
FunctionGemma knows the idea of tool calls, but it has no clue what CategorizeTool or ReminderTool are. Those are local to my app.
The model has to learn: emit a tool call, stop, let Swift inject a result, then resume. Off-the-shelf models just keep babbling.
My Swift JSONDecoder expects one format. If the model prefixes it with "Sure! Here's your JSON:" — the app crashes. Every sample in the training set is raw JSON, nothing else.
You paste 10 examples into the system prompt. The model sees them every inference but its weights never change. It might mimic the format — until you phrase something differently.
Ask "remind me about milk" instead of "remind me to buy milk" and the pattern breaks. The model is pattern-matching, not reasoning.
Every example eats tokens the model could use for reasoning. With 15 tools × 2 examples each, you've burned 1,500+ tokens before the user types a word.
103 training examples run through backpropagation. The model's weights actually change. At inference time the prompt is empty and the behavior is still there.
After 1,140 gradient updates, the model doesn't just copy examples — it extracts the underlying pattern: "user action → tool call → result → next call."
No examples in the prompt. Every token goes to actual reasoning. The fine-tuned behavior is baked into the weights, not pasted into the context.
Full fine-tuning of 270M params needs real GPUs and a training cluster. LoRA freezes the base model and only trains a couple of low-rank adapter matrices — it fit on 8GB of VRAM on an M-series Mac.
Same math Stable Diffusion uses to learn art styles. The adapters are tiny, and at merge time they fold right back into the base weights.
I wrote Python generators that emit .jsonl in Google's mobile-actions format — the model's existing instruction vocabulary, but with my tool names and my schemas.
Train LoRA adapters on the MacBook's Metal GPU, merge them back into FunctionGemma's weights, quantize into a .litertlm bundle, drop it into Xcode.
I swapped the base weights for the merged/fine-tuned .litertlm. Same browser, same WebGPU harness, same prompt.
CategorizeTool call, paused on EOF, consumed the injected result, then fired ReminderTool. Two hops, no filler, no hallucinated fields..litertlm file is added to the Xcode target. No re-exports, no server to stand up.WeatherTool, CalendarTool, EventKit-backed reminders, etc.).Happy to dig into any part of this — the data generation, the LoRA setup, the Swift side, whatever.
The fine-tuned model running in your browser via WebGPU — same weights, no iPhone needed.
Nothing to install. Nothing leaves the tab.
Open the chat panel →