Initializing…

MobileActions: Edge AI

Fine-Tuning SLMs for Multi-Step Tool Calling on iOS

→ or click to advance
1 / 14

The Goal: Local Execution

  • Natural Language to Native Code: Translating user intent into executable JSON schemas.
  • Multi-Step Reasoning: Handling complex workflows — categorize, update context, create reminder.
  • Real Native APIs: Bridging AI outputs directly to Swift's EventKit, Calendar, and OS tools.
100% on-device — no cloud
2 / 14

The Elephant in the Room: Siri & Gemini

☁

The Status Quo (Cloud)

The latest versions of iOS utilize Apple Intelligence and Gemini integration to handle complex tasks, but they heavily rely on cloud routing for heavy lifting. This requires an internet connection, introduces latency, and pushes data off-device.

⚙

The MobileActions Mandate (Edge)

100% Offline. No API keys. No cloud servers. No internet required. The goal: force complex, multi-step LLM reasoning entirely onto the iPhone's local A-series chip, guaranteeing zero latency and absolute user privacy.

3 / 14

Engine: FunctionGemma-270m

Small Language Model (SLM)

Unlike massive LLMs (100B+ parameters) requiring data centers, SLMs operate under 3B parameters, small enough to run in mobile RAM.

FunctionGemma (270 million)

Incredibly lightweight, pre-trained specifically by Google to understand JSON schemas and execute tool calls, rather than engaging in open chat.

4 / 14

The Bridge: LiteRT-LM

⚡

Inference Engine

Google's LiteRT-LM manages the actual lifecycle of the model on-device, wrapped in a custom Swift Actor to ensure thread-safe execution.

⑂

Interruption Loop

When the AI outputs JSON, LiteRT pauses inference, hands the data to our Swift ToolRegistry to run real iOS code, and feeds the results back in.

5 / 14

SHOW: The Baseline Failure

Testing in the Browser

LiteRT is cross-platform. Using WebAssembly and WebGPU, we test our raw FunctionGemma model directly in a browser before moving to Xcode.

  • The Test: "Remind me to buy milk."
  • The Problem: The base model skips steps, hallucinates list IDs, or outputs raw conversational text that breaks our Swift parser.
$ remind me to buy milk // Expected: structured tool call // Actual: { "tool": "I can help with that! Let me create a reminder for you." "params": { "text": "buy milk" "list": "genera // Parser crash: unexpected conversational token }
6 / 14

LEARN: The Need to Fine-Tune

Generic vs. Specific

Base models are "jacks of all trades." We need a model trained specifically on our iOS application's custom JSON tool schemas.

Multi-Turn Logic

The model must learn to stop talking, wait for the CategorizeTool to return the user's available lists, and only then fire the ReminderTool.

Zero Hallucinations

Swift expects strict JSON decoding. If the model adds conversational filler like "I can help with that: {json}", the app crashes.

7 / 14

LEARN: PEFT & LoRA

How to train locally?

Retraining 270 million parameters requires massive server farms. Instead, we use Parameter-Efficient Fine-Tuning (PEFT).

Using LoRA (Low-Rank Adaptation), we freeze the base model and only train a tiny, lightweight "adapter" layer.

(The same math used to teach Stable Diffusion specific art styles!)

IN
W
A
W
A
W
OUT
↕
frozen
adapter
frozen
adapter
frozen
Base model frozen — only adapter matrices trained
8 / 14

SHOW: Synthetic Data Generation

  • Python Scripts: Custom generators outputting .jsonl files matching the google/mobile-actions format.
  • Single-Step (405 samples): Simple tasks like fetching the weather.
  • Multi-Step (735 samples): Complex categorize → wait for context → create flows to force the model to learn patience and schema adherence.
{ "instruction": "Remind me to buy milk", "output": "[{\"tool\":\"CategorizeTool\",\"params\":{\"text\":\"buy milk\"}}]" } { "instruction": "What's the weather in Tokyo?", "output": "[{\"tool\":\"WeatherTool\",\"params\":{\"location\":\"Tokyo\"}}]" } { "instruction": "Create a shopping list and add eggs", "output": "[{\"tool\":\"CreateList\",\"params\":{\"name\":\"Shopping\"}}]" } { "instruction": "Schedule meeting tomorrow at 2pm", "output": "[{\"tool\":\"CalendarTool\",\"params\":{\"title\":\"Meeting\",\"time\":\"...\"}}]" } { "instruction": "Find Italian restaurants nearby", "output": "[{\"tool\":\"SearchTool\",\"params\":{\"query\":\"Italian restaurants\"}}]" } ... 1,140 total samples
9 / 14

The Pipeline

Executing the LoRA training loop locally on Apple Silicon (Mac GPU). Merging the adapter and exporting to a highly compressed .litertlm file.

1
Synthetic Data
→
2
LoRA Fine-Tune
→
3
Merge Adapter
→
4
Export .litertlm
→
5
Ship to iOS
10 / 14

SHOW: The Fine-Tuned Success

Back to the Browser

We take our newly trained .litertlm file and drop it back into our WebGPU testing environment.

  • The Test: "Remind me to buy milk."
  • The Result: The model flawlessly executes the JSON for CategorizeTool, waits, reads the injected context window, and executes the ReminderTool JSON perfectly.
Before (base model) { "tool": "CategorizeTool", "params": { "text": "Sure! Let me help you with that... Anyway, here's your reminder for milk" } }
After (fine-tuned) { "tool": "CategorizeTool", "params": { "text": "buy milk" } } ← clean, parseable JSON
11 / 14

LEARN: Moving to iOS

  • Drag & Drop: The exact same .litertlm file is bundled into the Xcode project.
  • Whisper Integration: On-device voice-to-text initiates the prompt.
  • Tool Registry: Swift intercepts the flawless JSON and executes native EventKit logic.
  • Result: A fully offline, highly capable iOS AI assistant.
let engine = EngineManager() let registry = ToolRegistry() let conversation = try await engine .createConversation() // Stream the model's response for try await token in conversation .sendMessageStreaming(text) { // Check for tool call in token stream if let call = registry .parseToolCall(token) { let result = try await registry .execute(call) conversation.injectResult(result) } }
12 / 14

Questions & Discussion

Thank you for exploring the mechanics of on-device fine-tuning.

13 / 14

Try It Live

Run the fine-tuned FunctionGemma model directly in your browser via WebGPU.

No cloud. No API key. No setup.

Open the chat panel →

14 / 14

MobileActions Demo