#Native Apps

#One-line summary

Three native client apps (macOS, iOS, Android) act as thin node clients that expose device capabilities (camera, microphone, screen, location, contacts, etc.) to the Gateway over WebSocket, while providing a local chat UI and voice interaction.

#Responsibilities

• Connect to the Gateway over WebSocket with Ed25519 device authentication and TLS certificate pinning
• Advertise device capabilities (camera, screen, location, voice, contacts, calendar, etc.) to the Gateway
• Execute Gateway-invoked commands (camera.snap, location.get, screen.record, canvas.navigate, etc.)
• Provide local chat interface with streaming responses and markdown rendering
• Support voice interaction: wake word detection and push-to-talk
• Display the A2UI canvas via embedded WebView with JavaScript bridge

#Architecture diagram

#Key source files

#Shared (Apple platforms)

#macOS

#iOS

#Android

#WebSocket protocol (shared across all platforms)

#Protocol version

All apps use protocol v3 (min=3, max=3).

#Connect handshake

1. Client opens WebSocket to ws://host:port or wss://host:port
2. Server sends { type: "event", event: "connect.challenge", payload: { nonce } }
3. Client constructs signature payload:
   "v2|deviceId|clientId|clientMode|role|scopes|signedAtMs|token|nonce"
4. Client signs with Ed25519 private key
5. Client sends "connect" RPC with:
   {
     proto: { min: 3, max: 3 },
     client: { id, displayName, version, platform, mode },
     caps: ["canvas", "camera", "screen", "voiceWake", "location", ...],
     commands: [...list of supported commands...],
     permissions: { camera, microphone, location, ... },
     role: "node",
     auth: { device: { deviceId, publicKey, signedAtMs, signature, token } }
   }
6. Server responds with:
   {
     ok: true,
     server: { host },
     auth: { deviceToken },
     canvasHostUrl,
     snapshot: { sessionDefaults: { mainSessionKey } }
   }

#Node invoke flow

Gateway → Client:
  { type: "event", event: "node.invoke.request",
    payload: { invokeId, command, params } }

Client executes command (e.g., camera.snap, location.get)

Client → Gateway:
  { type: "req", method: "node.invoke.result",
    params: { invokeId, result: {...} } }
  OR
  { type: "req", method: "node.invoke.result",
    params: { invokeId, error: { code, message } } }

#Device capabilities by platform

#Node commands by platform

#Platform differences

#TLS and security

Certificate pinning:
  - SHA256 fingerprint of server TLS certificate
  - Trust-on-first-use (user must approve fingerprint once)
  - Stored per-gateway in device auth store

Loopback exception:
  - No TLS required for localhost/127.0.0.1 connections

Tailscale domains:
  - .ts.net domains enforce TLS
  - Wide-area DNS-SD discovery supported

Device identity:
  - Ed25519 key pair generated on first launch
  - Private key stored in:
    - iOS: Keychain
    - Android: EncryptedSharedPreferences (BouncyCastle)
    - macOS: Keychain
  - Public key sent during connect handshake
  - Device token returned by server, stored for subsequent connects

#A2UI Canvas integration

All three platforms embed a WebView for the A2UI canvas:

Gateway → Node: "canvas.a2ui.push" { messages: [...] }
Node → WebView: JavaScript injection
WebView → Node: window.openclawCanvasA2UIAction.postMessage(json)
Node → Gateway: "node.invoke.result" { ... }

Canvas lifecycle:
  - canvas.present → show WebView
  - canvas.navigate → load URL
  - canvas.a2ui.push → inject A2UI messages
  - canvas.snapshot → capture screenshot (base64)
  - canvas.hide → hide WebView

#How it connects to other modules

• Depends on:

  • gateway/ — WebSocket server handles all communication, serves canvas host, manages device auth
  • apps/shared/OpenClawKit/ — shared Swift package for iOS + macOS (protocol, session, identity, commands)
  • apps/shared/OpenClawChatUI/ — shared chat UI components for iOS + macOS

• Depended by:

  • None — native apps are leaf clients

#My blind spots

• Exact SwabbleKit wake word engine internals — third-party dependency, not OpenClaw source
• How PeekabooBridgeHostCoordinator on macOS handles UI automation — seems like a significant capability
• APK auto-update mechanism on Android (AppUpdateHandler) — how does it discover and apply updates?
• Watch app features — WatchConnectivity is referenced but the actual watchOS app was not found in the source tree
• Whether Android supports the full set of iOS commands (contacts, calendar, reminders, photos) or if these are iOS-only
• How screen recording results are encoded and transmitted — size constraints for WebSocket frames
• Voice wake accuracy and battery impact across platforms

#Change frequency

• iOS: High — most feature-rich platform, new commands and capabilities added regularly
• Android: Medium — follows iOS features with some platform-specific additions (SMS, foreground service)
• macOS: Medium — menu bar app with focused feature set, Peekaboo integration evolving
• Shared (OpenClawKit): Medium — protocol changes propagate here first, then to platform code
• Protocol (v3): Low — wire protocol is stable; capability additions don't require protocol changes