# Jarvis × Ultron Mobile App — Phase 1 Contract

> Source of truth for the mobile build. Two components ship in this repo:
> `gateway/` (Python, Railway) and `mobile/` (Flutter, Android-first).
> Both must follow the WebSocket protocol in §3 exactly.

## 0. Product shape (approved 2026-07-10)

One app, two faces:
- **Jarvis side** (dark, arc-reactor blue `#4FC3F7`-ish family): voice orb home,
  chat, tasks/reminders, memory (notes + graph). No login — personal instance.
- **Ultron side** (existing Ultron dashboard design language): login via existing
  `app_users` (dashboard scrypt auth), WebView of the Railway dashboard in v1.
- Phase 1 = gateway + Flutter voice core only. Chat page, notifications,
  memory/tasks pages, Ultron side, phone-powers come in later phases.

Non-negotiables from Ahmed:
- **Same cloned voice** (Pocket TTS, `voices/jarvis_ref.wav` + mood clips) — served
  from the gateway. Voiceprint files NEVER go through git/GitHub; on Railway they
  live on a volume (`VOICE_DIR`).
- **Tone in, tone out**: phone reads Ahmed's vocal tone (prosody port) and sends it
  with each utterance; brain replies with `[emotion]` tags → mood-clip TTS.
- Persona (butler bearing, "sir", humour, tantrums, swearing) ports verbatim from
  `voice/llm.py` prompts — strip only the Mac/desktop tool clauses.
- Brain = **DeepSeek API** (OpenAI-compatible), swappable adapter (Anthropic later).
- Memory stays the existing cloud service (`MEMORY_API_URL` + `MEMORY_API_KEY`).

## 1. Gateway (`gateway/`) — Python 3.12, FastAPI, deploys to Railway

Mirrors how `memory-service/` deploys (own Railway service, root dir `gateway/`).

### Files
- `main.py` — FastAPI app: `GET /health`, `WS /ws` (everything runs over the WS).
- `brain.py` — DeepSeek chat-completions client with **tool-calling loop**.
  Streaming; regroup token deltas into sentences (reuse the sentence-splitting
  approach from `voice/llm.py`). Model env `MODEL` default `deepseek-chat`.
  Base URL `https://api.deepseek.com`. Adapter interface so the LLM vendor is
  swappable (see `voice/deepseek.py` for the existing client pattern).
- `persona.py` — builds the system prompt: port `_SYSTEM_PROMPT_MAC` from
  `voice/llm.py` minus desktop tool clauses; keep emotion-tag palette, tone-reading
  rules, speech style. At session start fetch and append: memory index/summary,
  preferences (`memory/preferences.md`, `memory/speech-style.md`,
  `memory/humour-level.md` — bundle these files into the gateway image; they are
  small and not secret), and `/values` from the memory service.
- `tools.py` — tool definitions + executors:
  - `memory_search(query)`, `memory_remember(text)`, `memory_forget(...)` →
    existing memory service REST (see `voice/memory_control.py` for endpoints/auth).
  - `task_add(text, due)`, `tasks_list()`, `task_done(id)` → memory service
    `/task*` endpoints.
  - `ultron_leads(query, city, phone_filter)` → Supabase RPC `search_leads`
    via PostgREST (`SUPABASE_REST` + `SUPABASE_SECRET` headers, same as dashboard
    `server.js`).
  - `ultron_team_activity()` → RPC `get_team_activity`;
    `ultron_stats()` → RPC `get_dashboard_stats`;
    `ultron_contact_stats(days)` → RPC `get_contact_stats`.
- `tts.py` — Pocket TTS wrapper: port the load/synth/mood-clip logic from
  `voice/tts_pocket.py`. Voice refs read from `VOICE_DIR` (default `./voices`
  locally; Railway volume in prod). Map `[emotion]` tags → mood stems exactly as
  `_TAG_TO_STEM` does. Output: 16-bit PCM WAV bytes per sentence. Lazy-load the
  model on first voice session; `WS hello` for chat-only mode must not load it.
- `requirements.txt`, `Dockerfile` (python:3.12-slim, uvicorn), `README.md`
  (Railway setup: env vars, volume mount for voices, how to upload refs with
  `railway volume`/scp — never git).

### Env vars
`DEEPSEEK_API_KEY`, `MODEL`, `MEMORY_API_URL`, `MEMORY_API_KEY`,
`SUPABASE_REST`, `SUPABASE_SECRET`, `GATEWAY_KEY` (bearer for clients),
`VOICE_DIR`, `PORT` (Railway sets).
Local dev: read `.env` from repo root (claude-voice/.env has DeepSeek+memory keys;
Supabase keys come from `/Users/ahmed/devFolder/Ultron/.env` — document both).

### Behavior
- Auth: first client frame must be `hello` with correct `auth`; otherwise close.
- On `utterance`: run brain loop. Stream each completed sentence as a `sentence`
  frame immediately; in voice modes also synthesize and stream its audio right
  after (don't wait for the full reply). Tool calls happen mid-loop as needed.
- `barge_in`: cancel in-flight generation + synthesis for the current turn.
- Keep per-connection conversation history in memory (cap ~40 turns). After each
  exchange fire-and-forget a passive memory save mirroring `_auto_remember`
  semantics (POST `/remember` with salience heuristics kept simple for v1).
- Tests (`pytest`, no network): sentence splitter, tool schema validity,
  emotion-tag extraction, protocol frame encode/decode.

## 2. Mobile (`mobile/`) — Flutter (SDK 3.29 compatible), Android first

### Packages (verify latest compatible versions)
`record` (raw PCM stream, audioSource: voiceCommunication → hardware AEC),
`sherpa_onnx` (Silero VAD + streaming on-device STT, English zipformer model),
`flutter_sound` (PCM stream playback for low-latency TTS),
`web_socket_channel`, `shared_preferences`, `permission_handler`.
State mgmt: plain `ChangeNotifier`s — no heavy framework.

### Structure
- `lib/main.dart` — app shell, dark theme, Jarvis blue accent.
- `lib/theme.dart` — two palettes defined now (jarvis blue / ultron) even though
  only Jarvis screens ship in phase 1.
- `lib/screens/home.dart` — the orb: idle/listening/thinking/speaking states
  (animated), live transcript line, mode toggle (Voice / Chat+Voice / Chat),
  text input row for chat modes.
- `lib/screens/settings.dart` — gateway URL + key (persisted), model download
  management, mic test.
- `lib/services/audio_pipeline.dart` — mic PCM (16 kHz mono) → sherpa Silero VAD →
  utterance segmentation (pre-roll ring buffer ~300 ms, end-silence ~600 ms) →
  streaming STT partials + final text.
- `lib/services/prosody.dart` — **port `voice/prosody.py` to Dart**: pace, pitch
  (autocorrelation), loudness, laughter heuristic, EMA baseline; emit the same
  `(voice: ...)` note format.
- `lib/services/gateway.dart` — WS client implementing §3; auto-reconnect.
- `lib/services/player.dart` — plays streamed WAV sentence chunks gaplessly;
  exposes `stop()` for barge-in.
- Barge-in: VAD stays live during playback (AEC handles echo); sustained speech
  while speaking → `player.stop()` + send `barge_in`, then treat as new utterance.
- STT/VAD models: downloaded on first run from the official sherpa-onnx release
  URLs to app documents dir, with progress UI (do NOT bundle in APK).
- Android: `minSdk 26`, permissions RECORD_AUDIO + INTERNET + POST_NOTIFICATIONS,
  foreground service type `microphone` for the active session.
- Must pass `flutter analyze` clean and build: `flutter build apk --debug`.

## 3. WebSocket protocol (`/ws`)

Client → server (JSON text frames):
```json
{"type":"hello","auth":"<GATEWAY_KEY>","mode":"voice|chat_voice|chat","client":"android/1","user":"ahmed"}
{"type":"utterance","text":"...","tone":"(voice: fast, laughing)","source":"voice|typed"}
{"type":"barge_in"}
{"type":"set_mode","mode":"chat"}
{"type":"ping"}
// Phase 2 — phone → gateway (see §3b):
{"type":"action_result","id":"<action id>","ok":true,"data":{...}}
{"type":"action_result","id":"<action id>","ok":false,"error":"..."}
{"type":"event","kind":"new_location","coords":{"lat":26.30,"lng":50.20},"place":"work office"}
{"type":"sms_in","sender":"SNB-AlAhli","body":"OTP 123456","ts":1720600000}
```
`hello.user` is optional (namespaces this phone for routines/proactive pushes;
defaults to the gateway's `MEMORY_GROUP`). On `event`, `place` is OPTIONAL: send
it only for a geofence ENTER that matched a known place (the gateway then fires
that place's routines instead of asking where he is).

Server → client:
```json
{"type":"ready","session":"<id>"}
{"type":"ack"}                                  // optional instant "on it" marker
{"type":"partial","text":"..."}                 // reserved, unused in v1
{"type":"sentence","seq":1,"text":"...","emotion":"warm|dry|excited|sad|calm|null"}
{"type":"audio_start","seq":1}
<binary frame(s): one complete WAV for that sentence>
{"type":"audio_end","seq":1}
{"type":"turn_end"}
{"type":"action","id":"<uuid>","action":"navigate|call|...","params":{...}}   // §3b
{"type":"error","message":"..."}
{"type":"pong"}
```
Rules: `sentence` always precedes its `audio_start`. In `chat` mode no audio
frames are sent. After `barge_in` the server stops emitting frames for that turn
and sends `turn_end`. Binary frames belong to the most recent `audio_start`.
The gateway may **start a turn on its own** (proactive: a new-location prompt, a
watched SMS, or a fired routine) — the client sees the normal `sentence` /
`audio_*` / `turn_end` sequence with no preceding `utterance`; treat it exactly
like any other turn.

### 3b. Device action protocol (phone executes; gateway orchestrates)

When the brain calls a **device tool** the gateway sends an `action` frame and
blocks that tool call until the phone replies with an `action_result` bearing the
**same `id`** (or the gateway times out after ~25 s → the tool reports failure to
the brain). One in-flight matching pair per id; the phone must always answer.

`action` frame: `{"type":"action","id","action","params"}`. The gateway resolves
saved state BEFORE sending (a saved place for `navigate`, a contact's number for
a `call`), so the phone just executes. Actions + params the phone must implement:

| `action` | `params` | Phone does |
|---|---|---|
| `navigate` | `{destination, address?, lat?, lng?, label?}` | launch `google.navigation:` / Maps directions (prefer `lat,lng`, else `address`/`destination`) |
| `play_music` | `{query}` | play via the music app (e.g. YouTube Music search+play intent) |
| `call` | `{name, number}` | place a call to `number` (CALL_PHONE). Gateway already resolved the contact; number is sensitive — don't log it |
| `take_screenshot` | `{}` | capture the screen (MediaProjection / Accessibility) |
| `open_camera` | `{mode:"photo"|"video"|"selfie"}` | open the camera in that mode |
| `open_app` | `{target}` | launch the named app/package |
| `open_url` | `{url}` | open the link |
| `set_timer` | `{seconds, label?}` | start a countdown timer |
| `get_location` | `{}` | one fresh fix; return `data:{lat,lng,accuracy?}` |

`action_result` frame (phone → gateway): `{"type":"action_result","id","ok":bool,`
`"data":{...}|"error":"..."}`. Set `ok:false` + a short `error` on refusal /
permission-denied / failure. `data` is free-form; for `get_location` include
`lat`/`lng`. An optional `data.message` string is spoken back verbatim by the
brain instead of the default confirmation.

**Events the phone emits** (§3 list above): `action_result` (resolves a pending
action), `event`/`new_location` (significant/unknown location, or a geofence
enter with `place`), and `sms_in` (a received text; the gateway forwards it to
the brain only if `sender` matches the SMS watch-list — exact/`*`/substring).
Place-based routines run entirely on the phone's side as geofences: when the
phone enters a saved place's fence it sends `new_location` WITH that `place`
label, and the gateway fires that place's routines.

## 4. Phase 2 — "Jarvis runs my phone" (approved spec 2026-07-10)

The full agentic-phone feature set. Split by where the work lives so the native
parts (which touch `mobile/`) don't collide with the UI redesign in flight.

### 4A. Gateway + memory (no `mobile/` conflict — can build first)
New tools on the brain, backed by the memory service (typed memories/collections):
- **Places**: `save_place(label, address|coords)`, `get_place(label)`, `list_places()`.
  Feeds navigation + the new-location prompt. Labels like "work office", "home".
- **Contacts (phonebook Jarvis remembers)**: `save_contact(name, number, relation?)`,
  `get_contact(name)`. e.g. "call my father" → resolves saved number. Persist in
  memory so he never forgets. Treat numbers as sensitive (don't echo in logs).
- **SMS watch-list**: `sms_watch_add(sender)`, `sms_watch_remove`, `sms_watch_list`.
  Senders like "SNB-AlAhli", "Barq", or "*" (all). The phone enforces the filter;
  gateway just stores the list + decides what to do with a forwarded message.
- **Routines**: `routine_create(name, trigger, action)`, `routine_list`, `routine_cancel`.
  trigger = {type: time|place|both, at?: "07:00", days?, place?: label}; action =
  {type: speak_brief|say_text|run, text?, brief?: "tasks"}. e.g. "each morning give
  me a brief of my tasks" = time 07:00 daily → speak_brief(tasks). The gateway
  generates the brief content (pull tasks/insights) when the routine fires.
- **Routine scheduler**: time-based routines fire from a gateway scheduler (APScheduler)
  → push FCM → app speaks the brief. PLACE-based triggers are on-device geofences
  (Android) that call the gateway to generate content. Hybrid by necessity.
- **New-location trigger**: phone detects a significant/unknown location → tells the
  gateway → Jarvis proactively asks "Sir, where are you now?" → answer becomes a
  `save_place`. Debounce so he doesn't nag on every GPS wobble.

### 4B. Android-native "hands" (touches `mobile/` — AFTER the UI redesign lands)
Platform (MethodChannel) layer, each action gated by an explicit user command +
runtime permission, with a confirm step for sensitive ones:
- **Navigation**: resolve saved place → `google.navigation:` / Maps directions intent.
- **Music**: "play X" → YouTube Music search/play intent.
- **Calls**: `placeCall(number)` for a resolved contact (CALL_PHONE permission).
- **Screenshot**: MediaProjection (per-session consent) or Accessibility `takeScreenshot`.
- **Camera**: open camera app (or capture) intent.
- **App/URL launch**: open any app or deep link.
- **Location**: FusedLocation + geofencing for 4A's place triggers + new-location.
- **SMS listener**: READ_SMS/RECEIVE_SMS → filter by watch-list → forward to gateway
  (Jarvis summarizes / notifies / acts). Sideloaded build, so the Play SMS policy
  doesn't block us.
- **Meeting recording**:
  - PHYSICAL/ambient (record surroundings via mic): fully supported. Start/stop,
    save, optional transcript via the STT path.
  - DIGITAL/call recording: ⚠️ Android 10+ blocks third-party apps from capturing
    call audio (OEM/region dependent, no root). Honest fallback: speakerphone +
    ambient recording, or VoIP/meeting-app audio where the OS allows. Flag this to
    Ahmed; don't promise silent in-call recording.
- **Permission/consent model**: Jarvis acts only on an explicit command ("with my
  permission when I tell him"). One-time OS grants for location/SMS/phone/camera/mic;
  MediaProjection consent per screenshot session. Default-assistant role (long-press
  power) is the summon path.

### 4C. Other later phases
- Chat page polish + FCM notifications (reminders/insights, deep-link into a thread).
- Tasks + memory pages (notes + graph). Ultron side (login + dashboard WebView).
- iOS lite client (Siri Shortcuts / App Intents; no always-listening).

## 5. Deploy (current)
- Gateway lives in the **`balanced-comfort`** Railway project (project id
  `61b79b54-131b-4a10-8a60-3a8466187786`) — the SAME project as `Jarvis`
  (memory-service), `Jarvis-Reflection` (nightly cron), and `falkordb-JarvisMemory`.
  Service name **`Jarvis-Gateway`**, domain
  **`jarvis-gateway-production-9d64.up.railway.app`** (app connects to
  `wss://jarvis-gateway-production-9d64.up.railway.app/ws`).
  (The standalone `jarvis-gateway` project + its `gateway-production-3326e` domain
  was a mistake — deleted; never recreate a separate project.)
- **Builder reality**: `railway up` (CLI) IGNORES the Dockerfile and always uses
  Railpack (mise) — RAILWAY_DOCKERFILE_PATH + railway.json `builder:DOCKERFILE`
  were both ignored. So the deploy MUST work under Railpack: `requirements.txt`
  carries ALL deps incl. `pocket-tts`, with `--extra-index-url
  https://download.pytorch.org/whl/cpu` at the top so torch resolves to the small
  CPU wheel (the plain PyPI CUDA torch blows Railpack's build limits). Start cmd is
  `python -m uvicorn ...` (railway.json) — a bare `uvicorn` isn't on Railpack's PATH.
  Voice clips ship in the build context (`gateway/voices/`, gitignored copy of
  repo-root `voices/`; `VOICE_DIR=/app/voices`). Secrets set as Railway service vars.
- App auto-connects to that `wss://` domain via `--dart-define` at build time
  (`lib/config.dart`); no manual URL entry.
- SEPARATE known issue: the nightly **reflection-service** crashes with
  `unknown url type: ttps` — its `MEMORY_API_URL` env var is typo'd `ttps://…`;
  correct value `https://jarvis-production-3e5f.up.railway.app`. Fix on that service.