# Claude Voice

Real-time, interruptible voice conversation with Claude on this Mac (M4).
Talk naturally; Claude answers out loud; **you can talk over it** and it
shuts up and listens — no push-to-talk, no walkie-talkie turns.

## Two ways to run

- **HUD (recommended):** double-click **`Claude HUD.app`** in this folder
  (or `hud/ClaudeHUD.app`). A Liquid-Glass overlay floats over your
  desktop — status orb + live transcript bottom-right, spinning-agent
  chips top-right, draggable image popups when Claude shows you something.
  A menu-bar icon (waveform) gives Start/Stop, Show/Hide, Mute, Quit, and the
  overlay's control row ends with a **red power button (⏻) that fully quits
  Jarvis** in one tap (stops the engine + all children, then terminates the
  app). The HUD owns the engine: quitting or killing it cleanly stops
  everything (verified: even a force-kill leaves no orphans). First launch, grant
  **Microphone** (and, for the global mute hotkey, **Input Monitoring**)
  to ClaudeHUD.app when macOS asks.
- **Terminal:** `./run.sh` — same engine, text logs instead of glass.
  Rebuild the HUD after changing `hud/ClaudeHUD.swift`: `hud/build.sh`.

**No APIs.** The conversation runs through the locally-installed Claude
Code CLI on the Max subscription (same zero-billing route as the Telegram
agent). Speech-to-text and text-to-speech are local models on the M4;
audio never leaves the machine.

**Jarvis mode (default).** Claude runs as **Sonnet 5** with Claude Code's
real tools — shell, file read/edit, web search — and permissions bypassed
(Ahmed's explicit choice, mirroring the Telegram agent). "Check if the
scraper is running", "fix that bug in enrich-web" actually do it; default
working dir `~/devFolder` (`WORKDIR` to change). `TOOLS=0` for a
chat-only session.

**Delegation — the pre-warmed agent pool (`voice/agent_pool.py`).** The
talking Claude never grinds on tasks itself: real work goes to a
*background agent* via `mcp__agents__dispatch` — `worker` (Sonnet 5,
default), `heavy` (Opus) for complex jobs, `genius` (Fable 5) when asked
or truly hard — while the conversation stays free for the next request.

Agents are **separate Claude sessions owned by the engine**, not in-session
Task-tool subagents, which is what makes them reliable:
- **Pre-warmed (N+1).** `WARM_SPARES` (default 1) fully-connected worker
  sessions idle in the pool; dispatch hands one the prompt and it is
  *already working* (measured: **0.00 s**, versus 10-20 s to cold-spawn a
  subagent mid-turn). The moment a spare is taken, a replacement starts
  warming — there is always one ready. `heavy`/`genius` cold-start (~3 s).
- **Supervised.** Every running agent has a dedicated reader consuming its
  stream the whole time — so a finished/dead/stuck agent is noticed *while
  the room is quiet*, which the old in-session path structurally could not
  do (its stream was only read during a brain turn). Each tool call emits
  `task_progress` → the HUD chip shows **what it's doing right now**; a
  no-activity watchdog (`WORKER_STALL_S`, default 300) reports a stall out
  loud; a crash reports the death out loud instead of silence.
- **Killable.** `agent_kill` (voice) or the ✕ on a HUD chip disconnects the
  session immediately and clears the chip in the same breath — no zombie UI.
- **Bounded.** `AGENT_MAX` (default 4) running at once; a 5th is refused.

Agents never speak to Ahmed: the pool captures each one's final message and
delivers it to the **master**, who relays it in his own words —
**proactively interjecting during silence** ("Quick interruption — the email
is sent. Back to what you were saying…"), or folding it into the reply when
a new request arrives at the same moment. Announcements are barge-in-able
like any other speech. `AGENT_POOL=0` restores the old Task-tool path
(subagents then report by appending to `control/report.txt`).

**Voice lock.** The assistant only answers its owner. First run: your
first 5 utterances (≥2 s each) enroll your voice (TitaNet speaker
embeddings via sherpa-onnx, 14 ms/check); afterwards other voices are
ignored, with the score printed. It **keeps learning your voice**:
confident matches (≥0.65) are folded into the profile with a small
anchored-EMA step, so recognition improves across rooms and moods while
one absorbed mistake can't corrupt the profile. `VOICE_LOCK=0` disables,
`SPEAKER_RESET=1` re-enrolls, `SPEAKER_THRESHOLD` tunes strictness
(default 0.50; raise toward 0.6 after a few days of adaptation).

**Tone sensing (he hears HOW you said it).** Every utterance is read for
*delivery* — speaking pace, pitch range, loudness and laughter — off the same
audio buffer the voice-lock already uses, in ~2 ms of pure numpy (no model, no
GPU; `voice/prosody.py`). When your delivery deviates from your own rolling
baseline it rides into both brains as a short note glued to the turn, e.g.
`(voice: fast, urgent)` / `(voice: slow, flat)` / `(voice: laughing)`. It does
**not** classify emotion — it reports the delivery and lets Claude infer intent
(flat + over-positive words → dry/sarcastic, and it answers accordingly) the way
a person does. Rate and pitch-swing are the reliable signals (the AEC mic's AGC
flattens absolute volume); sarcasm/joking are *inferred by the brain*, not
measured. Neutral turns inject nothing. The brain also answers *in* tone via an
`[emotion]` tag palette (mirror your sarcasm, warm up when you're low), rendered
on the Mac by the committed `voices/jarvis_<mood>.wav` clips. `PROSODY=0`
disables sensing.

**Mute hotkey.** `Ctrl+Alt+M` toggles the mic globally (needs Input
Monitoring permission for your terminal; `MUTE_KEY` env to rebind), or
type `m`+Enter in the app terminal. Muted = mic feeds silence.

**Self-surgery.** Claude can modify itself mid-conversation via the
`control/` command channel it drives with its own file tools:
- `control/config.json` — instant hot-reload, no restart: `{"voice":
  "am_adam", "speed": 1.1}` applies within a second ("change your voice").
- `control/say.txt` — text written here is spoken immediately; background
  workers use it to announce results the moment they finish.
- `control/restart` — validated self-restart for real code changes: a
  subagent edits the source, then the app **compile-checks the new code
  first** (broken edits are refused with the error, old code keeps
  running), speaks a heads-up, re-execs itself, and **resumes the same
  conversation** (session id persisted in `control/session_id`; `RESUME=0`
  or deleting that file starts fresh). Verified end-to-end: hot-reload,
  spoken notify, refused restart on sabotaged code, clean restart with
  memory intact ("PISTACHIO" recall test).

**Full Mac & browser control (Gmail).** Beyond shell/files, Jarvis has two
MCP servers wired into his session: a **real Chrome** (Playwright MCP) and
**native macOS app control** (macos-automator-mcp, AppleScript/JXA). So "open
Gmail, switch to the alrugaib account, search for Elie and reply to him"
actually works — he drives a logged-in browser, not a blind `osascript`.

- **How the browser avoids Google's login block:** Google rejects logins from
  automation-launched browsers. So Jarvis never logs in — he *attaches* over
  CDP to a **dedicated Chrome you launched yourself**, where you're already
  signed in. Launch it once with `./chrome-jarvis.sh` (a separate
  `~/.jarvis-chrome` profile, because Chrome 136+ blocks remote-debugging on
  your default profile). **First run: sign into both Google accounts (personal
  + alrugaib) in that window by hand.** They persist forever; Jarvis reattaches
  every session. Jarvis will run `chrome-jarvis.sh` himself before the first
  browser action, so normally you just talk.
- **Native apps:** the `mac` server runs AppleScript/JXA with a 200-script
  knowledge base — Mail, Messages, Notes, Calendar, Finder, launching/switching
  apps, UI scripting. First use prompts for Automation/Accessibility permission.
- **Toggles:** `MCP_BROWSER=0` / `MCP_MAC=0` disable either server;
  `JARVIS_CHROME_CDP` / `JARVIS_CHROME_PORT` / `JARVIS_CHROME_PROFILE` override
  the Chrome endpoint/port/profile. Both servers are pinned npm packages,
  pre-install once: `npm i -g @playwright/mcp @steipete/macos-automator-mcp`.

## Run

```bash
./run.sh
```

First run downloads models (~2.9 GB total, cached afterwards). Startup
after that is ~6 s. Then just talk.

| Env var | Default | Options |
|---|---|---|
| `MODEL` | `sonnet` (= Sonnet 5) | `haiku`, `opus` |
| `TOOLS` | `1` (Jarvis mode) | `0` = conversation only, no tools |
| `WORKDIR` | `~/devFolder` | Claude's working directory for tool use |
| `VOICE` | `af_heart` | any Kokoro voice (`af_sarah`, `am_adam`, `bf_emma`, …) |
| `VOICE_SPEED` | `1.0` | e.g. `1.2` |
| `VOICE_ENGINE` | `auto` (Mac→pocket, else chatterbox) | `pocket`, `chatterbox`, `kokoro` |
| `POCKET_VOICE` | `george` | catalog voice while cloning weights are gated |
| `DUPLEX` | `1` | `0` = classic barge-in (no backchannel classification) |
| `LOCAL_MODEL` | best installed (qwen3:4b-instruct…) | any Ollama tag for the fast brain |
| `STT` | `parakeet` | `whisper` (slower, but supports Arabic) |
| `STT_LANG` | auto | e.g. `ar` (whisper only) |
| `INTERRUPT` | `1` | `0` = half-duplex (mic ignored while Claude speaks) |
| `AEC` | `1` | `0` = skip the echo-cancelled mic, use the raw mic |
| `AEC_GAIN` | `6.0` | boost for the quiet VoiceProcessingIO capture |
| `AEC_BYPASS` | — | `1` = helper runs without cancellation (debug) |
| `VOICE_LOCK` | `1` | `0` = answer anyone (no speaker verification) |
| `PROSODY` | `1` | `0` = don't read vocal tone (pace/pitch/laughter) |
| `SPEAKER_THRESHOLD` | `0.50` | voice-match strictness (raise to ~0.6 later) |
| `SPEAKER_RESET` | — | `1` = forget the voice profile, re-enroll |
| `MUTE_KEY` | `<ctrl>+<alt>+m` | global mute hotkey binding |

## Work integrations — Gmail (multi-account) + Microsoft Teams

Both are off until configured, and neither breaks startup when absent.

**Gmail (`voice/gmail_control.py`) — work + business mailboxes.** Reuses the
same Google setup as Calendar: one desktop OAuth client (`google_oauth.json`),
one token *per account* (`google_gmail_<label>.json`). One-time:
1. In the same Google Cloud project as Calendar: **enable the Gmail API**, and
   on the OAuth consent screen **add scopes** `gmail.modify` + `gmail.send`
   (add yourself as a test user if the app is unverified).
2. Set `GMAIL_ACCOUNTS=work,business` (default) and optionally
   `GMAIL_DEFAULT=work`. The first action on each label opens a browser once —
   **sign in with that account** (work = alrugaib, business = the other) and it
   caches + refreshes forever. Jarvis routes by "work" / "business" in speech.

**Teams (`voice/teams_control.py`) — as you, via Graph.** Needs a tenant admin
(you) to register one app. One-time in **Entra ID → App registrations → New**:
1. Note the **Application (client) ID** and **Directory (tenant) ID**.
2. **Authentication → Allow public client flows = Yes** (enables device-code).
3. **API permissions → Microsoft Graph → Delegated:** `Chat.ReadWrite`,
   `ChatMessage.Send`, `User.ReadBasic.All`, `User.Read` → **Grant admin
   consent**.
4. Put `TEAMS_CLIENT_ID` (and `TEAMS_TENANT_ID`, or leave `organizations`) in
   `.env`. First Teams action prints a short code + `microsoft.com/devicelogin`
   URL; enter it once. Token caches to `teams_token_cache.json`.

**Asana (`voice/asana_control.py`) — projects, tasks, due dates, assignees.**
Ahmed's *project management* (distinct from `mcp__tasks__*`, the personal to-do
list). One-time: create a **Personal Access Token** at
[app.asana.com/0/my-apps](https://app.asana.com/0/my-apps) → *Personal access
tokens* → *Create new token*, then add it to `.env`:

```
ASANA_PAT=          # Asana personal access token (empty = Asana tools off)
ASANA_WORKSPACE=    # optional: workspace gid or name, if the account has several
```

No OAuth, no extra deps (stdlib `urllib`).

**One implementation, two doors.** All the REST logic lives in
`jarvis-tools-service/asana.py`; `voice/asana_control.py` is a thin wrapper that
imports that same module and re-exposes it as in-process SDK-MCP tools, and the
hosted connector (`jarvis-tools-service/mcp_server.py`) wraps the identical
functions. So voice Jarvis and the claude.ai connector have **identical Asana
capability** — one bug fix fixes both. Never re-implement a call in the wrapper.

19 tools: `asana_workspaces`, `asana_projects`, `asana_project_create`,
`asana_project_update` (incl. **archive**), `asana_project_members` (invite people
to a project), `asana_project_status`, `asana_task` (full detail),
`asana_tasks`, `asana_task_create`, `asana_task_update`, `asana_task_move`
(a real move: addProject **+** removeProject), `asana_task_complete`,
`asana_task_delete` (recoverable 30 days), `asana_task_people`
(assignee/followers/tags), `asana_subtasks`, `asana_comment`, `asana_comments`
(read the thread), `asana_search`, and `asana_api` — the escape hatch for any
other Asana REST endpoint (goals, portfolios, custom fields, webhooks, project
delete, …).

Notes from the live API: **archive ≠ delete** (project delete is deliberately not
a tool — it's unrecoverable; archiving is reversible). Task **search is a paid
feature** (402 on a free workspace) → the tool falls back to typeahead, which
matches names only, and Asana's index **lags 15-45 s**, so a just-created task is
found via `asana_task`/`asana_tasks`, not `asana_search`. **Start dates
(`start_on`) are paid-only** too. Project create needs a `team` only when the
workspace is an *organization*.

Toggles: `MCP_GMAIL=0`, `MCP_TEAMS=0`, `MCP_ASANA=0` (also off automatically when
`ASANA_PAT` is empty).

## Stack (researched July 2026, all latencies measured on this M4)

| Stage | Engine | Latency |
|---|---|---|
| VAD / turn-taking | Silero VAD v5 (onnx) + Smart Turn v3 semantic end-of-turn | <1 ms/frame, 12 ms/check |
| Speech→text | **parakeet-mlx** (Parakeet TDT 0.6b v3, MLX) | 4 s utterance → 0.5 s |
| Fast brain (chat) | **Qwen3 4B instruct** via Ollama (voice/local_brain.py) | first word ~0.3–0.5 s |
| Full brain (actions/code) | **Claude Agent SDK** → Claude Code → Max sub | first sentence ~1.5–2 s (instant spoken ack masks it) |
| Text→speech (Mac) | **Pocket TTS 100M** (CPU, 2 cores, cloned + streaming; Kokoro fallback) | first audio ~40 ms |
| Text→speech (Windows) | **Chatterbox** on CUDA (Kokoro fallback) | ~0.5 s/sentence |

Reply audio starts ~2 s after you stop talking. Claude streams
sentence-by-sentence: the first sentence plays while the rest is still
being generated and synthesized. One persistent Claude session per run,
so it remembers the whole conversation.

## Turn-taking / barge-in design

- **Listening:** speech starts at VAD prob ≥ 0.60; utterance ends after
  700 ms of silence; 1 s pre-roll ring buffer so the first syllable isn't
  clipped; bursts < 250 ms are ignored as noise.
- **Echo cancellation (the fix for open speakers):** the mic is captured
  through `helper/aecmic`, a tiny Swift binary driving Apple's
  VoiceProcessingIO unit directly via the raw AudioUnit API (bus 1 input,
  bus 0 disabled — the `AVAudioEngine.setVoiceProcessingEnabled` wrapper
  produced bit-exact-zero capture on macOS 26; don't go back to it). Any
  audio this Mac plays — including Claude's own voice — is subtracted from
  the mic before the app sees it. Measured at 100% volume: raw mic hit VAD
  1.00 with 100+ frames over the barge-in threshold during playback; the
  AEC path showed VAD max 0.02 and zero frames while live signal kept
  flowing (`--bypass` mode on the same binary hears the speakers loud,
  proving the capture path itself is alive). VPIO runs quiet (~0.02 peak),
  so `AEC_GAIN` (default 6) boosts frames and utterances are normalized
  before transcription.
- **While Claude speaks:** with AEC, barge-in triggers at VAD ≥ 0.75
  sustained ~260 ms (anything the mic hears is really you). If the AEC
  helper is missing or dead the app falls back to the raw mic with the
  strict 0.95 / ~480 ms settings — usable with headphones, unreliable on
  speakers. On barge-in: playback cut in <50 ms, queued sentences dropped,
  generation interrupted, your speech becomes the next turn.
- **DUPLEX listening (the human bit):** while Jarvis speaks, a short burst
  from you is *transcribed and classified* instead of blindly cutting him
  off — "yeah / mm-hm / go on" are backchannels (he keeps talking), a stop
  phrase ("okay stop", "shut up") just silences him, and anything else
  interrupts AND becomes the next turn immediately — no wake word, no
  re-asking (an 8 s wake-word grace follows every barge-in). Sustained
  speech (>1.5 s) always cuts him instantly. Needs the AEC mic or
  `HEADPHONES=1` (a leaky raw mic would transcribe his own voice);
  `DUPLEX=0` reverts to plain barge-in.
- **Post-interrupt drain:** after a barge-in the aborted turn's leftover
  stream is drained from the Claude session — without this every answer
  lags one question behind.
- **Anti-ducking:** macOS lowers all other audio while a voice-processing
  mic is open (FaceTime behavior) — that silently muffled the TTS. The
  helper sets the ducking config to minimum; measured playback is ~5x
  louder than the ducked state.
- If barge-in still misfires in a noisy room: `INTERRUPT=0`.

## Troubleshooting

- **It never hears you / transcript empty:** macOS mic permission is
  per-terminal-app and fails as *silent zeros*, not an error. System
  Settings → Privacy & Security → Microphone → enable your terminal.
  The app warns at startup if the mic delivers pure silence.
- **It interrupts itself on speakers:** shouldn't happen with the AEC
  mic active (check startup says `mic: echo-cancelled`). If it does,
  lower the volume, use AirPods, or run `INTERRUPT=0`.
- **It never hears you only on the AEC path:** run `AEC=0 ./run.sh` and
  report it — that would mean VoiceProcessingIO is gating your voice.
- **Arabic:** `STT=whisper STT_LANG=ar ./run.sh` — transcription
  understands Arabic (Claude too), but the Kokoro voice only speaks
  English. An Arabic voice would need Piper (`ar_JO-kareem`) — see
  upgrade notes.

## Upgrade path (researched, not built)

- **Jarvis's spoken emotion — DONE (2026-07-10).** The 5 mood clips
  `voices/jarvis_<warm|excited|dry|sad|calm>.wav` were generated on the M4 via
  Chatterbox/MPS (`tools/make_mood_clips.py`) and committed to this private repo,
  so **every device — including the Windows box — gets them without regenerating**
  (same policy as `jarvis_ref.wav`). Pocket TTS preloads them at startup and the
  Mac full-brain prompt now emits `[emotion]` tags, so he speaks in tone (dry when
  you're dry, warm when you're low). To re-record with different delivery, rerun
  `VOICE_ENGINE=chatterbox python tools/make_mood_clips.py` and re-commit.
- **Faster end-of-turn (~0.2 s instead of 0.7 s):** pipecat *smart-turn
  v3* semantic turn detector — 8 MB onnx, ~12 ms inference.
- **Live partial transcripts:** parakeet-mlx `transcribe_stream` supports
  streaming partials while you're still talking.
- **Arabic voice out:** Piper `ar_JO-kareem` (62 ms/sentence).

## Files

```
main.py            orchestrator: state machine, barge-in logic
chrome-jarvis.sh   launches the dedicated logged-in Chrome Jarvis drives (CDP)
voice/llm.py       Claude Agent SDK session (streaming, sentence chunks,
                   + browser & native-Mac MCP servers via _mcp_servers())
voice/stt.py       parakeet-mlx (STT=whisper → mlx-whisper for Arabic)
voice/tts.py       Kokoro-82M via kokoro-onnx (models/ holds the onnx)
voice/tts_pocket.py  Pocket TTS (Mac default): cloned, streaming, CPU-only.
                   [emotion] tags swap mood reference clips —
                   voices/jarvis_<warm|excited|dry|sad|calm>.wav (generate
                   them once with Chatterbox on the RTX box)
voice/vad.py       Silero VAD wrapper
voice/speaker_id.py TitaNet voice lock (enroll + anchored-EMA learning)
voice/prosody.py   vocal-tone reader (pace/pitch/loudness/laughter, pure numpy,
                   baseline-relative) → the (voice: …) note both brains see
tools/make_mood_clips.py  one-off: render the 5 jarvis_<mood>.wav clips (Chatterbox)
                   so Pocket TTS can speak [emotion] tags on the Mac
voice/control.py   self-surgery channel (hot-reload / say / show / restart)
voice/events.py    @@EVT JSON stream the HUD tails (EMIT_JSON=1)
hud/ClaudeHUD.swift  Liquid-Glass menu-bar app (owns + visualizes engine)
hud/build.sh       compiles ClaudeHUD.app (SwiftUI/AppKit, ad-hoc signed)
voice/audio_io.py  mic capture (AEC helper or raw) + interruptible playback
helper/aecmic      Swift VoiceProcessingIO echo-cancelled mic -> stdout PCM
test_e2e.py        headless pipeline test (synthetic audio, barge-in)
test_echo.py       acoustic echo test (raw vs AEC mic during playback)
probe_llm.py       Claude latency probe
```