# Reflection engine — Jarvis's proactive brain (Railway cron)

A tiny, dependency-free script (pure Python stdlib). On a nightly schedule it
reads the shared memory graph (over the memory API), asks **DeepSeek V4 Pro** to
reason over the whole thing, and writes back what it finds — insights, entity
summaries, tasks, a refreshed core profile, mined patterns, and cleanup — so any
device pulls it and Jarvis raises it proactively. **No agent, no Claude Code —
just DeepSeek + HTTP.**

```
Railway cron (nightly) ──▶ reflect.py ──GET /memories,/graph──▶ memory API
                                       ──DeepSeek V4 Pro (reason, 11 passes)
                                       ──POST back────────────▶ memory API
Mac HUD / phone ◀── GET /insights,/core,/values,/tasks ── Jarvis surfaces them
```

## The 11 passes (ORDER matters — each later pass sees the earlier cleanup)

The run is bracketed by a **heartbeat** and runs in this exact order:

`heartbeat "started"` → **1–6** (original) → **7** hygiene → **8** pattern →
**9** synthesis → **10** core → **11** archive → `heartbeat "ok"`.

1. **Insights** — up to 6 non-obvious `opportunity` / `observation` / `nudge`
   findings → `POST /insight` (server dedupes near-identical ones).
2. **Entity summaries** — a tight current-state summary onto the top ~10 entities
   (`POST /summary`).
3. **Goals → tasks** — extracts Ahmed's open commitments as tasks, deduped
   against open tasks (`POST /task`).
4. **Consolidation** — retires near-duplicate memories, keeping the newest of each
   group (`POST /retire`).
5. **Theory-of-mind reads** — a psychological "read" onto the top ~8
   people/companies (`POST /read`).
6. **Value model** — refines Ahmed's north-star profile, refining the prior rather
   than wiping it (`POST /values`).
7. **Entity hygiene** — `POST /entities/dedup_backfill` lets the service auto-merge
   the confident (≥0.95) pairs and hand back an *ambiguous* band; **one** batched
   DeepSeek call adjudicates the whole band (capped at 20 pairs/night) and the
   confirmed merges go to `POST /entities/merge`. Mistyped generic `thing`
   entities are **flagged log-only** (there is no retype endpoint) and land in the
   run note.
8. **Pattern mining** — **one** DeepSeek call over the timestamped corpus (record
   date + real-world `refers_to` date) mines recurring routines/cycles, each with
   evidence. Saved as `kind="pattern"` via `/remember`, linked to 2–3 evidence
   ids, **max 5/night**, deduped against the pattern memories already stored
   (fetched and passed in as "already known").
9. **Synthesis** — **one** DeepSeek call infers 3–5 higher-level conclusions that
   *follow from* the memories but aren't directly stated. Saved as
   `kind="synthesis"`, linked to their sources, capped at 5, deduped likewise.
10. **Core block** — a dense, current ~200-word profile of Ahmed (active projects
    + state, key people, priorities, preferences, nothing stale) → `POST /core`.
    Runs **last of the generative passes** so it reflects the night's dedup/merge/
    pattern/synthesis work. Built from the highest-value memories + the entity
    summaries refreshed in pass 2, and refreshes the prior core (dropping stale
    lines) rather than reinventing it.
11. **Archive sweep** — retires memories that are **ALL of**: kind
    `observation`/`event`, importance ≤4 (or absent), access_count 0 (or absent),
    **> 90 days old**, and **not linked** to another memory. Before retiring a
    cluster (≥3 sharing an entity) it first saves one summary fact linking them so
    the gist survives. **Capped at 30 retirements/night.** Nothing recent, undated,
    or linked can ever be retired.

Every generative pass is threaded with Ahmed's stated preferences (`kind=feedback`
/ `kind=preference`) so the whole reflection aligns to what he's told Jarvis to do
(or stop doing).

## Heartbeat — why reflection can no longer die silently

The whole run is wrapped so a crash is **visible**:

- `POST /reflection_ran {status:"started"}` at the very start,
- `{status:"ok", note:<counts>}` at the end,
- `{status:"error", note}` on any fatal/uncaught exception, then it re-raises.

A failing heartbeat can **never** take the run down (it swallows its own errors),
and every pass is individually guarded so one failing pass doesn't abort the rest
— only a genuinely fatal error (e.g. the memory API being down, or the very first
DeepSeek call failing) stamps `error`. The dashboard/VPS memory-watch reads
`/reflection_ran` and alerts if a night is missed or errored.

> **⚠️ The Railway cron schedule lives ONLY in the Railway UI — it is invisible to
> this repo.** There is no cron definition in the codebase, the Dockerfile, or any
> config file here. The `0 2 * * *` schedule is set in the service's Settings on
> Railway. **If the service is deleted and recreated (or migrated to a new
> project), the cron silently does not come back** — the container will exist but
> never fire, and nothing in git will tell you. This is exactly the failure mode
> that once left reflection quietly dead. The mitigations are the
> `/reflection_ran` heartbeat above **plus** the VPS memory-watch alert: if no
> `started`/`ok` stamp lands on schedule, they raise it. After any Railway
> service recreation, **re-add the Cron Schedule by hand and confirm a heartbeat
> lands.**

## Newer memory-service endpoints (graceful 404 fallback)

Some passes call endpoints added by a parallel effort on the memory service:
`/reflection_ran`, `/core`, `/core_block`, `/entities/dedup_backfill`,
`/entities/merge`. These are called through a 404-tolerant helper (`mem_opt`, and
the heartbeat's own try/except) so if the deployed memory service is older and
doesn't have one yet, that call just logs and skips — the rest of the run still
completes. Memories may also now carry `importance` / `access_count` /
`last_accessed` / `valid_from|to` / `refers_to`; all are read defensively (they
may be absent on older nodes, and `/memories` doesn't return the importance/access
fields yet, so ranking/archival degrade safely to recency + link/age checks).

## Deploy on Railway (same project as the memory service)

1. **New service → Deploy from GitHub repo**, root directory
   `claude-voice/reflection-service` (has the Dockerfile).
2. **Variables:**
   - `MEMORY_API_URL` — the memory service's URL (public, or its
     `*.railway.internal` host — same project, so internal works)
   - `MEMORY_API_KEY` — same key the devices use
   - `DEEPSEEK_API_KEY` — your DeepSeek key (permanent V4-Pro pricing)
   - *(optional)* `REFLECT_MODEL` (default `deepseek-v4-pro`), `DEEPSEEK_URL`
     (default `https://api.deepseek.com`), `MEMORY_GROUP` (default `ahmed`)
   - *(optional)* `REFLECT_MAX_MEMORIES` — context cap, **default 800**. The corpus
     is paged (the service caps each page at ~200) newest-first up to this many;
     if it ever over-returns, the trim keeps the highest-value by
     importance → access_count → recency.
   - *(optional)* `REFLECT_DRY=1` — print what every pass *would* do without
     writing anything. **It applies to EVERY pass, including the heartbeat, the
     entity dedup/merge, core, pattern, synthesis, and archive** (dedup_backfill is
     told `dry_run=1` too).
3. **Schedule it as a cron** — in the service's **Settings → Cron Schedule**, set
   `0 2 * * *` (02:00 UTC daily). Railway runs it once on schedule and it exits.
   **Remember: this schedule is not in the repo** (see the warning above).

Run manually / test:

```bash
REFLECT_DRY=1 python reflect.py     # prints every pass's findings, writes nothing
```

## Cost per night

It is no longer a single call. A full night is roughly **15–27 DeepSeek calls**
(1 insights + up to 10 entity summaries + 1 goals + 1 consolidate + up to 8 reads
+ 1 values + up to 2 hygiene + 1 pattern + 1 synthesis + 1 core + up to 1
archive). Rough token budget: **~120k input + up to ~30k output tokens/night** —
on the order of **~10–15¢/run** at V4-Pro pricing, still cheap for a nightly job.
Every DeepSeek call has a bounded input slice and a bounded `max_tokens`, so the
cost can't run away even if the memory graph grows.