# YouOS Architecture

## Overview

YouOS is a retrieval-augmented generation (RAG) system that drafts email replies in the user's personal style. It combines full-text search, semantic embeddings, persona modeling, an on-device fine-tuned model, and continuous learning.

## Components

### Ingestion (`app/ingestion/`)
- **Gmail threads & Google Docs**: via a configurable Google backend (`ingestion.google_backend`) — `gog` CLI (default), Google's `gws` CLI, or the native Google API (`youos[google]` + OAuth); local JSON imports also supported
- **WhatsApp**: export-file parsing
- **Organic pairs**: replies you sent without YouOS are captured as training pairs too

Each source produces:
- `documents` — the raw source material
- `chunks` — retrieval-sized text segments
- `reply_pairs` — inbound message + user's reply pairs

### Retrieval (`app/retrieval/`)
- **FTS5 / BM25 full-text search** via SQLite for lexical matching
- **Semantic reranking** using MLX embeddings (Qwen2.5 mean-pooled, LRU-cached)
- **Metadata scoring**: recency, source type, account isolation, same-thread boost, subject/topic signals, sender type/domain boosts, quality scores
- **Mode detection**: work vs personal signal classification; multi-intent detection

### Generation (`app/generation/`)
- Assembles prompt from: per-mode persona config, retrieved exemplars, sender context, thread history, and relevant facts
- Drafts on the **fine-tuned local model by default** (Qwen + your LoRA). The cloud
  (Claude) is only a cold-start before your model is trained, or an explicit fallback
  (`review.draft_model` / `model.fallback`)
- Returns a draft with confidence score, precedent references, and the model label that
  actually produced it (surfaced as a per-draft badge and in the Stats "Drafting with" row)

### Warm model server (`app/core/model_server.py`)
- Wraps `mlx_lm.server` (OpenAI-compatible HTTP) so the local model + LoRA adapter load
  **once** and stay warm — fast, on-device drafting and streaming without per-draft cold starts
- Reloads automatically when the adapter changes (`model.server`, default enabled)

### Evaluation (`app/evaluation/`)
- **Golden benchmark**: rule-based scoring (keyword hit rate, brevity, mode match, confidence) over curated cases; cases auto-generated from corpus or loaded from fixtures
- **Voice-match** (`voice_match.py`, `model_compare.py`): scores a draft against the reply you actually wrote (lexical / length / style / semantic), powering `youos compare-models` so you can verify which backend sounds most like you
- **Readiness gate**: drafting is held behind a soft banner until the model is trained *and* benchmarked

### Autoresearch (`app/autoresearch/`)
- Iterative config optimization loop
- Mutates retrieval parameters and prompt variants
- Keeps improvements, reverts regressions
- Runs nightly after ingestion and fine-tuning, on a weekly-rotating benchmark sample (prevents overfitting to fixed cases)

### Autonomous triage (`app/agent/`) — opt-in
Background loop that sweeps unread inbox, filters noise, and queues drafts for review. Never auto-sends.

- **`inbox_fetch.py`** — pulls unread messages from each enabled account via the configured Google backend (gog/gws/native), bounded by `agent.window` and `agent.limit`
- **`needs_reply.py`** — two-tier classifier. Hard-skip rules (per-sender skip list → List-Unsubscribe → mailer-daemon → automation domains → CI subjects → empty body) eliminate noise; soft penalties (noreply, operational mailboxes, very long digests, cold outreach) reduce score without rejecting outright. Returns `NeedsReplyVerdict(needs_reply, score, reasons, cold_outreach, surface_for_review)` — borderline scores (0.30–0.59) surface for review without being drafted
- **`triage.py`** — orchestrates a sweep: fetch → classify → draft survivors (threading `agent.standing_instructions` into the prompt via `extra_constraint`, honoring `agent.strict_local` and `agent.daily_draft_cap`) → persist to `agent_pending_drafts` → log one row to `agent_audit`
- **`scheduler.py`** — `_loop(app)` wakes every `agent.interval_minutes`, re-reads config each iteration so flag changes take effect without restart; macOS notification on new drafts. No-ops under `PYTEST_CURRENT_TEST`
- **`store.py`** — pending-draft CRUD (idempotent on `message_id`), daily-cap counter, and sweep audit log

The `/triage` page shows the pending queue (Save edits, Push to Gmail Drafts, Copy, Mark sent, Dismiss with optional categorical reason, Save as training pair), surface-for-review collapsibles, and Recent activity. `Push to Gmail Drafts` (`app/ingestion/gmail_write.py`) creates a real Gmail Draft on the original thread via the configured backend — all three implemented: `gog` (Phase 2.1, shells out to gog CLI), `gws` (Phase 2.2, shells out to Google's gws CLI), `native` (Phase 2.3, googleapiclient + `gmail.compose` scope). Dismissals carry an optional categorical reason (`noise` / `wrong_sender` / `wrong_content` / `already_handled` / `other`) aggregated by `GET /api/agent/dismissal_stats`. `noise` dismissals feed `agent.skip_senders` (manual via `/api/agent/skip_senders/promote` or automatic via `agent.auto_promote_skip_senders`); `wrong_content` dismissals can be saved as feedback pairs (`POST /api/agent/pending/{id}/save_as_feedback_pair`) that flow into the LoRA training pipeline.

All seven `agent.*` flags (`enabled`, `interval_minutes`, `accounts`, `window`, `limit`, `threshold`, `notify_macos`, `standing_instructions`, `skip_senders`, `daily_draft_cap`, `strict_local`) are toggleable via `/settings` or `youos config set`.

### Web UI (`templates/`)
- Feedback collection with streaming draft generation + cold-start loading overlay
- Review queue for human-in-the-loop training
- Stats dashboard (model status, by-model breakdown, style-drift detection, troubleshooting)
- Settings (`/settings`) for feature flags, About (`/about`), setup wizard (`/welcome`)
- Gmail extension page + bookmarklet fallback

## Data flow

```
Gmail / Docs / WhatsApp ─► Ingestion ─► SQLite DB
   (gog / gws / native)                    │
                                  FTS5 + Embeddings
                                           │
Inbound email ─► Retrieval ─► Generation ─► Draft
                              (local model,    │
                               warm server)    │
                                       User edits draft
                                           │
                                   Feedback pair saved
                                           │
                            Export JSONL ─► LoRA fine-tune
                                           │
                            Autoresearch ─► Config optimization
```

## Database

SQLite with FTS5 virtual tables for full-text search. Schema in `docs/schema.sql`.

Key tables: `documents`, `chunks`, `reply_pairs`, `feedback_pairs`, `benchmark_cases`, `eval_runs`, `sender_profiles`, `ingest_runs`, `facts`, `agent_pending_drafts`, `agent_audit`.