Kwasi — Personal Assistant¶
Kwasi is a personal AI assistant available via Telegram (primary), WhatsApp, and CLI. It handles scheduling, email, notes, tasks, reminders, web search, weather, GitHub, browser automation, and more — all through natural conversation.
This documentation covers how the system actually works: its architecture, data flows, memory model, tools, and deployment setup.
Module layout changed (2026-06 cleanup)
A behavior-preserving decomposition split the three former "god files" into smaller modules. Where prose below says a tool, loop, or handler is "in agent.py / main.py / bot.py", the implementation now lives in:
- Agent tools →
app/agent_tools/<domain>.py(one module per domain;agent.pyis now thin and just wires + re-exports them). Tool implementations are still inapp/tools/. - Background loops →
app/background.py(spawned bymain.py's lifespan). - HTTP routes (
/message,/reflect,/embed-backfill) →app/routers/api.py. - Telegram →
bot.pykeeps the dispatch core;helpers.py/media.py/commands.pyhold the shared utils, media handlers, and slash commands. - Prompt text →
app/prompt_text.py;AgentDeps/build_deps→app/deps.py.
The original modules re-export their public symbols, so the data flows and behavior described here are unchanged. See CLAUDE.md → Recent Changes for the full map.
At a Glance¶
| Concern | Choice |
|---|---|
| Framework | Pydantic AI — agent loop, tool registration, typed structured output |
| LLM | Three model roles: MODEL_NAME (primary), MINI_MODEL_NAME (skill synthesis), REFLECTION_MODEL_NAME (nightly reflection) — supports Gemini, Claude, GPT-4 |
| Primary interface | Telegram (long-polling via python-telegram-bot) |
| Second interface | Telegram Mini App — a page inside the chat: today, memory, approvals, status. Read and correct, never sends |
| Other interfaces | WhatsApp webhook · CLI REPL · browser dashboard |
| Web framework | FastAPI — health check, reflection endpoint, WhatsApp webhook, browser dashboard, Mini App, health/phone ingest, alert webhook |
| Storage | PostgreSQL + pgvector, in production and in tests. The SQLite adapter was deleted in Aug 2026 — one StoragePort, one implementation |
| Email & calendar | Gmail (personal + work) + Outlook via MCP tool wrappers |
| Multimodal | Gemini Vision/STT — images, PDFs, voice messages |
| Voice output | Microsoft Neural TTS via edge-tts — voice-in → voice-out |
| Browser | Playwright (headless Chromium) — JS-rendered pages + form submission |
| GitHub | PyGitHub — PRs, issues, notifications, repo summaries |
| Jira | Jira Cloud REST API — issues, projects, JQL search, create/update tickets |
| Google Drive | Drive API — search, read, list recent, meeting transcripts (personal + work) |
| Notion (Spec 015) | Two-way capture into a connected Notion "Notes" database — save, search, list, archive. Notion is the source of truth; no local mirror |
| Microsoft To Do sync (Spec 013) | The tasks table mirrors bidirectionally to the To Do default list — inline push on change, reconcile loop for pulls. To Do wins on conflict; a failed fetch never deletes |
| Calendar writes (Spec 014) | Create / update / delete across Outlook and both Google accounts, targeted by spoken calendar name (work, family, …) resolved against a cached writable-calendar map |
| Slack | Slack SDK — read channels, unreads, post messages, search history |
| Task delegation | Three E2B-sandbox backends — delegate_to_coding_agent (OpenCode default / Claude Code) + a dedicated delegate_web_task (browser-use + Gemini — DOM-driven scraping & multi-step form-fills); approval-gated, daily/wall-clock capped, async result delivery |
| Credential vault | Fernet-encrypted website logins (app/vault.py, VAULT_MASTER_KEY) — delegate_web_task(credential=...) logs into real accounts without the LLM ever seeing the plaintext |
| Meeting follow-up | Polls work Gmail for Gemini meeting notes, extracts Lawrence-owned action items, and sends a Telegram card to save as intentions/note |
| Intent routing | Keyword classifier → 13-domain agent dispatch (email, calendar, memory, github, news, slack, jira, drive, meetings, diagnostics, health, database, utility); context-aware follow-up inheritance; semantic fallback with three confidence bands (≥0.65 single domain; 0.45–0.65 top-K composed agent; <0.45 full agent) |
| Self-management | Kwasi changes its own config and deployment from Telegram — set_runtime_config (instant, no restart), railway_set_env / railway_redeploy (approval-gated), diagnose_self health snapshot. Allowlist derived from a single CONFIG_REGISTRY; secrets blocked; proactive ✅/⚠️ after a self-redeploy. Requires the RAILWAY_* vars. |
| Observability | Logfire (infra), Langfuse (LLM traces + managed prompts + scores), and optional W&B Weave (WEAVE_ENABLED) — all on one shared OpenTelemetry provider |
| Quality signal | A 👍/👎 reaction on any Kwasi message becomes a Langfuse user_feedback score — covering the ~95% of turns the approval gate never sees. Each sent message records its genre, so "which proactive genres earn any engagement" is one query |
| Self-alerting | POST /observability/alert forwards Langfuse Monitors and Logfire alerts to Telegram — the push counterpart to diagnose_self. HMAC-verified, fails closed, deduplicated |
| Evaluation | eval/ on pydantic-evals — free CI-gated routing eval, real-agent tool-selection sweep with an LLM judge, results in both Logfire & Langfuse Experiments, and a production-trace mining loop that feeds new cases back into the corpus. See Evaluation. |
| Plugin system | File-drop skill registry — drop a .py into app/skills/ — built-ins: Content Curator, Travel Briefing, CV, Deep Research, Meeting Intelligence, Podcast Insight Sync |
| Personality | Explicit directives in system prompt: direct voice, Ghanaian cultural context, banned apology openers, retry-first error handling, situational tone calibration |
| Semantic search | Gemini gemini-embedding-001 (3072 dims) — vector embeddings stored as halfvec(3072) in Postgres (pgvector, HNSW) for meaning-based search across notes, history, and saved articles |
| Wearable / health data (Spec 010) | Sideloaded Kotlin Android bridge (bridge-android/) reads Google Health Connect every ~15 min and POSTs to POST /health/ingest. Stored as a single normalised health_samples table; queried by a dedicated health_agent |
| Phone Awareness | Ambient attention context — app screen-time from the Android bridge, surfaced via 2 universal read tools, a nightly digest, and the reflection profile. Notification ingest was removed in Aug 2026: 16 days of data carried counts, never content. Sensory-only; no phone actuation. See Phone Awareness |
| Command Centre | /dashboard/?view=command in the browser, and the Mini App's Status tab on a phone — Railway deploy state, Logfire exceptions, loop health, redeploy, console deep links. All three read the same sources as diagnose_self; tokens never leave the server |
| Model resilience | run_with_resilience retries transient 429/5xx against the primary model with backoff, then falls back to the mini model — added after prod logged 86 quota errors and 11 overloads with no retry anywhere |
| Deployment | Railway (Dockerfile) |
Key Design Principles¶
Single AgentDeps instance. All interfaces (Telegram, WhatsApp, CLI) use the same build_deps() factory and share one AgentDeps object per process. Each request gets a shallow per-request copy (via dataclasses.replace) with the correct telegram_chat_id and active_categories — the shared singleton is never mutated.
Fail-safe everything. Storage errors never surface to the user. Tool failures are reported gracefully. The system prompt context fetch is wrapped in try/except so a broken DB never kills the agent.
Storage is one port, one implementation. A StoragePort protocol defines the interface; PostgresAdapter is the only implementation, in production and in tests. A second SQLite adapter existed until Aug 2026 — 2,800 lines expressing the same contract in a different type system, which meant every schema change was written twice and parity bugs lived in the gap. Deleting it made the protocol the seam it was always meant to be. Local dev and tests need a pgvector container.
Two-tier memory pipeline. Facts stated in conversation are extracted within seconds (post-message mini-model pass). A nightly Reflection Engine then distils the full history into a narrative profile, intentions, behavioral learnings, and topic-cluster Summary: <topic> notes. Both tiers feed the same user_facts and notes tables — the agent always has the freshest possible context. Before each agent run, up to three semantic retrieval layers (notes, summaries, saved articles) inject relevant context into the user turn, sharing a 1,000-token budget with XML tagging so the model can distinguish retrieved memory from instructions.
Background loops run inside FastAPI lifespan. All seventeen — reminders, daily briefing, scheduled tasks, alerts, nightly reflection, To Do sync, phone digest, and the rest — are asyncio tasks started on startup and cancelled on shutdown. No separate workers, no queues, and no external scheduler: railway.json declares no cron.
Extensible by file-drop. Adding a new skill requires only creating a file in app/skills/ — no changes to core agent code.
Intent-routed for efficiency. Every message (Telegram and WhatsApp) is classified by a keyword matcher in microseconds. The message is then handled by a focused domain agent carrying only its relevant tools — reducing token usage per call and improving model accuracy. The full agent is always the fallback for ambiguous or cross-domain requests.
Quick Navigation¶
- Curriculum — Pedagogic walkthrough of the whole codebase in 8 sessions (start here if you want to understand the system)
- Architecture — Component diagram and how the pieces fit together
- Message Lifecycle — Sequence diagrams for text, voice, image, and document messages
- Memory & Reflection — How short-term history and long-term context work
- Tools Reference — Every tool available to the agent (native, skills, MCP, GitHub, browser)
- Phone Awareness — Ambient notification + app-usage context: ingest, redaction, classification, and what Kwasi does with it
- Background Loops — Reminders, briefing, scheduled tasks, nightly reflection
- Storage Layer — Schema, adapters, and the storage protocol
- Evaluation — How the agent is evaluated: routing, tool-selection, LLM judge, and the production-trace feedback loop
- Deployment — Railway setup and full environment variable reference