Personal assistant · Pydantic AI · Railway · Telegram-first

The anatomy of Kwasi, at four altitudes.

Every diagram answers one question, uses one shared color language, and was verified against the deployed code — including the paths the existing docs don't draw. Tap any diagram to zoom, pan and pinch. Sections descend from the world outside the container to the internals of each subsystem.

158tools on full agent
13+1domain + briefing agents
46MCP tool wrappers
38approval-gated entries
21postgres tables
17background loops
9managed prompts
ALTITUDE 1 · WORLD

System context

Kwasi is one FastAPI process on Railway. Everything on the left is Lawrence's world pushing in; everything on the right is a service Kwasi reaches out to. Postgres is the only other Railway service.

Who talks to Kwasi, and what does Kwasi talk to?hand-drawn context map
LAWRENCE'S WORLD Lawrenceone allowlisted user Telegram app chat · 👍/👎 reactions · approval keyboards · Mini App (4 tabs) bridge-android sideloaded Kotlin app Health Connect ← watch UsageStatsManager WorkManager, ~15 min Browser dashboard + Command Centre Dev laptop CLI REPL · auth scripts Alert webhooks (push) Langfuse Monitor + Logfire POST /observability/alert HMAC · 15-min dedup Kwasi RAILWAY · FASTAPI · PYTHON 3.13 Telegram bot (primary) + 7 HTTP routers Pydantic AI agent core — 158 tools Approval gate — 38 consequential tools Memory: facts · reflection · embeddings 17 background loops (proactive side) One OTEL provider → 3 backends Vault (Fernet) · config registry Skills ×6 · planning · eval harness Deployed from main · Dockerfile · env is the config surface (requirements.txt, not uv.lock) PostgreSQL + pgvector second Railway service · 21 tables · halfvec(3072) embeddings · HNSW index asyncpg AI MODELS Gemini — primary · mini · reflection · gemini-embedding-001 · Vision/STT Anthropic API — claude_code delegation only GOOGLE (PERSONAL + WORK) Gmail ×2 · Calendar ×2 · Drive ×2 · Maps OAuth refresh tokens via credential broker (invalid_grant auto-heal) MICROSOFT GRAPH Outlook mail + calendar · Microsoft To Do (bidirectional task sync — To Do wins) WORK SURFACES GitHub (PR reviews) · Slack · Jira KNOWLEDGE & UTILITY APIS Notion (notes DB, source of truth) · Tavily · WeatherAPI · IDFM transit · YouVersion · googlemaps · edge-tts (no key) EXECUTION & SELF-CONTROL E2B sandboxes (code exec + 3 delegation backends) · Railway GraphQL (self-redeploy) OBSERVABILITY CLOUDS Logfire · Langfuse · W&B Weave (optional) Bot API poll + send POST /health · /phone ingest DASHBOARD_SECRET local run HMAC webhook
Field notes
  • The Mini App page is served by Kwasi (GET /miniapp/, an unauthenticated shell) but opened inside Telegram — data calls authenticate with Telegram-signed initData, never DASHBOARD_SECRET.
  • All inbound push surfaces are secret-gated and fail closed when unconfigured: health/phone ingest, alert webhook (503 without its secret), /message, /reflect.
  • Anthropic is reached only from inside an E2B delegation sandbox with a metered API key — never from the main agent.
ALTITUDE 2 · STRUCTURE

Inside the container

One process, read top to bottom: interfaces feed a per-turn pipeline, the pipeline picks an agent, agents call tools, tools cross the approval gate before touching the world, and everything persists through one StoragePort. Two rails run beside every layer: observability watches it, background loops act without being asked.

What are the layers, and which rail sees which?band map
Observability rail
  • one OTEL TracerProvider
  • telegram.turn root span per turn
  • Logfire → infra + loops
  • Langfuse → LLM traces + scores
  • Weave → optional mirror
  • heartbeats per loop
  • 👍/👎 → user_feedback
Interfaces
Telegram bot — dispatch, media, commands, callbacks, reactions Mini App — 14 API routes WhatsApp webhook CLI REPL Routers: dashboard · health · phone · observability · api (/message, /reflect)
▼
Per-turn pipeline (Telegram)
fetch_message_history classify_intent → select_agent inject_context (1000-tok budget) planning gate (Spec 008) run_with_resilience (retry → mini fallback)
▼
Agent core — model bound per request, never at construction
full_agent — 158 tools 13 domain agents — utility · memory · news · github · slack · email · calendar · diagnostics · jira · drive · meetings · health · database composed agents — ≥2 domains, lru_cache(32) briefing_agent — loops only system prompt = persona + top-10 learnings + live DB schema + vault meta (byte-stable → prompt cache holds)
▼
Tool surface
26 agent_tools modules → implementations in app/tools/ 46 MCP wrappers → gmail ×2 · outlook/To Do · gcal · gdrive · logfire (all sync, via to_thread) 6 skills — read_later · travel · cv · research · meeting_notes · podcast sync delegation → E2B (3 backends) introspection — read own source, query own DB, diagnose self
▼
Approval gate — Telegram only
38 tools intercepted → PendingAction + sentinel, human confirms on phone 17 action types one shared resolve path for keyboard + Mini App fail-closed: only the CLI skips approval
▼
Memory & storage
StoragePort → single PostgresAdapter (SQLite deleted 2026-08) 21 tables — conversation · knowledge · proactive state · safety ledger · ingest · vault context KV doubles as system state: dedup keys, OAuth tokens, runtime config, msgtrace embeddings: row first, vector best-effort
Proactive rail — 17 loops
  • reflection (2 AM)
  • reminders · approval expiry
  • briefing · recaps · digests
  • meeting prep + follow-up
  • alerts + intentions
  • To Do sync · PR review
  • scheduled tasks (cron)
  • 7 currently silenced (Phase 0)
Field notes
  • Domain agents are compositions, not silos: every one carries the _UTILITY_FNS baseline (db, introspection, search, self-diagnosis), so "focused" means a focused prompt, not exclusive tool access.
  • One live Phase 0 experiment is left in this picture: ROUTING_ENABLED=false still sends every turn to the full agent, and decides whether router.py + routing/agents.py (~1,600 lines) get deleted.
  • The other read out, and the code is gone (#132, 2026-08-30). Replaying both drift detectors over 2,000 stored interactions beat waiting on logs: the semantic half had fired 0 times ever — nine anchors embedded at every boot, earning nothing — the regex half not since 2026-06-08, and most of its 31 historical matches were Kwasi being correct. PR #72 (instructions vs system_prompt) was the real cause of that drift; this had only ever been the compensation.
How does the process boot, and in what order?flowchart

create_app()

Settings() + export AI keys to os.environ

init_observability — one OTEL provider,
Logfire + Langfuse + Weave attach here

include 7 routers + GET /health

lifespan: build_deps → PostgresAdapter
(table DDL runs lazily on first pool connect)

load_overrides — re-apply persisted
runtime config from context KV

pre-warm 13 routing anchor sets
(deliberately NOT inside a span)

wire persistence: Outlook MSAL cache ·
Google OAuth broker → context KV

start_telegram_bot
(allowed_updates = ALL_TYPES — reactions need it)

recover_orphan_delegations →
'your delegation was interrupted'

confirm_pending_self_redeploy → ✅ / ⚠️

spawn 17 background loops (each env-gated)

serve · on shutdown: cancel loops, stop bot,
close pool, flush Langfuse

Field notes
  • The anchor pre-warm is deliberately not wrapped in a span. logfire.span writes to Logfire’s own tracer, which Langfuse never sees — but it is a valid OTEL span, so has_active_span() reported a parent and let each of the 59 anchor embeddings trace. Langfuse received 59 spans whose parent it had no record of and promoted every one to a root trace: 151 of 226 root traces over seven days were these, renewed on every deploy.
  • Table DDL is not a boot step — it runs lazily on the first pool connect, which is why no explicit storage bootstrap appears in the lifespan.
How is an agent's tool set assembled?flowchart

include_skills

none

only utility

one domain

two or more

_UTILITY_FNS baseline
db_query · db_execute · read_source_file ·
grep_source · diagnose_self · search_history ·
semantic_search · attention tools · verse …

_build_domain_agent()
dedups the lists — a duplicate once
silently ate the tool it duplicated

domain fn list
(e.g. _GITHUB_FNS, _MEMORY_FNS)

MCP set
(e.g. _EMAIL_MCP, _CALENDAR_MCP, _TODO_MCP)

6 skills via @skill registry

13 prebuilt domain agents

select_agent(categories)

full_agent — 158 tools

utility_agent

that domain agent

composed agent
lru_cache(32) on the frozenset

ALTITUDE 3 · RUNTIME

One turn, end to end

A Telegram text message crosses seven phases. Two short-circuits (edit corrections, bare URLs) never reach the agent at all, and a multi-step plan stops the turn at an approval card before the agent runs.

What actually happens between "message sent" and "reply delivered"?sequence · code-verified
Langfuse / OTELPostgresapproval_gateagent.run (Gemini)plannerrouterbot.handle_messageTelegramPHASE 1 · gates — before any span existsPHASE 2 · open telemetryPHASE 3 · enrichPHASE 4 · planning gatealt[plan with 2+ steps]PHASE 5 · runalt[gated (38 tools, telegram)][ungated or non-telegram]loop[tool loop]PHASE 6 · deliveralt[voice trigger matched]PHASE 7 · post-turn, spawn_with_otel_contextLawrencemessage1update (long poll)2allowlist check3pending edit? → revise PendingAction, return4bare URL? → save to read_later, return(no agent, no trace)5langfuse_root_span("telegram.turn") + turn_span6fetch_message_history — 10 chrono,or 3 recent + 3 semantic7classify_intent(text, prev-turn hint)8categories → select_agent (full / domain / composed)9inject_context — notes, summaries, read-laterunder a 1000-token budget10prepend datetime line to the user turn11classify_complexity(text)12ExecutionPlan13PendingAction("plan") + trace_id14plan preview + Confirm/Cancel — turn ends,agent never runs15"…" placeholder — ProgressReporter narrates tool calls onto it16run_with_resilience(primary, fallback=mini)17consequential tool call18save PendingAction + trace_id19sentinel [APPROVAL_PENDING:uuid]20execute immediately21response text22voice note (edge-tts)23edit placeholder — markdown, split at 409624record_message_trace (genre=chat) — before keyboards25scan tool calls for sentinels → attach approval keyboard(s)26root span output = response27log_interaction (+ best-effort embedding) · request entry · audit28extract_facts_from_exchange (mini model → fact-quality gate)29detect_and_save_capability_gap30
What no doc diagram showed until now
  • Bare URLs never reach the agent. A message that is only a URL is saved to read-later and returns before the root span even opens — it is invisible to Langfuse.
  • record_message_trace fires before the approval keyboards attach, not after (the docs draw it the other way).
  • Post-turn work uses spawn_with_otel_context, not bare create_task — that's why fact extraction nests under the turn's trace instead of orphaning.
  • The placeholder message is a live surface: ProgressReporter edits tool-call narration onto it, which is why the final markdown edit is the one most likely to hit Telegram's edit throttle.
Which gate can end the turn early, and in what order?decision flow

no

yes

yes

no

yes

no

plan ≥ 2 steps

no

incoming text

allowlisted?

ignored

edit
pending?

revise the PendingAction
via mini model — stays pending

bare
URL?

auto-save to read_later
no agent · no span

open telegram.turn root span

classify_intent → select_agent

inject_context + datetime line

multi-step?

plan approval card —
agent does not run yet

run_with_resilience → agent.run

deliver · keyboards · post-turn

ALTITUDE 3 · RUNTIME

Intent routing

Zero-cost keywords first, then embeddings against 13 semantic anchors with three confidence bands. The whole layer is behind a kill switch — and the switch is currently off, deciding whether ~1,600 lines get deleted.

How does a message pick its agent?decision tree

false — Phase 0 experiment, live in prod

true

yes — substring over 13 domains

no

≥ 0.65

0.45 – 0.65

< 0.45

yes

no

keyword_hits + semantic_top3
logged per decision

message text

ROUTING_ENABLED?

full_agent, 158 tools
(every turn since 2026-08-21)

keyword
hit?

matched categories + utility
(method: keyword, zero cost)

embed message vs 13 anchor sets

best cosine
score

single best domain

top-K ≤ 3 within 0.10 of best →
composed agent, LLM picks

one-domain
prior hint?

inherit that domain

no categories

select_agent

none → full_agent

only utility → utility_agent

1 domain → prebuilt agent

≥2 → composed (lru 32)

Langfuse span

Field notes
  • The 13 domains: email · calendar · github · memory · news · slack · jira · drive · meetings · health · database · diagnostics · utility. Keywords and anchors use the identical key set.
  • The middle band exists because binary cutoffs misrouted borderline scores — attaching the wrong tool set is worse than letting the LLM choose among three.
  • An offline eval harness (eval/) gates routing in CI on the free keyword path, and a trace-miner promotes real prod misroutes into the dataset.
ALTITUDE 3 · RUNTIME

Approval gate

The safety spine: 38 consequential tools don't execute — they park a PendingAction and hand the agent a sentinel. A human on a phone is the executor. Both resolve surfaces (inline keyboard, Mini App) share one code path, so they cannot diverge.

What states can a pending action move through?state machine

approval_gate saves PendingAction,
returns sentinel to the LLM

Edit — mini-model revises payload,
re-validated by the originating tool's own guards
(scores user_edit)

Confirm — keyboard or Mini App

executor runs, audited
(scores user_approval 1.0)

executor raised
(scores user_approval 0.0)

Cancel
(scores user_approval 0.0)

expiry loop — 30 min default,
24 h for meeting follow-up cards

pending

confirmed

executed

failed

cancelled

expired

How do two different surfaces resolve one action identically?sequence
LangfusePostgresexecutor (ACTION_REGISTRY)app.approvalMini App · Approve tabInline keyboarddatabase · deployment · config · delegation ask forfingerprint or face first (BiometricManager, Bot API 7.2)one shared confirm_action / cancel_action —the third time this repo paid for duplicated logicalt[executor succeeds][executor raises]Mini App edits only revise — validate_payload runs theoriginating tool's guards (e.g. db_execute's DDL + table bans),and the action stays pendingcallback approve:<id>POST /approvals/<id>/confirmget_pending_action (pending, unexpired)execute_approved_action(tool_name)resultstatus=confirmed · audit entry with channelscore user_approval = 1.0(trace_id was captured at gate time)status=failedscore user_approval = 0.0
Field notes
  • Only the CLI skips the gate. Loops and /message get a card in the briefing chat; WhatsApp and unknown interfaces are refused — AgentDeps.interface decides, and its default is "system".
  • The gate protects itself: db_execute refuses SQL touching audit_log, pending_actions, vault_credentials — the tables that make approval mean anything.
  • Approval cards show live consequences before Confirm: a transactional dry-run prints "will affect N rows" for raw SQL; delegation cards name the resolved backend.
ALTITUDE 4 · SUBSYSTEM

Memory & storage

Three write cadences (per exchange, per save, nightly) and four read paths into every prompt. The hard-won rule of this subsystem: a prompt cannot enforce a filter; code can — quality gates and overwrite protection live in Python, not instructions.

What writes memory, what gates it, and what reads it back?data flow

READ PATHS

WRITE PATHS

every exchange

extract_facts_from_exchange
mini model, fire-and-forget

fact_quality gate — code, not prompt:
rejects kwasi_*/railway_* self-state,
jira mirrors

user_facts upsert —
source='user' rows cannot be
overwritten by extraction

2 AM reflection (LLM)
wall-clock anchored

profile · facts · intentions (≤3 follow-ups) ·
learnings + topic-summary notes

save_note · save_interaction ·
save_read_later

row INSERT first, then embed —
gemini-embedding-001, halfvec(3072),
vector UPDATE is best-effort

build_system_prompt — profile +
top-10 learnings by recurrence +
live DB schema + vault meta

fetch_message_history —
last 10 chrono, or 3 recent + 3 semantic

inject_context, 1000-token budget:
notes ≥0.6 → summaries ≥0.6 →
read-later tag overlap, XML-wrapped

recall_facts on demand — cap 60
(once returned all 760, ~45k chars)

Field notes
  • The source='user' guard exists because prod showed a correction made in the evening being silently reverted by the 2 AM reflection — 541 reflection-updated facts in a week vs 2 user facts.
  • Learnings rank by recurrence_count then recency, capped at 10 — 160 one-off incidents once steered every turn as policy.
  • Embedding is awaited inline but after the user's reply is sent, so durability costs the user nothing.
What forgets, and on what clock?retention map

hourly cleanup

msgtrace rows past TTL —
prefix-scoped delete (starts_with,
not LIKE; empty prefix raises)

nightly reflection

*_context facts unseen 90 d
(EPISODIC_FACT_TTL_DAYS — suffix-scoped:
a broader sweep once matched a real phone number)

5-min loop

pending approvals past expiry

scripts/memory_cleanup.py
dry-run by default

the backlog no automatic
retention path reaches

ALTITUDE 4 · SUBSYSTEM

Background loops

The proactive half of the system: 17 asyncio loops spawned at boot. Seven are deliberately silenced since the Phase 0 reduction (measured 9.19:1 proactive-to-user ratio); each loop's 👍/👎 engagement is now measurable per genre, which decides who comes back.

Every loop, by cadence — who's on, who's silenced, and what gates it?cadence board
LoopWhenGateDedup / guardStatus
Continuous — every minute or few
User scheduled tasksevery 60 s (croniter)briefing chatper-task cron stateon
Remindersevery 5 minTelegram tokendue-time checkon
Approval expiryevery 5 minTelegram tokenexpires_aton
Alert rules + intentionsevery 5 minbriefing chatrule state · ≤3 follow-ups per intentionon
Meeting prepevery 5 min, fires ~30 min aheadbriefing chatsystem:meeting_prep:<event> · skips solo blocks (no attendees)on
Meeting follow-upevery 30 minwork Gmail tokensystem:meeting_followup:gmail:<msg>on
To Do sync (reconcile)every 300 sflag + Outlook credslast_synced watermarkon
Daily
Reflection02:00 UTCREFLECTION_SECRETwall-clock re-anchored (drifted 02:00→08:25 once)on
Morning briefing08:00 UTCbriefing chatsystem:briefing (today's date)on
Email intelligence09:30 localbriefing chat + email credsempty time = disabledsilenced
PR review09:00 localGitHub token + org allowlistsystem:pr_review:*, 18 h TTLsilenced
Evening recap21:00 localbriefing chatdate keysilenced
Phone digest21:30 localbriefing chat + flagsystem:phone_digestsilenced
Weekly
Weekly recapFri 18:00 localbriefing chatweek keyon
Weekly prepSun 18:00 localbriefing chatweek keysilenced
Read-later digestSat 09:00 localbriefing chatweek keysilenced
Journal digestSun 19:00 localbriefing chatweek keysilenced
Field notes
  • Expected proactive volume with the current on-set is ~5.1 messages/day — meeting prep (2.8/day) and alerts (1.2/day) dominate, not the briefings.
  • An empty *_TIME is a first-class "disabled" value, so any silence is reversible from Telegram via set_runtime_config without a redeploy.
What machinery does every loop share?flowchart

re-anchor to wall clock
(never 'sleep 24h' — that drifts)

env gate — empty *_TIME
means disabled

context-KV dedup key —
survives container restarts

do the work
(often the briefing_agent — a real LLM run)

send via Telegram

record_message_trace(genre) —
👍/👎 engagement measurable per loop

write heartbeat — read by diagnose_self,
Command Centre, Mini App Status

ALTITUDE 4 · SUBSYSTEM

Integrations & delegation

Four distinct coupling styles, chosen deliberately: MCP wrappers for mail/calendar/drive, a bidirectional sync engine for To Do, source-of-truth reads for Notion and Calendar, and sandboxed delegation for work no tool covers.

How do 46 agent tools reach Google and Microsoft without ever holding a password?component flow

persist rotated refresh tokens

46 @agent.tool wrappers
(interfaces/mcp/client.py)

asyncio.to_thread —
every upstream SDK is sync

gmail_server
personal 8 · work 7

outlook_server
mail 7 · calendar 5 · To Do 6

google_calendar_server
multi-calendar resolver,
writable-only write targets

google_drive_server
personal + work

logfire_server
query · exceptions · schema

credential_broker —
token cache + lock,
60 s expiry skew

Google APIs

MSAL token cache

Microsoft Graph

context KV — system:oauth:*

How does a dead refresh token heal itself?sequence
context KVGoogle OAuthcredential_brokerMCP wrapperre-authing the env var now self-heals —this failure bit twice and used to need manual SQLalt[invalid_grant, and the env token differs from stored][invalid_grant, env token is the same]get_access_token(provider, account, scopes)under lock — cached token with >60 s left? return itrefresh(stored token — DB-seeded beats env)retry with the env refresh tokenfresh access tokenadopt + persist the env token (system:oauth:…)raise — account genuinely needs re-authaccess token
What happens when Kwasi hires help it doesn't have a tool for?delegation pipeline

delegate_to_coding_agent(task, backend?)

pre-checks: backend credentials ·
daily cap (delegation_max_per_day)

delegate_web_task(task) — a dedicated tool,
so browser intents route correctly

approval gate — the card names
the resolved backend

Delegation row + spawn run_delegation —
agent replies 'working on it' immediately

E2B sandbox — template per backend

opencode — Gemini,
plain stdout answer, cheap default

claude_code — metered ANTHROPIC_API_KEY only,
stream-json: cost cap, tool telemetry,
progress ≤1 msg / 30 s

web_task — browser-use + Gemini,
runner injected via setup_files,
task via WEB_TASK_PROMPT env,
headless, BROWSER_ALLOWED_DOMAINS

vault (Fernet) — decrypted only at run time,
into browser-use sensitive_data placeholders;
the LLM never sees plaintext

result → separate Telegram message
+ Note tagged #delegation

container restart → recover_orphan_delegations
marks running→failed, notifies

How do two task lists stay one task list?sync sequence
Microsoft Graph · defaultList_todo_sync_loop (300 s)tasks tabletask executorsPUSH — inline, best-effort (same posture as embeddings)PULL — reconcile every interval · conflict rule: To Do winsalt[fetch fails]create / update / complete / deletepush_created / push_updated / push_deletedfetch all itemsbail — a failed fetch never deletes localslinked item strictly newer than last_synced → apply remoteitem at the watermark → skip (echo guard —our own push never round-trips)linked but absent from a SUCCESSFUL fetch → delete localunmatched remote → adopt same-title local, else createremaining unlinked locals → push up
How does the physical world get in?ingest flow

POST /health/ingest
X-Health-Secret

POST /phone/usage/ingest
X-Phone-Secret

Galaxy Watch

Google Health Connect

bridge-android — sideloaded Kotlin,
WorkManager ~15 min

UsageStatsManager —
screen time · unlocks

health router

phone router

health_samples — 11 metric types,
idempotent on (type, start, device)

app_usage_daily —
per (date, package, device) upsert

4 read tools · briefing snapshot ·
reflection health block · health_metric alerts
(resting HR derived: 5th percentile of ~1/min stream)

attention tools · nightly phone digest ·
reflection attention block

Field notes
  • Push model only — the server never polls the phone. Unset secrets make each ingest surface inert.
  • The notification half was removed (#104) after 15 days of real data: 83% was one repeating "Data warning", and messaging rows carried counters, not content. What's worth capturing next, if ever: notification dwell time.
  • Notion is the deliberate contrast to To Do sync: no mirror, no loop — Notion is the source of truth, read live.
ALTITUDE 4 · SUBSYSTEM

Observability & feedback

One tracer provider, three clouds watching the same spans — and a set of taps that turn user behavior into scores on the exact trace that earned them, even when the signal arrives minutes later.

Where does every span go?fan-out

FastAPI spans
(bridge ingest routes excluded —
they once burned retention to ~15 min)

one shared OTEL TracerProvider
(init_observability)

Pydantic AI agent spans
Agent.instrument_all()

turn spans + background loops

OTEL_RESOURCE_ATTRIBUTES — set before any
provider exists, so langfuse.environment + release
land on the Resource every span inherits

Logfire — infra, loops, exceptions,
tail-sampled

Langfuse — LLM traces, sessions,
prompts, scores, experiments

W&B Weave — optional OTLP mirror,
off by default

Field notes
  • Setting LANGFUSE_TRACING_ENVIRONMENT / LANGFUSE_RELEASE is not enough. The Langfuse SDK builds a Resource carrying both, but only applies it when it creates the tracer provider; Logfire configures first here, so the Resource is silently dropped and only SDK-created spans get stamped.
  • The split was exact inside one verified turn: telegram.turn and every @observe span carried environment=production, while the agent run, 4 generations and 11 tools carried environment=default. Cost lives entirely on the generations — so filtering to environment = production returned $0.00, silently.
  • OTEL_RESOURCE_ATTRIBUTES is the provider-agnostic fix: Resource.create() merges it whoever calls it, so it must be set before any provider exists. Found by reading the live project after the v4 migration, not by a failing test.
How does one turn nest?span tree
telegram.turn — root span · session=telegram:<chat> · input/output   └─ telegram/message — stage span       ├─ message_history_retrieval (@observe)       ├─ telegram/routing — keyword_hits · semantic_top3       ├─ agent run — model request(s), tool spans, usage rollup       └─ post-turn fact extraction + capability gap — still nested, via spawn_with_otel_context
What measures whether any of this is worth it to the user?signal taps

Confirm / Cancel at the gate

user_approval 1.0 / 0.0

Edit at the gate

user_edit

exception during a turn

agent_error

👍 / 👎 reaction on ANY message —
the only signal covering turns
that never hit the gate

system:msgtrace:<chat>:<msg> —
written at send time, carries the genre
(chat, briefing, alert, digest …)

user_feedback (boolean) +
per-genre engagement — the query that
decides which silenced loops return

Langfuse scores, attached to the trace_id
captured when the signal was armed

10 managed prompts — Langfuse UI ·
code-constant fallbacks · prompts.lock.json
sha256 pins · check_drift() at boot ·
sync_prompts.py is the only mutator

Field notes
  • Reactions only arrive because polling passes allowed_updates=Update.ALL_TYPES — Telegram excludes message_reaction from the default set, and naming any type switches the default off entirely. A test asserts the polling call itself.
  • Delayed signals share one shape: capture trace_id at arming time (PendingAction.trace_id, msgtrace rows), score whenever the human gets around to it.
ALTITUDE 4 · SUBSYSTEM

Self-management

Kwasi reconfigures and redeploys itself from a chat message — safely, because one registry derives every allowlist, secrets are deny-listed by shape, and a redeploy must prove it survived.

Two control planes: instant vs baked-in?flowchart

PLANE B — baked into the Railway env

PLANE A — instant, survives restart

CONFIG_REGISTRY — single source of truth:
ConfigField(field, env, validator, coerce,
runtime_settable, env_settable)

ALLOWED_CONFIG_KEYS
(runtime allowlist — derived)

RAILWAY_ENV_ALLOWLIST
(env allowlist — derived)

independent deny-substring gate:
TOKEN · SECRET · KEY · PASSWORD ·
DATABASE_URL · CREDENTIAL

set_runtime_config (approval-gated)

live Settings in memory —
effective on the very next message
(model is bound per request)

context KV system:config:*

load_overrides at boot

railway_set_env (approval-gated) —
validates via the registry BEFORE upsert,
so a bad MODEL_NAME can't crash-loop

Railway GraphQL upsert

railway_redeploy

write system:pending_self_redeploy
(chat, expected config, prev deploy id)

next boot: confirm_pending_self_redeploy —
verifies a NEW deployment happened,
posts ✅ or ⚠️, clear-before-send

ungated read-only companions:
diagnose_self · railway_deployment_status ·
railway_deploy_logs

Field notes
  • Plane A is what makes the two live experiments abortable from Telegram without a redeploy.
  • RAILWAY_API_TOKEN itself is never agent-settable — the deny-substring gate blocks it regardless of any registry entry.
ALTITUDE 4 · SUBSYSTEM

Telegram Mini App

The first pull-shaped surface: a web page inside the chat that finally shows the ~650 facts and pending approvals that were previously write-only. Auth is cryptographic, not a shared secret — and the docs have zero diagrams for it, so these are the first.

How is a request trusted before anything in it is read?auth sequence
/miniapp/api/*verify_init_datapage JSGET /miniapp/ — unauthenticated shellTelegram appnothing in the payload — not even user id —is read before this signature check passesalt[any step fails]open Mini Appstatic HTML — contains no data by designfetch with Telegram-signed initDataverify FIRST — payload untrusted until then1 · bot token configured? initData present?2 · parse fields, pop hash3 · HMAC("WebAppData", bot_token) →HMAC over sorted k=v lines → compare_digest4 · auth_date window — ≤ 24 h old, ≤ 5 min future5 · parse user JSON, require id6 · same allowlist as the bot —an EMPTY allowlist blocks everyonegeneric 401 — the reason stays server-sidedata
Which tab hits which of the 14 endpoints, and what makes each real?surface map

Today

GET /api/today — one round trip
GET /api/agenda — calendar kept separate,
so first paint never waits on Google

Memory

GET /api/facts (search) ·
PATCH /api/facts/:key · DELETE /api/facts/:key —
a correction becomes source='user',
which extraction can no longer overwrite

Approve

GET /api/approvals + count badge ·
confirm / revise / cancel —
all through app.approval,
the same path as the inline keyboard.
Risky classes gated by biometrics

Status

GET /api/status — deploy state, heartbeats,
recent exceptions · POST /api/redeploy ·
GET /api/delegations (+ detail)

Field notes
  • Four page-level bugs were invisible to all ~1,400 tests and found only by driving the real page: a confirm API that throws below Bot API 6.2, stale-response rendering, escaped-JSON walls, and a literal × on delete buttons. UI truth needs a browser.
  • Editing an approval payload re-runs the originating tool's own validators — an edit path that skipped db_execute's guards would be worse than no edit path.
Jump to