Memory & Reflection¶

Kwasi has three layers of memory working together: short-term conversation history, a permanent facts store updated in real-time, and a long-term narrative profile (plus topic-summary notes) built nightly by the Reflection Engine.

Memory Architecture¶

graph TD
    subgraph ShortTerm["Short-Term (per session)"]
        H[Last 10 interactions\nfetched from storage per request]
        H --> MH[message_history\nModelRequest / ModelResponse pairs]
    end

    subgraph PostConv["Post-Conversation (fire-and-forget, bot.py)"]
        PCE[extract_facts_from_exchange\nruns immediately after every message]
        PCE --> UF
    end

    subgraph PermanentFacts["Permanent Facts (durable; narrow nightly decay — see Fact decay)"]
        UF[(user_facts table\nkey-value pairs\nfetched at runtime via recall_facts)]
        WU[User explicit:\n'remember that my home is...']
        WA[Agent proactive:\nuser mentions address mid-conversation]
        WR[Reflection auto-extraction:\nnightly LLM pass extracts new facts]
        WU --> UF
        WA --> UF
        WR --> UF
    end

    subgraph LongTerm["Long-Term Narrative (nightly 2 AM)"]
        I[All interactions in last 24h] --> RS[ReflectionService]
        RS --> LLM[LLM — tool-less agent]
        LLM --> UC[UserContext\nmarkdown profile ≤550 words\n6 sections]
        LLM --> NF[New facts JSON array]
        LLM --> SN2[Conversation cluster notes\nSummary: topic]
        UC --> CT[(context table)]
        NF --> UF
        SN2 --> Notes[(notes table)]
    end

    subgraph SystemPrompt["Every Request — build_system_prompt (@agent.instructions)"]
        CT --> PS["'Your Memory of This User'\nnarrative profile · capped 4,800 chars"]
        AL[(agent_learnings\nstatus=active)] --> BG["'Behavioral Guidelines'\nranked by recurrence, then recency\nhard cap 10"]
        SCH["Live database schema\ndescribe_schema() · process-cached"] --> Agent
        Notes --> SRJ[find_relevant_summaries\ninjected into the user turn if similar]
        PS --> Agent[Agent run]
        BG --> Agent
        MH --> Agent
        SRJ --> Agent
    end

    subgraph AtToolTime["Fetched at tool-call time, NOT injected"]
        UF -.->|recall_facts| Agent
    end

Memory timeline¶

Tier	Latency	What gets saved
Post-message	Seconds	Explicit facts stated in the current exchange (`user_facts`)
Nightly	2 AM UTC	Full profile rewrite, intention extraction, behavioral learnings, topic-cluster `Summary: <topic>` notes

Note (2026-05-09): The 30-minute session summariser was removed. It fired on every quiet period per chat with no quality gate, writing fuzzy Summary: notes whose keys diverged from the nightly reflection's keys (so they didn't dedup) and whose contents (e.g. "user asked to delete X, assistant refused") got semantically re-injected into later prompts, biasing the agent to repeat past refusals. Topic-summary recall now happens only via the 2 AM nightly reflection. Same-day topic-summary recall is the accepted trade-off.

Short-Term Memory¶

On every message (from any interface), the handler fetches recent interactions for that user via fetch_message_history() in app/utils/message_utils.py, then passes them to build_message_history() which applies a token budget before sending them to the model:

recent = await fetch_message_history(
    storage=deps.storage,
    user_id=user_id,
    message=text,
    settings=deps.settings,
)
message_history = build_message_history(recent)  # drops oldest if > 6,000 est. tokens

Two retrieval modes, controlled by ENABLE_SEMANTIC_HISTORY:

Chronological (default) — last recent_count + semantic_count interactions ordered newest-first. Same behaviour as before this feature shipped.
Semantic (ENABLE_SEMANTIC_HISTORY=true) — last 3 verbatim turns (preserves dialog coherence) plus top-3 semantically-relevant older interactions, with a small recency boost on similarity so equally-relevant recent matches surface first. Falls back to chronological on any failure (embed error, search error, hydrate error).

Both modes are wrapped in @observe(name="message_history_retrieval") so the retrieval step appears as a child observation under telegram.turn in Langfuse with metadata {mode, recent_count, semantic_count, semantic_enabled}.

build_message_history() then estimates tokens at ~4 chars/token and drops the oldest interactions first when the total exceeds TOKEN_HISTORY_BUDGET = 6000. The resulting list is converted into Pydantic AI ModelRequest / ModelResponse pairs and passed as message_history to agent.run().

Persona-drift scrubbing. Past assistant turns containing strong "I'm a generic LLM" patterns (e.g. "as a large language model", "I'm trained by Google", "I don't have a tool/access/codebase") are detected by a calibrated regex applied to the lead 400 chars of each replayed response. On match the entire response is replaced with "[earlier response omitted]" before being passed back as history. Without this, the model anchors to its own prior break-character text on the next turn — confirmed in Langfuse traces where Kwasi correctly routed to diagnostics_agent with grep_source attached but still hallucinated "as a language model, I don't have source code" due to recent assistant turns saying the same. Scrub count is logged at INFO level so frequency can be tracked in Logfire. Patterns are conservative — they distinguish drift signals ("I don't have a tool") from clean no-results responses ("I don't have any unread emails").

Why a token budget instead of a fixed count? A hard limit of 10 interactions behaves inconsistently — 10 one-line exchanges cost far fewer tokens than 10 multi-paragraph exchanges. The budget-based approach keeps context window usage predictable regardless of message length.

Tunable settings (all in app/config.py): - ENABLE_SEMANTIC_HISTORY (bool, default false) - SEMANTIC_HISTORY_RECENT_COUNT (default 3) - SEMANTIC_HISTORY_SEMANTIC_COUNT (default 3) - SEMANTIC_HISTORY_THRESHOLD (default 0.6)

Semantic Context Injection¶

Before every agent run, up to three retrieval layers prepend relevant context to the user message. All three layers share a single 1,000-token budget (CONTEXT_TOKEN_BUDGET) — they are filled in priority order until the budget is exhausted.

flowchart TD
    A[User message arrives] --> B{GOOGLE_API_KEY set?}
    B -- No --> Z[Send raw message to agent]
    B -- Yes --> C[budget = 1,000 tokens]
    C --> D[Layer 1: find_relevant_notes\n≥0.6 similarity · top 2 · +recency boost\nexcludes Summary and Research prefixes]
    D --> E{Notes found\nand budget > 0?}
    E -- No --> G
    E -- Yes --> F[Append XML block\nbudget -= note_tokens]
    F --> G[Layer 2: find_relevant_summaries\n≥0.6 similarity · top 2 · +recency boost\nmatches Summary: prefix only]
    G --> H{Summaries found\nand budget > 0?}
    H -- No --> J
    H -- Yes --> I[Append XML block\nbudget -= summary_tokens]
    I --> J[Layer 3: find_relevant_read_later\ntag-overlap matching · up to 3 items]
    J --> K{Items found\nand budget > 0?}
    K -- No --> M
    K -- Yes --> L[Append XML block]
    L --> M[Prepend datetime: local time in user timezone]
    M --> Z

Each layer wraps its output in XML tags so the model can distinguish retrieved memory from instructions:

<context type="notes">
- Note title: first 200 chars of content
Build on these — don't repeat them verbatim.
</context>

<context type="summaries">
- Topic name: first 200 chars of summary
Use for continuity — don't reference it explicitly.
</context>

<context type="read_later">
- "Article title" (https://...)
  Summary: ...
  Your note: ...
Mention these and include the URL.
</context>

Threshold history. Layers 1 and 2 ran at 0.7 / 0.75 until May 2026. They were lowered to 0.6 after measurement showed a ~4× miss-to-hit ratio in the 0.55–0.70 band — the model wanted that context, the threshold was just too strict. The recency boost (≤+0.05, decays to 0 at 90 days) prevents the looser threshold from surfacing stale matches.

Datetime in the user turn: the current local time is prepended as [Thursday, April 24, 2026 — 09:15 AM Europe/Paris] to the user turn (not the system prompt). This keeps the ~3,000–5,000 token system prompt identical across requests, qualifying it for Gemini's implicit prompt cache. A changing timestamp in the system prompt would bust the cache on every request.

Logfire span attributes: each Telegram message span records est_tokens_history, est_tokens_context, est_tokens_user_turn, context_layers, and history_interactions so token distribution is observable in Logfire per request.

Permanent User Facts¶

The user_facts table stores specific, verifiable facts about the user — the kind that get pruned from or never make it into a 550-word narrative profile: addresses, phone numbers, names, dietary restrictions, and similar.

How facts get written¶

Path	Trigger	`source` value	Latency
Explicit	User says "remember that my home is X"	`"agent"`	Immediate
Proactive	Agent calls `remember_fact` mid-conversation	`"agent"`	Immediate
Post-conversation	`extract_facts_from_exchange()` fires after every message; mini-model extracts explicitly stated facts. Pre-checks existing value — skips if unchanged, upserts if new or different.	`"agent"`	Seconds
Auto-extraction	Nightly reflection extracts new facts from 24h conversations	`"reflection"`	2 AM UTC
Correction	You edit a fact in the Mini App's Memory tab	`"user"`	Immediate

A user-asserted fact outranks an extracted one

save_user_fact will not let a reflection or agent write overwrite a row whose source is "user":

ON CONFLICT (key) DO UPDATE SET ...
WHERE user_facts.source <> 'user' OR EXCLUDED.source = 'user'

The upsert used to be unconditional, which made correcting a fact pointless: a value fixed in the evening was reverted by the 2 AM reflection re-extracting the same key, and because source was overwritten too, the correction erased the evidence it had ever happened. Production showed the shape of it — 541 reflection facts updated within the week against 2 user facts, both untouched since March.

The rule lives in the ON CONFLICT clause rather than in the callers, because there are three writers and a fourth would not know to check. To hand a fact back to extraction, delete it — a deleted fact is simply re-derived, an asserted one is not overruled.

Agent tools¶

All three tools are registered on every domain agent (via _UTILITY_FNS) — not just the memory agent. The agent may learn personal facts during any conversation, regardless of domain.

Tool	What it does
`remember_fact(key, value, category)`	Upsert a fact by key. Overwrites previous value if the key already exists.
`recall_facts(query="")`	Search keys + values, or with an empty query list the 60 most recently updated facts grouped by category (see the cap below).
`forget_fact(key)`	Delete a fact by key.

Key naming: snake_case, descriptive. Examples: home_address, workplace, partner_name, preferred_transport, dietary_restrictions, manager_name, morning_routine, birthday.

Categories: location, personal, preference, work, health, general.

How facts get read¶

Facts are not injected into the system prompt — they are fetched at runtime when the model calls recall_facts(). (Only the narrative profile is injected. See Where the profile is used.) Output is grouped by category:

*Location*
- home_address: 124 Avenue Perretti, Neuilly-sur-Seine
- workplace: La Défense, Paris

*Personal*
- partner_name: Ana

The unfiltered listing is capped at 60

A bare recall_facts() returns the 60 most recently updated facts, then a [N older facts not shown — call recall_facts with a query to search them.] footer. Production had reached 760 facts (~45k chars, ~12k tokens), and the entire dump landed in one tool result that then rode along in message history for the rest of the turn. Most-recently-updated wins, since that is what a bare "what do you know about me" actually wants. A query is unaffected — search still spans everything.

The system prompt _MEMORY_INSTRUCTIONS instructs the agent: - Check permanent facts before saying you don't know something about the user - Save personal information proactively without asking permission - Priority order: permanent facts → reflection profile → recall_facts search → search_history - Never say "I don't have your address" without checking all four sources

Fact decay¶

decay_user_facts() runs once per nightly reflection and deletes a narrow class of rapidly-stale auto-facts:

Pattern	Grace period
Date-stamped snapshot keys (`_2026_05_07`, `_2026_05_07_2222`)	7 days
`task_completed_*`	30 days
External-system shadows — `db_row_count_`, `outlook_unread_count_`, `microsoft_todo_task_`, `read_later_article_`	immediate
Degenerate boolean keys — `has_system`, `has_work`, `has_outlook_email`, `has_wife`, `has_children`	immediate
`*_context` keys not updated in `EPISODIC_FACT_TTL_DAYS` (90)	90 days

The *_context sweep was added in August 2026, after the existing patterns were measured against real production data and matched zero rows.

Decay must not eat a phone number

The _context suffix scoping is deliberate and was arrived at empirically. A production dry-run of a broader contact_* / meeting_* sweep matched contact_ann_fesu_phone and contact_ann_fesu_email — durable contact details, exactly the thing permanent facts exist to hold. Any future widening of decay should be dry-run against prod first.

Post-Conversation Memory Pipeline¶

app/memory/post_conversation.py runs one background function after every handle_message call in bot.py via asyncio.create_task(). It is fire-and-forget — it never raises and never blocks the response to the user.

Immediate fact extraction¶

extract_facts_from_exchange(user_message, agent_response, storage, settings) fires immediately after every Telegram message. It sends the last exchange (user + agent, capped at 1000 chars each) to mini_model_name with a prompt that extracts only explicitly stated facts:

"By the way, I moved to Berlin last month."
→ [{"key": "home_city", "value": "Berlin", "category": "location"}]

"Hi, how's it going?"
→ []

Before upserting each fact: 1. get_user_fact(key) is called to check the existing value 2. If value is identical — skip (no write, no re-embedding) 3. If value is different or absent — save_user_fact() upserts it

This means a fact like "I moved to Munich" correctly overwrites an existing home_city = "Berlin" within seconds of being stated, rather than at 2 AM.

Topic summaries¶

Topic-cluster summaries (Note(title="Summary: <topic>")) are produced only by the nightly reflection's _summarise_conversations() step (see below) — there is no shorter-loop summariser. find_relevant_summaries() in message_utils.py then injects matching summaries as context into future conversations.

Long-Term Memory — The Reflection Engine¶

The Reflection Engine (app/memory/reflection.py) runs nightly at 02:00 UTC, driven by the in-process _reflection_loop (there is no external cron — see Background Loops → Reflection Loop). It produces four outputs from the last 24 hours of interactions:

An updated narrative profile (UserContext) — six sections, ≤550 words
A structured facts list — JSON array of new/changed facts, saved to user_facts
A structured intentions list — JSON array of newly detected personal commitments, saved to pending_intentions
A structured learnings list — JSON array of behavioral corrections, saved to agent_learnings

Reflection cycle¶

flowchart TD
    A([2 AM UTC trigger]) --> B[Fetch interactions\nsince 24h ago]
    B --> C{Any interactions?}
    C -- No --> D([Skip — log and return])
    C -- Yes --> E[Fetch existing UserContext\nfrom context table]
    E --> EF[Fetch existing user_facts + intentions + learnings\nto avoid re-extraction]
    EF --> PB[Append phone signals block\nattention snapshot + 7d app usage\ninto the conversations slot]
    PB --> F[Build reflection prompt\nexisting profile + facts + intentions + learnings + conversations]
    F --> G[Call LLM — tool-less Agent]
    G --> H{Parse output\n---PROFILE--- / ---FACTS--- / ---INTENTIONS--- / ---LEARNINGS--- markers}
    H --> I[Updated profile markdown\n≤550 words, 6 sections]
    H --> J[New facts JSON array\nonly new or changed facts]
    H --> JI[New intentions JSON array]
    H --> JL[New learnings JSON array]
    I --> K[Save UserContext\nto context table]
    J --> L[Save each UserFact\nto user_facts\nsource='reflection']
    JI --> LI[Save each PendingIntention\nto pending_intentions]
    JL --> LL[Save each AgentLearning\nto agent_learnings\ncandidate → active at ≥2 cycles]
    K --> M([Re-anchor to next 02:00 UTC])
    L --> M
    LI --> M
    LL --> M
    M --> A

How learnings reach the model¶

Learnings saved to agent_learnings with status='active' are injected into every system prompt under a ## Behavioral Guidelines header ("Rules learned from past feedback. Follow these precisely."), grouped by category.

They are ranked by recurrence_count first, last_seen second, then hard-capped at 10 (_MAX_INJECTED_LEARNINGS).

Why a cap and a ranking, not a recurrence threshold

Under a recency-only cap, a single night's one-off incidents could fill the entire section. Production reached 160 active rules, every one at recurrence_count=1 — so roughly 25 unrepeated observations were steering every turn as if they were settled policy.

A hard recurrence_count >= 2 filter was considered and rejected: the count only increments on an exact rule text match (save_agent_learning's ON CONFLICT), and the reflection model rephrases the same lesson each night, so repeats rarely collide. The gate would have deleted the feature rather than tightened it.

Ranking instead lets genuinely repeated rules win the slots when they exist, while the lower cap bounds the damage when they don't.

Ambient phone signals¶

When Phone Awareness is configured, reflection also receives an attention + rhythm block — get_attention_snapshot() plus get_app_usage_summary(days=7) — appended to the conversations slot rather than added as a new prompt placeholder. The profile's "Patterns & Rhythms" section absorbs it naturally. Fail-safe: an empty string on any error.

The reflection prompt¶

The LLM is given four inputs: 1. The existing profile (or "start fresh" if none exists) 2. The existing facts — so it knows what's already stored and avoids duplicates 3. The last 24h of conversations — formatted as [timestamp]\nUser: ...\nKwasi: ... 4. Today's date (for temporal context)

It must produce output in this exact format:

---PROFILE---
## Personal
...
## Goals & Values
...
## Communication Style
...
## Patterns & Rhythms
...
## Preferences
...
## Notes
...

---FACTS---
[{"key": "home_address", "value": "...", "category": "location"}]

---INTENTIONS---
[{"text": "call doctor about knee", "follow_up_days": 3, "context": "Mentioned knee pain was getting worse"}]

---LEARNINGS---
[{"rule": "always confirm before deleting any item", "category": "decision_making"}]

Profile rules: Six sections with word budgets. Specific values (addresses, names) belong in FACTS, not in prose. Maximum 550 words. Return unchanged if nothing new was learned.

Facts rules: JSON array of objects with key, value, category. Only include facts that are new or have changed since the existing facts list. Empty array [] if nothing new.

Intentions rules: JSON array of objects with text and follow_up_days. Only include soft commitments not already in the existing intentions list. Empty array [] if nothing new. Examples: "I should call the dentist", "I want to start running again", "I need to finish that report".

Parsing and fallback¶

_parse_reflection_output(text) returns a tuple[str, list[dict], list[dict], list[dict]] (profile, facts, intentions, learnings): - Splits from the end: ---LEARNINGS--- first, then ---INTENTIONS---, then ---FACTS---, then ---PROFILE--- - Profile text = everything between ---PROFILE--- and ---FACTS--- - Facts = JSON parsed from the section between ---FACTS--- and ---INTENTIONS--- - Intentions = JSON parsed from between ---INTENTIONS--- and ---LEARNINGS--- - Learnings = JSON parsed from after ---LEARNINGS--- - If markers are absent: entire output treated as profile, no structured data extracted (backwards-compatible fallback with a logged warning) - Markdown code fences around the JSON arrays are stripped automatically

Where the profile is used¶

Every time the agent runs, build_system_prompt() fetches the current UserContext from storage and injects it under a "## Your Memory of This User (narrative)" header (the narrative profile is injected; permanent facts are not — the prompt instructs the model to call recall_facts() for those) that clarifies it should actively shape tone and priority, not just inform. The profile is capped at 4,800 characters (~1,200 tokens) before injection — if it has grown beyond that, it is truncated with a [...profile truncated for context] notice. This is fail-safe: if storage is down or the context table is empty, the agent still runs — just without the profile.

Triggering reflection manually¶

The nightly loop is what runs in production. POST /reflect is an ops-only manual trigger over the same ReflectionService.run():

curl -X POST https://your-app.railway.app/reflect \
  -H "X-Reflection-Secret: your-secret"

The response includes facts_added, facts_updated, intentions_added, and learnings_added — counts of new records saved that cycle.

Semantic Search¶

Beyond keyword matching, Kwasi can find content by meaning using vector embeddings. When a note, interaction, or saved article is written to storage, an embedding is generated in the background via embed_text() (app/tools/embedding.py) using Gemini gemini-embedding-001 (3072 dimensions). If the embedding API is unavailable, the record is saved without an embedding — keyword search still works.

The `semantic_search` tool¶

Available on memory_agent, briefing_agent, and the full agent.

semantic_search(query: str, sources: list[str] | None = None, limit: int = 5)
# sources: "notes" | "interactions" | "read_later" — default: all three

Returns results grouped by source with a similarity percentage. Requires GOOGLE_API_KEY.

Recency weighting¶

find_relevant_notes(), find_relevant_summaries(), and the semantic context injection layer apply a small recency boost when ranking results. Notes written within the last 90 days receive up to +0.05 added to their cosine similarity score — so a note from last week at 0.78 similarity scores higher than an equivalent note from 18 months ago at 0.78. The boost decays linearly to 0 at 90 days. Notes without a created_at timestamp get no boost and sort by raw similarity only.

This means equally-relevant context from recent conversations surfaces first without entirely suppressing older material.

Where semantic search is used¶

Scenario	Tool
"Find anything about my career plans"	`semantic_search` (all sources)
"That conversation where I mentioned burnout"	`semantic_search(sources=["interactions"])` as fallback after `search_history`
"Find that article I saved about AI regulation"	`semantic_search(sources=["read_later"])` — primary, no keyword search exists for read_later
"Find anything about X" (unknown source)	`find_everything` — runs keyword + semantic in parallel
Note/task not found by keyword	`semantic_search` as automatic fallback

StoragePort also exposes hybrid_search() — runs a keyword search and a dense vector search independently, then fuses the two ranked lists with Reciprocal Rank Fusion (RRF, k=60), so items found by both rank highest. The similarity field on those results holds the RRF score, not a cosine value.

Backfilling existing data¶

Rows written before semantic search was added have embedding = NULL. To embed them:

curl -X POST https://your-app.railway.app/embed-backfill \
  -H "X-Reflection-Secret: your-secret"
# Returns: {"notes": 12, "interactions": 847, "read_later": 5}

Rate-limited to ~10 rows/second. For large histories, this may take several minutes — the endpoint processes synchronously.

Information Priority¶

When the user asks about themselves, the agent checks in this order:

1. Permanent facts (user_facts table — always in system prompt)
        ↓ not found
2. Reflection profile (UserContext — always in system prompt)
        ↓ not found
3. recall_facts(query) — active search over user_facts
        ↓ not found
4. search_history(query) — keyword search over past conversations
        ↓ not found
5. semantic_search(query) — meaning-based search over notes, interactions, saved articles
        ↓ not found
6. Acknowledge it's not known — offer to remember it now

What Gets Logged¶

Every interaction (from any handler) is saved via log_interaction():

Field	Content
`user_message`	The text sent to the agent (or transcript for voice)
`agent_response`	The full response returned
`tools_used`	JSON array of tool call records extracted from the agent result
`channel`	`"telegram"`, `"cli"`, or `"whatsapp"`
`user_id`	User ID as string (Telegram user ID, WhatsApp phone number, or `None` for CLI)
`created_at`	UTC timestamp

Logging is fire-and-forget — a failure never blocks the response.

Bootstrap: Seeding Initial Facts¶

After deploying, seed the facts store by telling Kwasi key facts in one message:

"Remember these facts about me: - My home address is 124 Avenue Perretti, Neuilly-sur-Seine - I work at [company] at [address] - My partner is [name] - I prefer public transport"

Kwasi will call remember_fact for each one. From that point, every prompt includes them permanently — no conversation history needed, no reflection cycle required.