Dream Phases

Background memory consolidation in three phases — REM, Deep, Lucid. Together they turn raw conversation into curated long-term knowledge without any single LLM call doing too much.

Why three phases?

When you chat with an agent, useful facts get said in passing. „I moved to Berlin." „My new car is a hybrid." „We decided to ship the dark-mode toggle next sprint." A naive memory layer would either dump every turn into long-term storage (noise) or rely on the agent to manually save (unreliable + tedious). somora splits the work:

1. REM notices what's new in a recent session and proposes facts to keep — small per-agent jobs, you approve each one. Result lands in the agent's private memory inbox.
2. Deep rolls multiple inbox entries up into the shared wiki pages — bigger consolidation runs across all agents, every ~12 h. Auto-applies (you'd be drowning in approvals otherwise) but the wiki stays editable in Obsidian.
3. Lucid audits the wiki for contradictions, dead refs, missing pages and links worth adding — every ~7 d. Findings come back as proposals you walk through with the agent in a dream_review loop.

Each phase has its own LLM worker (small/cheap for REM, strong/expensive for Deep+Lucid), its own cadence, and its own approval semantics. No single prompt has to do everything; the system just keeps grinding incrementally in the background while you use somora normally.

The mental model

   ┌─ REM ─────────────────┐    ┌─ Deep ─────────────────┐    ┌─ Lucid ──────────────┐
   │  Session → Memory     │    │  Memory → Wiki         │    │  Wiki cleanup        │
   │  per-agent            │    │  platform-wide         │    │  platform-wide       │
   │  ~30 min idle         │    │  ~12 h scheduled       │    │  ~7 d scheduled      │
   │  small/local model    │    │  strong model          │    │  strong model        │
   │  approval required    │    │  auto-applies          │    │  approval required   │
   └───────────────────────┘    └────────────────────────┘    └──────────────────────┘
                │                            │                            │
                ▼                            ▼                            ▼
   memory inbox grows           wiki gets new pages /         wiki gets fixed:
   with new facts the           merges of new content;        contradictions resolved,
   agent should keep            consumed memory files         dead links repaired,  
                                are deleted                   missing pages created

Each phase has one job. Each runs at its own cadence. Each uses an LLM worker model you configure separately. The phases compose into a clean flow: facts come in via REM, get consolidated by Deep, get audited by Lucid.

Phase REM — Session → Memory

REM ("Rapid Eye Movement") extracts new factual information from a session's transcript and proposes additions to the agent's private memory inbox.

Job

Read the session JSONL since the last REM run, identify FACTS the user asserted (not jokes, not transient state, not things already in memory or the wiki), and surface them as pending findings for your approval.

Triggers

Three triggers, all per-agent:

Manual: /reset YES — when you reset a session, somora archives the current main.jsonl to a timestamped name and starts a fresh main. If REM is enabled for the agent, it spawns a manual run over the part of the archive REM has not read yet, and marks the archive as read when that run succeeds — so the idle worker does not read it a second time. Result lands in ~/.somora/agents/<name>/memory/.dreams/<id>.dream.md and shows up in dream_list.

Automatic: idle-timer — when REM is enabled, an in-process worker watches each agent's chat activity. After idleMinutes of no chat (default 30), the worker:

1. Resumes any previously-paused dream first (don't waste prior work).
2. Otherwise picks the most-recently-active session whose last activity is past its dreamReadThroughTs marker. Archived sessions count: filing a conversation away says you are done with it, not that it should be forgotten, so a session archived before REM caught up still gets its last stretch read.
3. Runs an extraction over the delta range — minus any Lucid review loop inside it (dream_review start … end): facts the user clarifies there are written to the wiki directly, so REM skips that window instead of re-extracting them as duplicate findings.
4. On success, bumps the marker — and goes straight on to the next unread session, up to eight in one cycle. A backlog is read in one stretch of silence instead of one session per idle interval. The cycle ends at the first run that does not succeed (the worker is probably still down), and a session whose last attempt failed is picked last, so it cannot hold up the others.

Catch up now: dream_run({phase: 'rem'}) or POST /agents/:agent/dream/run-rem — starts that same cycle at once instead of waiting for the idle timer, e.g. after a worker outage. Answers started, busy or nothing_to_do.

It comes back on its own. When a cycle ends with work still left — a run that failed, a paused dream, a session nobody has read yet — the worker schedules itself again rather than waiting for you to chat. Four attempts at one, two, four and eight idle intervals, then it goes quiet until real activity restarts the count. A backend that is down for an hour is caught; one that is down all afternoon does not become an all-afternoon retry loop. A successful run resets the count, so draining a backlog of several sessions is not charged against it.

A failed range keeps its record until a run actually reads it. A successful run only clears the failures it covered, so a shorter manual run cannot make the evidence of an unread stretch disappear.

Manual REM runs do not pause when you start chatting again — they're user-initiated, bounded, just run to completion. Automatic REM runs abort on user activity (the in-flight extraction gets paused; resumed on next idle).

Worker model

Configured per-agent in agent.yaml:

rem:
  enabled: true
  model: <alias>             # alias from config.yaml or 'provider/modelId'
  # fallback: <alias>        # optional backup worker — or [a, b], tried in turn
  idleMinutes: 30
  chunkTokens: 50000         # range-split for very long sessions
  chunkTimeoutMs: 600000     # 10 min per chunk (room for local models)
  participate_in_wiki: true  # default true; false = REM only, never Deep
  # thinking: medium         # optional; off | low | medium | high
                             # only honored if the REM worker model has the
                             # 'reasoning' capability — otherwise dormant.

model is required when REM is enabled: there is no implicit default and REM never falls back to the agent's chat model, so a run can never silently land on an expensive model. A small local model (30B class) is good enough for atomic-fact extraction with the right prompt, and REM runs often, so cost matters. A strong hosted model works too.

rem.fallback — a backup worker, or a chain of them. A local worker is away whenever its box switches profiles, benchmarks or reboots. With fallback: set to a second model (typically a cheap hosted one on a different provider) — or to an ordered list, fallback: [glm, deep41flash], like the chat fallback — a run whose current worker is unreachable — connection refused, 5xx, timeout, 429 after the SDK's retries — continues on the next backup from the chunk that hit the outage; chunks already finished are kept, the failed chunk is retried on the backup, and the backup stays in charge until it is unreachable itself, when the next one in the chain takes over. When the chain is exhausted the chunk fails as it would without a backup. A worker that is currently marked unreachable (the note every cascade shares, fallback.retryUnavailableMinutes, default 60 min) is not tried: the run starts on the first backup that is not marked, recorded in the dream file as a switch at chunk 0. What does NOT switch: a 4xx rejection (bad parameter, unsupported reasoning level, auth) — that is a config problem and the dream fails visibly; and a run paused by user activity — that pauses as before. The agent's chat model:/fallback: are never used implicitly: REM only ever runs on workers you named. Both refs are validated when the run starts, so a typo in fallback: fails the dream immediately rather than on the day the primary is down. The dream file records worker_fallback_ref and, when the switch happened, worker_switch (from, to, reason, at_chunk); the Markdown body shows the same line, and the log carries dream.worker_fallback.

What REM sees

Each REM run feeds the worker:

• Transcript chunk (1+ chunks if session is long)
• Existing memory inbox contents for this agent
• Wiki context (index.md + top-N relevant wiki pages, embedding-matched against session content)
• Vault-recall snippets (if vault is configured)
• The REM system prompt (in src/dream/rem-extract.ts)

The transcript says who wrote each message. Not every user_message is the person: another agent's agent_ask, a sentinel trigger's prompt, a tmux or job wake arrive the same way. They are labelled OTHER-AGENT(<name>) and SYSTEM(<kind>); only USER lines (a voice consult counts, it is the person relayed) are the person speaking. A stable fact another agent passed on may still become a finding — often it is the only way an orchestrated agent learns it — but it is recorded with its source ("Laut <name>: …"), never as something the user said. Trigger text is never a source of facts.

The wiki context is critical: REM dedupes against the wiki, not just memory. New facts that contradict the wiki get surfaced as memory_write so Deep can later merge them in.

Output: findings

A REM run produces pending findings, each with:

• action — memory_write | memory_edit | memory_delete | vault_hint
• slug — kebab-case identifier
• proposed_content — what should land in memory if approved
• reason — why this finding (quotes user's statement)

Approve with dream_apply, reject with dream_dismiss. Memory file is written/edited/deleted only on approval. If a finding was correct but already handled outside the dream flow (you edited the wiki yourself, the agent documented it in chat), close it with dream_dismiss({resolved_manually: true, reason}) — it is then recorded as resolved_manually instead of dismissed, so the history reads "done elsewhere" rather than "rejected".

Mechanical dedup

Small worker models don't reliably act on the "propose only NEW facts" instruction — recurring sessions (weekly sentinels) can produce the same findings run after run. A code-level filter runs after extraction, before findings are stored:

• Exact slug collision with a loaded wiki page → the finding is dropped (logged as dream.rem.dedup_dropped). With an existing memory note it depends on what the note says: if the note already carries the finding's content it is a repeat and dropped; if it says something else — the worker reuses the obvious slug when the person corrects a fact — the finding is kept under a slug of its own (<slug>-update-<date>, logged as dream.rem.dedup_reslugged), so the correction reaches the review and nothing is overwritten.
• High content similarity (hybrid search against memory + wiki) → the finding is kept but marked likely_duplicate with a duplicate_of pointer plus a matched_excerpt showing the text the similarity actually fired on. Wiki audit logs (logs/…, written by Deep) are excluded from the comparison corpus — they record that something was processed, not the knowledge itself.

The flag marks topic overlap, not fact identity. Embedding similarity cannot tell "waiting for the GO" from "GO was given" — on a fast-moving project every finding scores high against the project's own wiki page, including genuinely new state. Two guardrails:

• When the finding carries concrete tokens (numbers, dates, versions) that appear nowhere in the matched text, it is additionally marked novel_details: true and the review output says so explicitly — do not batch-dismiss those; they are usually a new fact on a known topic.
• For everything else, glance at the matched_excerpt before dismissing. Batch-dismissal is safe when the excerpt already states the finding's fact, not merely its topic.

Platform-wide tunables in config.yaml (defaults apply when the block is absent):

rem:
  dedup:
    enabled: true             # default true
    similarityThreshold: 0.85 # cosine similarity of the finding's
                              # embedding to the closest existing
                              # memory/wiki chunk, 0..1. NOT the fused
                              # search score (that is a rank within one
                              # query — the top hit is always 1.0 before
                              # boosts). Paraphrases of the same fact
                              # ~0.85-0.95, same topic / different fact
                              # ~0.6-0.8. Lower = mark more.

Coverage judge

Similarity finds topic overlap; it cannot say whether the fact is already written. On a hundred hand-reviewed findings the cosine flag caught five real duplicates. The coverage judge asks a model instead: it reads the whole pages of the closest candidates (memory notes and wiki pages the similarity search returned) and answers one question — is every substantive element of this finding already stated there (covered), or does it add at least one fact, number, date, decision or conclusion none of them state (adds_new)? On the same hundred findings the question caught 93, with 91 % of its marks right.

What the verdict does — nothing is deleted, the finding stays in the review either way:

• covered at or above minConfidence → the finding is marked likely_duplicate like a similarity hit, duplicate_of names the page as <source>:<slug>@judge (when similarity had already marked it, its pointer and number stay), and matched_excerpt shows the paragraph of that page that carries the finding's numbers or dates.
• adds_new → no mark; if similarity had marked the finding, it is additionally flagged novel_details, the review's "do not batch-dismiss" signal.
• Every judged finding carries judge_verdict, judge_confidence (0–100), judge_reason (the model's one sentence, in English like every model-facing text somora produces) and, for covered, judge_by. dream_list / dream_get show them beside matched_excerpt.

A finding without candidates is not judged; a call that fails or replies unreadably leaves the finding without a verdict (logged as dream.rem.judge_failed / judge_unreadable); past maxPerRun the rest of the run goes unjudged and the log says so once. The run's dream.rem.dedup_summary line carries judged, judge_marked, judge_cleared, judge_failed, judge_skipped and judge_model.

Off by default. The model is the REM worker of the agent whose dream is running unless model names another one — a smaller, cheaper model does fine, the question is easier than the extraction. A named model is validated when the run starts, like rem.fallback; without one the judge follows the worker, so when a run switched to rem.fallback the judge asks the fallback too. Cost: one call per finding with candidates, a few thousand input tokens each (the pages), so on a hosted model keep maxPerRun in mind.

rem:
  dedup:
    judge:
      enabled: false          # default false
      # model: <alias>        # default: the agent's REM worker (rem.model)
      candidates: 4           # distinct pages the judge reads per finding
      maxPageChars: 6000      # each page is cut to this
      minConfidence: 80       # a `covered` verdict below this marks nothing
      maxPerRun: 60           # findings judged per run; the rest go unjudged
      timeoutMs: 120000
      # thinking: low         # optional; honoured when the model can reason

Phase Deep — Memory → Wiki

Deep consolidates all agents' memory inboxes into the shared wiki.

Job

For each memory file across all participating agents, decide one of three outcomes:

• Skip — transient (daily log, scratchpad), already covered, too thin.
• Promote — new wiki topic, create dedicated page with frontmatter, sections, cross-references.
• Merge — topic exists in the wiki, integrate new content into the existing page (preserves structure, updates ## Aktueller Stand and ## Zeitleiste typically).

After successful Promote or Merge, the source memory file is deleted. Wiki = canonical, inbox stays a clean queue.

Triggers

Schedule — every 12h (configurable).

Manual — dream_run({phase: 'deep'}) from any agent's chat, or POST /dream/run-deep over HTTP. Optional force: true bypasses the hash-cache (re-evaluates every memory file).

Deep operates on whatever memory files are currently on disk. It does not trigger REM. New observations only enter memory when you approve a REM finding via dream_apply; Deep picks them up on its next run.

Worker model

Configured platform-wide in config.yaml:

wiki:
  deep:
    enabled: true
    intervalHours: 12
    model: <alias>           # required — alias from config.yaml or 'provider/modelId'
    # thinking: medium       # optional; per-engine reasoning_effort. Honored
                             # when the worker model has the 'reasoning'
                             # capability. On a strong model this tends to
                             # improve skip/promote/merge judgement at the
                             # cost of more rate-limit budget.

Without model Deep does not run: the scheduler logs dream.deep.no_worker_model_configured and returns without touching anything.

Single-prompt logic (skip / promote / merge in one call)

Per memory file, Deep does ONE LLM call that decides skip/promote/merge. The worker sees:

• The memory file's body, and when its content was stated: a note that came from a REM finding carries stated_at (the end of the conversation it was read from) — not the day the finding was applied, which for a conversation dreamed late can be months off
• The wiki map: every folder with what kind of page lives in it and how many, the template folders not created yet, and the filing rules (see wiki.md)
• Top-8 relevant wiki pages, embedding-matched against the memory body. A page over 8000 characters arrives shortened to its first 6000 — good enough to decide with, not to rewrite from: a merge into a page the worker saw shortened, or did not see at all, is asked again with that one page in full, and the write is refused if the page changed while the worker was deciding

When the memory contradicts a fact on the page, Deep compares dates before touching it: a memory that is clearly newer updates the fact and notes the revision in the timeline; one that is older than the page's statement — or whose order cannot be told — leaves the current fact alone and is added to the timeline as a dated earlier entry.

Returns a structured MemoryFateDecision:

{ "kind": "skip", "reason": "..." }

{ "kind": "promote", "subfolder": "personen", "slug": "personen/jane-doe",
  "type": "person", "title": "Jane Doe",
  "body": "## Aktueller Stand\n…", "related": [...],
  "newFolder": { "path": "…", "purpose": "…" } }   // only for a folder the map lacks

{ "kind": "merge", "wikiPath": "personen/familie-klein",
  "body": "...full updated body...",
  "logSummary": "familie-klein aktualisiert: ..." }

Deep applies the decision verbatim — after three checks against the wiki. A page over wiki.deep.maxPageChars (default 50 000) takes no more content: instead of merging, Deep is asked once more to write the note as a sub-page under it (projekte/somora/traum-pipeline, its own current state and timeline) — the migration had folded 97 reports into one 100 KB page, and pages like that are never to grow again. Then two checks against the wiki map. A promote into a folder that neither exists nor is proposed by the template is refused unless the model gave the folder a purpose (then the folder is created and described in the structure file); a folder deeper than one subfolder is refused. And the page name is checked across all folders: when a page of that name already exists elsewhere, the promote becomes a merge into that page, asked again with the page in full. No second LLM call otherwise — with one exception, the merge guard below.

Merge guard (anti-clobber)

The merge decision asks the worker for the full updated page body, and Deep writes that body as the new page. On large pages this is a real failure mode: instead of integrating the new fact, a model may return a summary of the page — and the page shrinks from 22 KB to 3 KB with the old content gone. Deep auto-applies, so nothing catches it in between.

Before writing, Deep therefore compares body sizes. When the new body is shorter than minRatio × the existing body, the merge is refused:

• the wiki page stays untouched,
• the source memory file is not deleted — it survives for the next run, so the content still exists in two places rather than none,
• a dream.deep.merge_shrink_blocked warning is logged with both sizes,
• the candidate is not written to the skip-cache, so the next Deep run retries it with a fresh LLM call.

wiki:
  deep:
    mergeShrinkGuard:
      enabled: true            # default
      minRatio: 0.5            # refuse when newBody < 0.5 × existingBody
      minExistingBytes: 2000   # pages below this are never guarded

minExistingBytes exempts small pages: shrinking a 400-byte stub is normal editing, and guarding it would only stall consolidation.

If a specific page trips the guard repeatedly, that is the signal that it has outgrown full-body merges — split it into sub-pages.

Hash-cache

Skipped memory files get cached by body-hash in ~/.somora/agents/<name>/memory/.deep-skip-cache.json. On the next Deep run, files whose hash matches the cached entry are skipped without an LLM call. When most memories are unchanged between runs, a Deep run costs no worker tokens at all.

The cache invalidates automatically when:

• Memory body changes (hash mismatch → re-evaluate)
• Promote/merge consumed the file (entry pruned)
• File is deleted (opportunistic cleanup on next loadCache)
• The skip is older than wiki.deep.skipCacheDays (default 30; 0 = never). A skip is a verdict against the wiki of that day — "too thin for a page of its own" stops being true once the page exists, and the note belongs on it. Expiry is spread over up to a quarter of the period per note, and at most 10 expired notes per agent are looked at again in one run, so a batch skipped on one day does not come back as one expensive run. Skipped again, a note rests for another period.

dream_run({phase: 'deep', force: true}) ignores the cache for one run.

No approval

Deep runs auto-apply. No per-finding review — the trade-off is that Memory→Wiki is mechanical consolidation, not subjective. If a Deep run makes a bad decision you don't like, you fix it in Obsidian (the wiki is just markdown) or wait for Lucid to flag it.

The audit trail lives in <vault>/<wiki-subfolder>/logs/YYYY-MM.md — monthly append-only logs of all Promotes and Merges with one-line summaries.

Phase Lucid — Wiki cleanup

Lucid audits the existing wiki, surfaces objectively-verifiable quality issues, and hands them off to a conversational review loop where you walk the findings with one of your agents and that agent writes the changes via loop-scoped wiki_* tools. Lucid itself never auto-edits the wiki.

Job

Walk the wiki (subfolder by subfolder), identify issues that are objectively provable from the wiki content, surface a SHORT list (max 8 per batch — one batch per subfolder plus the cross pass). Lucid is intentionally narrow:

Finding kind	What it means
contradiction	Two pages assert mutually exclusive facts about the same subject. Cite specific text from each.
duplicate_page	Two pages describe the same thing under different names — the migration unites only same-name pages. The page that should survive comes first; dream_apply unites them (the model writes the merged body, the other page goes to the report archive, links follow).
misfiled_page	A page plainly not of its folder's kind, judged against the wiki map. Only in a wiki on the folder template; the finding names the folder, dream_apply moves the page there.
oversized_page	A page over wiki.lucid.oversizedChars (default 50 000). Found without a model, only in a wiki on the template; the review can split it into sub-pages (wiki_create + wiki_edit + wiki_move).
not_migrated	One per run in a wiki without the template: Lucid checks content only here, somora wiki migrate is available. Dismiss it once to keep the wiki as it is.
dead_ref	[[wiki-path]] references a page that doesn't exist.
wanted_page	Topic referenced by ≥3 wiki pages but missing its own page.
link_suggestion (set by Lucid itself, see below)	A page mentions a named entity in prose AND a wiki page exists with that name AND there is no [[wikilink]] from one to the other. Strict: only for clearly identifiable named entities, not generic words.

Subjective polish (stylistic rewrites, "this could read better", "feels old") is deliberately NOT in scope. Those decisions belong in the review conversation, not pre-baked as findings. Three further kinds (stale_claim, outdated, inconsistent_xref) are accepted when a run file is read, so archived runs in processed/ still parse, but no run produces them.

Triggers

Schedule — every 7 days (configurable).

Manual — dream_run({phase: 'lucid'}) or POST /dream/run-lucid.

Cluster strategy

Lucid cuts the wiki into calls by size: one call carries as many pages of one folder (subfolders are folders of their own) as fit wiki.lucid.batchChars (default 100 000 characters); a page larger than that travels alone; a big folder goes in numbered parts. A grown wiki with seventy small folders gets seventy small calls, a wiki on the template with three big folders gets those in parts — the same rule for both. Beside its pages every call sees the wiki map (the folders and what lives in each) and the other pages of its top folder as one line each, not the whole index.md. Then a final cross-folder pass looks across folders — but it sees only the opening line of each page. That is enough for a dead link or a missing page that spans folders; a contradiction between the body of personen/jane-doe and the body of projekte/familie-leo-podcast is out of its sight.

The size limit is not only for scale — claude-cli's stdin-stream parser fails on very large single user-messages. As a side-effect the worker reads each part with full focus, which produces higher-quality findings than scanning everything at once.

Worker model

Configured platform-wide:

wiki:
  lucid:
    enabled: true
    intervalDays: 7
    model: <alias>           # required — alias from config.yaml or 'provider/modelId'
    requireApproval: true
    maxCallsPerTurn: 3       # wiki_* calls the loop holder may make per turn
    batchChars: 100000       # page text per Lucid call; a bigger folder goes in parts
    oversizedChars: 50000    # pages above this are reported (template wikis only)
    maxFindings: 12          # kept per run for review, weighted: contradictions, dead refs,
                             # duplicates, misfiled/oversized, wanted pages
    autoLinks: true          # link suggestions are set by Lucid, not reviewed
    autoLinksPerRun: 30
    seenDays: 90             # a dismissed finding is not filed again for this long
    # thinking: medium       # optional; same semantics as wiki.deep.thinking.
                             # Lucid is judgement-heavy (consistency + dead-
                             # ref detection) so medium thinking is often
                             # the sweet spot on cost/quality.

Without model a Lucid run fails with no worker model configured for lucid.

Output

A LucidRun JSON file in ~/.somora/wiki-lucid/<run-id>.json. The cap of 8 findings applies to each call (one per batch, plus the cross pass); a run over ninety batches could return hundreds, so wiki.lucid.maxFindings (default 60) keeps the weighty ones — contradictions, dead refs, duplicates, misfiled and oversized pages, wanted pages; a pair filed by two calls counts once; what was dropped is logged (dream.lucid.run_capped). Link suggestions are not reviewed at all: Lucid sets them itself — the first plain mention of the name becomes a [[link]] — up to wiki.lucid.autoLinksPerRun (30) per run, and records each as applied or, when the phrase was not found as plain text, dismissed with the reason (autoLinks: false turns this off). A finding a person dismissed is not filed again for wiki.lucid.seenDays (90). And while a run still has findings waiting, no new run starts — the scheduler looks again six hours later, dream_run({phase:'lucid', force:true}) runs anyway. The file also records batches_total and batches_failed: a run in which every batch failed, or that was aborted, is failed — not a clean wiki with zero findings. Each finding is informational only — fix.kind: 'no_op' with the description of the issue. The actual editing happens in a dream_review loop (next section), not via dream_apply.

When the review loop closes (or you dismiss the whole run), the file moves to ~/.somora/wiki-lucid/processed/ with the loop summary appended for the audit trail.

Reviewing Lucid findings — the dream_review loop

Findings are walked by an agent in conversation with you, not via button-click approval. You start the loop with one agent, talk through each finding, the agent writes changes via loop-scoped wiki_* tools when you OK each step, and you close the loop when done.

> you:    have a look at the lucid result
scribe:    dream_list   → finds the Lucid run
         dream_get(id) → reads all findings
         dream_review({dream_id, action: 'start'})  ← opens the loop
         "Here is what Lucid found: 5 contradictions, 2 dead refs.
          Starting with the first: page X says Y, page Z says W.
          My proposal: <concrete change>. OK?"

> you:    yes, do it
scribe:    wiki_edit({...})   ← writes the page
         "Done. Next finding: ..."

> [several rounds of walk-discuss-edit]

> you:    good, wrap it up
scribe:    dream_review({dream_id, action: 'end', summary: '...'})  ← closes
         findings you did not get to stay open; the run is archived only
         when none are left (or with dismiss_rest: true); a later loop
         continues where this one stopped, and Lucid runs no new scan
         while findings wait

While the loop is active for an agent:

• The agent gets wiki_edit / wiki_create / wiki_delete / wiki_move (loop-scoped)
• Read-only file tools (file_read, file_search, file_list, analyze_file) stay available so the agent can look up source material before proposing an edit. file_write / file_patch are hidden — only wiki_* may mutate.
• The agent's exec_* / agents_* / skill_* / tmux_* tools are temporarily hidden so the conversation stays focused. Hiding is enforced at the call, not only in the listing: a hidden tool invoked from a definition the model already had is refused.
• CLI engines (claude-cli, codex-cli, grok-cli) receive the tool list once per turn. Starting the loop mid-turn does not make wiki_* callable in that same turn — they appear on the next turn. Practical pattern: dream_review start, end the reply, continue after the user speaks. dream_dismiss with resolved_manually: true closes a finding you resolved by other means at any time.
• Other agents continue normal operation but cannot start their own loop until this one ends — somora-instance-global lock
• The TUI status line shows 📝 wiki-review:<agent>
• Per-turn cap: max 3 wiki_* calls in a single turn (wiki.lucid.maxCallsPerTurn) so the agent cannot batch-edit without checking in. Resets on every user message.
• Auto-expiry: 24h idle without activity → loop auto-closes as a safety net in case the agent forgot to call action: 'end'

wiki_edit accepts body changes (newBody) and/or frontmatter ops (relatedAdd, relatedRemove, sourcesAdd, sourcesRemove) — all optional. Pass only what you want to touch. Useful pattern: clear a dead related: ref with wiki_edit({wikiPath, relatedRemove: ['dead/page']}) without touching the body.

Reviewing findings — the dream_* tool surface

REM and Lucid have different review flows because the cost of a wrong auto-apply differs:

• REM findings are atomic memory writes — small, easy to audit individually, isolated to one agent's inbox. They use the per-finding approval pattern (dream_apply / dream_dismiss).
• Lucid findings are wiki edits that ripple across multiple pages and need the user's judgement. They use the conversational dream_review loop described above.

Tools:

dream_list([include_processed])         List pending dreams (REM + Lucid);
                                        include_processed:true adds resolved ones
dream_get(dream_id)                     Show full content of a dream
dream_apply(dream_id, finding_id)       Accept REM finding → applied
dream_dismiss(dream_id, [finding_id],   Reject (one finding or whole run);
              [reason],                 resolved_manually:true records
              [resolved_manually])      "handled outside the dream flow"
                                        instead of "rejected"
dream_run({phase, [wait], [force]})     Trigger Deep or Lucid manually
dream_review({dream_id, action, [summary]})
                                        Open/close the wiki review loop
                                        for a Lucid run (Lucid only)

dream_list returns a kind discriminator per entry:

• kind: 'memory' — REM finding, scoped to the agent in whose chat you're calling. Other agents don't see it.
• kind: 'wiki_lucid' — Lucid finding, platform-wide. Any agent with the dream toolset sees the same set.

Typical REM flow:

scribe> dream_list
   → memory dream: 5 pending findings
scribe> dream_get dream_id=<id>
   → full finding list with action, slug, reason, proposed_content
> walk through with me, finding by finding
scribe> dream_apply  (or dream_dismiss)
   → repeats until all resolved → dream_done: true

Typical Lucid flow:

scribe> dream_list
   → wiki_lucid run: 6 pending findings
scribe> dream_get dream_id=<id>
   → full finding list (all fix.kind = 'no_op' informational)
scribe> dream_review({dream_id, action: 'start'})
   → loop opens, wiki_* tools become available, file_*/exec_* etc. hide
[walk-discuss-edit conversation rounds]
scribe> dream_review({dream_id, action: 'end', summary: 'F1 applied as edit X, F2 dismissed, ...'})
   → loop closes, run archived

dream_apply on a Lucid finding (always no_op) marks it applied without writing anything — useful only as acknowledge-and-move-on if you don't want the loop. The actual fix path is the loop.

Triggering manually

REM is per-agent: /reset YES reads the session it archives, dream_run({phase: 'rem'}) (or POST /agents/:agent/dream/run-rem) catches up every unread session of the calling agent, and the idle timer does the same on its own.

Deep and Lucid are platform-wide:

# In any agent's chat:
> ruf bitte dream_run({phase: 'deep'}) auf
> dream_run({phase: 'lucid'})
> dream_run({phase: 'deep', force: true})  # bypass hash-cache

# Or HTTP directly:
curl -X POST http://127.0.0.1:18737/dream/run-deep   -d '{"wait":true}'
curl -X POST http://127.0.0.1:18737/dream/run-lucid  -d '{"wait":true}'

wait: true blocks the request until the run finishes (returns full outcome). Default wait: false is fire-and-forget — agent gets a "started in background" reply, run completes on its own. force is honoured by run-deep only.

Two read-only routes expose the state: GET /dream-states (per-agent REM activity and pending counts plus the Deep/Lucid running flags) and GET /dream/loop-state (the active review loop, if any).

Configuration cheat-sheet

# config.yaml — platform-wide
wiki:
  enabled: true
  vaultSubfolder: somora                 # → <vault>/somora/
  language: de                           # de | en — scaffolding + prose language
  deep:
    enabled: true
    intervalHours: 12
    model: <alias>                       # required
    # thinking: medium                   # optional; reasoning_effort for the
                                         # Deep worker LLM. Off by default.
    mergeShrinkGuard:
      enabled: true                      # refuse merges that shrink a page
      minRatio: 0.5                      # newBody < 0.5 × existingBody → skip
      minExistingBytes: 2000             # smaller pages are never guarded
    skipCacheDays: 30                    # a cached skip of an unchanged note is
                                         # looked at again after this; 0 = never
  lucid:
    enabled: true
    intervalDays: 7
    model: <alias>                       # required
    requireApproval: true
    maxCallsPerTurn: 3
    # thinking: medium                   # optional; reasoning_effort for the
                                         # Lucid worker LLM. Off by default.
  search:
    boostWiki: 1.4                       # wiki hits rank above memory
    boostMemory: 0.85
    boostVault: 0.65
rem:
  dedup:
    enabled: true                        # mechanical dedup after extraction
    similarityThreshold: 0.85
    judge:
      enabled: false                     # coverage judge (see REM → Coverage judge)
      # model: <alias>                   # default: the agent's REM worker
      candidates: 4
      maxPageChars: 6000
      minConfidence: 80
      maxPerRun: 60

# agent.yaml — per-agent
rem:
  enabled: true
  model: <alias>                         # required when enabled
  # fallback: <alias>                    # optional backup worker
  idleMinutes: 30
  chunkTokens: 50000
  chunkTimeoutMs: 600000
  participate_in_wiki: true
  # thinking: medium                     # optional; dormant unless the REM
                                         # worker model has the 'reasoning'
                                         # capability.

All three thinking fields are optional and unset = engine default (no reasoning_effort sent). See thinking.md for the per-engine mapping table and dormant-state semantics.

Where things live

~/.somora/agents/<agent>/memory/             ← memory inbox (REM writes here)
~/.somora/agents/<agent>/memory/.dreams/     ← REM run files awaiting approval
~/.somora/agents/<agent>/memory/.dreams/processed/   ← resolved REM runs
~/.somora/agents/<agent>/memory/.deep-skip-cache.json ← Deep hash-cache
~/.somora/wiki-lucid/<run-id>.json           ← Lucid run files
~/.somora/wiki-lucid/processed/              ← resolved Lucid runs
<vault>/<wiki-subfolder>/                    ← the wiki itself
<vault>/<wiki-subfolder>/index.md            ← auto-regenerated topology
<vault>/<wiki-subfolder>/logs/YYYY-MM.md     ← monthly Deep audit log

What dreams won't do

• Auto-write to memory or wiki without approval — except Deep, which runs auto-apply (no approval) but is bounded to the structured MemoryFateDecision and writes only the markdown the LLM produced.
• Edit content outside the agent's memory or the wiki subfolder — the rest of your Obsidian vault stays read-only from somora's perspective.
• Sync across machines automatically — somora is single-host. Use git or syncthing on the wiki subfolder if you want cross-device sync (memory inboxes are ephemeral inboxes, not worth syncing).

Design rationale

Each phase has its own worker model so you can tune cost vs quality independently:

• REM runs often → cheap local model is fine for atomic-fact extraction.
• Deep runs occasionally → strong model is worth it for quality consolidation.
• Lucid runs rarely → strong model definitely worth it for finding contradictions.

Each phase has its own approval policy so you control surface area:

• REM proposes; you decide what enters memory.
• Deep auto-applies; mistakes are recoverable in Obsidian.
• Lucid proposes; you decide what gets fixed in the wiki.

The wiki is the only place where stable knowledge lives. The memory inbox is volatile by design — files come in, get consolidated, get deleted. The vault is read-only context (your Obsidian notes outside the wiki subfolder are yours, not somora's).

See also

• memory.md — how the memory inbox indexes and retrieves
• wiki.md — the shared long-term wiki layer
• agents.md — per-agent configuration including REM
• tools.md — full tool reference (dream_* is one toolset among the others)