Skip to content

Prompt cache ​

Model providers remember the beginning of a prompt they have seen before and do not process it again. somora builds every request so that this beginning stays the same from turn to turn. This page explains the order, what breaks it, and which settings matter.

What you get ​

  • Cheaper turns. Cloud providers bill cached input at a fraction of the normal price.
  • Faster first word. A local model skips re-reading the cached part, which saves seconds on a long conversation.
  • Nothing to set up. The order is built in, for every engine.
  • Recall that does not cost the cache. Recalled notes change on every turn and still leave the earlier conversation cached.
  • A number to watch. The chat header shows how much of the last turn's input came from the cache.

Check it ​

Send two messages in the same session and look at the input tokens in the status line, for example Σ↑ 12k+80k¢. The part marked ¢ was read from the cache. From the second turn on it should cover most of the input.

To see the system prompt exactly as the next turn sends it, open the agent's window in the web client and choose Full prompt, or call:

bash
curl -H "Authorization: Bearer $(somora key)" 'http://127.0.0.1:18737/agents/<your-agent>/prompt-preview?session=main'

The preview never changes the session.

The one rule ​

A cache matches from the first character up to the first difference. Everything after a changed character is processed again.

So the order is always the same: what never changes comes first, what changes rarely comes next, and what changes on every turn comes last.

ChangesExampleWhere it goes
Never or rarelypersona, team, wiki overview, skillssystem prompt
Per sessionsession name, pinned projectend of the system prompt
Every turnrecalled notes, the frame of the turnin front of the user message

Order of the system prompt ​

The system prompt is put together from eight parts, in this order:

#PartChanges when
1Self-pointer: who the agent is, its paths and resourcesresources or paths change in the config
2Persona: SOUL.md, AGENTS.md, USER.mdyou edit one of the files
3Team blockteam.yaml or the list of agents changes
4Tool reminderconstant, except the ask_user line, which is there only in a turn someone watches
5Wiki overviewnever within a session
6Skills lista skill is added, removed or its description edited
7This sessionnever within a session
8Projectthe session's project is switched or the project is updated

Parts 1 to 6 are the same for all sessions of one agent, so a provider can reuse them across sessions. The project is last because it is the part most likely to change in the middle of a session.

A sub-agent turn adds one note to part 1. Empty parts are left out: no team, no wiki, no pinned project.

Tool schemas travel in the API's tool channel. On openai-compatible (catalog mode, the default) the tools not sent in full are listed by name after part 8, and on codex-cli the developer instructions end with the names of the deferred tools. Both change only when the set of tools changes.

The wiki overview is frozen ​

The wiki overview is read on the first turn of a session and stored with the session. Later turns reuse the stored text, even after Deep has rewritten the wiki index. A fresh map in the middle of a session would change a block in front of the whole conversation.

Recall is not affected: automatic recall and memory_search always use the live index. A new session, or a reset of the current one, gets a new overview.

Builder agents ​

A builder (kind: builder) has a different system prompt with the same idea: identity and environment, harness rules, compact team, the builder's own rules, repository instructions, skills, session, project.

The environment block names the model and today's date, so a builder's prefix changes when either does. A builder gets no automatic recall block.

Where the per-turn block goes ​

Everything that belongs to one turn only is collected in one block and placed in front of the user's text, never in the system prompt. In order:

  1. the Lucid review block, while a review is open
  2. the frame of the turn: the header of a message from another agent, a sentinel's evidence, the instructions beside a wake-up
  3. the <memory-context> block with the recalled notes

How that block reaches the model depends on the engine.

EngineConversation is kept bySystem prompt is sentThe user message contains
claude-clithe Claude session, resumedevery turn, unchangedcatch-up, per-turn block, your text
codex-clithe Codex thread, resumedas developer instructions on every thread start and resumeattachment notes, per-turn block, project block on a resumed thread, catch-up, your text
grok-clithe Grok session, resumedin the session's agent profile, replaced on every resumeper-turn block, project block on a resumed session, attachments, your text
openai-compatiblesomora, rebuilt from the session file on every requestevery request, as the one system messageper-turn block, your text

"Catch-up" is the summary of turns another engine answered, sent when you switch engines within a session.

claude-cli, codex-cli and grok-cli keep the conversation themselves. somora only sends the new turn, and the provider's cache covers the rest.

On codex-cli and grok-cli a resumed conversation also gets the project block in front of the user message, so the model sees a new pin at once.

The openai-compatible engine ​

This engine has no session on the provider's side. somora sends the whole conversation on every request, and it must come out identical each time.

Recalled notes are stored with the message ​

The per-turn block is saved in the session file next to the message it was sent with, in the field ephemeral of the user_message event:

jsonc
{"kind":"user_message","ts":...,"text":"what you typed","ephemeral":"<memory-context>...</memory-context>"}

When the conversation is rebuilt, each earlier user message gets its own stored block back in front of its text. The request for turn 6 therefore starts with exactly what was sent for turn 5. Only the new message is new.

The price is a larger session file: each user message also stores its block, typically 500 to 2000 characters.

Two things keep the blocks from piling up. A section that is still in the conversation is not injected again (memory.autoInject.skipRepeats). And after a pause longer than the provider's dropMemoryBlocksAfterIdleMinutes, the earlier blocks are dropped from the rebuilt conversation, once. The cache has expired by then, so the one changed prefix costs nothing extra; from the next turn on the cache holds again.

Why not a second system message ​

Putting the recalled notes in a late system message looks cleaner and does not work here. On turn 1 the block sits at position 2. On turn 2 it sits at position 4, and position 2 now holds the first user message. The cache stops matching right after the system prompt.

Some model servers also accept a system message only at the very start. For the same reason the compaction summary is appended to the one system message and is not sent as a second one.

Tool calls in the history ​

Earlier turns that used tools are replayed in the native shape, one assistant message per reply of the model: the text it wrote in that reply with its tool_calls, then one role: tool message per result. The closing message holds only the last reply.

jsonc
{"role":"assistant","content":"I'll write the note:","tool_calls":[{"id":"c1","type":"function",
  "function":{"name":"file_write","arguments":"{\"path\":\"notes.md\"}"}}]}
{"role":"tool","tool_call_id":"c1","content":"{\"ok\":true}"}
{"role":"assistant","content":"Done, the file is written."}

The rules:

  • Each reply keeps its own text next to its own calls. Calls the model made together stay in one message.
  • Each call is followed directly by its result, in order.
  • A call without a recorded result is dropped. A crashed turn leaves such calls, and most providers reject them.
  • Each replayed result is cut to 800 characters (MAX_REPLAYED_TOOL_RESULT_CHARS, a fixed value). The model can call the tool again when it needs the full output.

The native shape matters for more than the cache. A history in which the assistant never calls a tool teaches a weaker model not to call tools.

The same goes for the order of the text: all of a turn's interim text replayed after its calls reads as "tool work is narrated", and a model then narrates instead of calling. Turns recorded before the text was stored per reply therefore come back with their closing reply only, and so does a reply that narrates tool work without having called a tool.

The rebuild is deterministic: the same session file gives the same request, so the cache holds.

What breaks the cache ​

EventEffect
Editing SOUL.md, AGENTS.md or USER.mdEverything after the self-pointer is processed again, once.
Changing team.yaml, adding or removing an agentOnce, from the team block on.
Adding, removing or editing a skillOnce, from the skills list on.
Switching or updating the session's projectOnce, from the project block on. The parts before it stay cached.
Changing the set of tools an agent seesOnce. Tool definitions, and the listing at the end of the system prompt, sit in front of the conversation.
A turn nobody watches (an agent's request, a Sentinel wake) after one a person started, or the reverseFrom the tools on: ask_user and its line in the tool reminder are offered only when someone watches.
A compaction on openai-compatibleOnce. The summary joins the system message and the older messages are dropped.
Switching the model or provider of a sessionThe new model starts with an empty cache.
memoryInjectMode: systemEvery turn. The system prompt changes with each recall.
A long pauseProviders drop a cache after some idle time.

All but the last two are one-time costs. Avoid editing persona files or skills in the middle of a long, expensive session if the cost matters.

The dream worker ​

REM reads a long session in chunks, one model call per chunk. Each call uses the same order: what is the same for the whole run comes first, the chunk comes last.

Agent name: <your-agent>
<existing_memory>...</existing_memory>             same for every chunk
<wiki_index>...</wiki_index>                       same for every chunk
<wiki_relevant_pages>...</wiki_relevant_pages>     same for every chunk
<vault_referenced>...</vault_referenced>           same for every chunk
<transcript>...</transcript>                       differs per chunk

From the second chunk on, everything above the transcript can come from the cache.

Settings ​

All settings live in config.yaml. The values shown are the defaults.

yaml
providers:
  local:
    engine: openai-compatible
    baseUrl: ...
    apiKey: ...
    memoryInjectMode: inline-user
    dropMemoryBlocksAfterIdleMinutes: 60
    models: [...]

agentLoop:
  toolUsageReminder: true

wiki:
  search:
    overviewMaxChars: 4000
    overviewTopNSlugs: 30
SettingDefaultMeaning
providers.<name>.memoryInjectModeinline-userOnly for openai-compatible providers. inline-user puts the per-turn block in front of the user message and keeps the cache. system appends it to the system prompt and loses the cache on every turn.
providers.<name>.dropMemoryBlocksAfterIdleMinutes60Only for openai-compatible providers. After this many minutes without a turn, the earlier memory blocks are dropped from the rebuilt conversation. Match it to the provider's cache lifetime. 0 keeps them.
agentLoop.toolUsageRemindertrueAdds the constant tool reminder to the system prompt when the agent has tools. Switching it changes the prefix once.
wiki.search.overviewMaxChars4000Size limit of the wiki overview. A larger value means a larger constant prefix, paid once per session.
wiki.search.overviewTopNSlugs30Most sections listed when the overview falls back to section names and counts.

Set memoryInjectMode: system only for a model server that mishandles a memory block inside user messages. claude-cli, codex-cli and grok-cli have no such setting: the placement is fixed.

Where the numbers come from ​

EngineField read from the provider
claude-clicache_read_input_tokens
codex-clicachedInputTokens
grok-clicachedReadTokens
openai-compatibleprompt_tokens_details.cached_tokens

somora passes the value on as tokens_in_cached in the usage of a turn. It is a part of tokens_in, not an addition to it.

Troubleshooting ​

The cached part is always zero. Not every provider caches, and not every provider that caches reports it. Some local servers use their cache and still report nothing. Check the provider's own dashboard or compare the time to the first word before you change anything.

The cached part drops to almost nothing on every turn. Look for memoryInjectMode: system on the provider. Then check whether something in the system prompt changes between turns: call the prompt preview twice and compare the two results.

The cached part drops once and recovers. That is one of the one-time events under What breaks the cache.

You changed how prompts are built and want proof. A cached-token number from one response is not enough. Run two turns in one session and compare the two requests message by message. Every position before the new user message must be identical in role and content.

See also ​

  • Memory: the recall block and its settings
  • Wiki: the overview block and how it shrinks for a large wiki
  • Projects: the project block at the end of the system prompt
  • Team: the team block
  • Skills: the skills list
  • Builder: the builder's system prompt in detail
  • Compaction: when older messages are replaced by a summary
  • Models: provider fields, including memoryInjectMode
  • Display: the token figures in the status line
  • API: GET /agents/:agent/prompt-preview and the user_message event