Prompt budget

Prompt budget & cache stability#

The tool loop re-sends the whole system instruction and every tool declaration on every step (loop/tool_loop.ts), and Gemini's implicit cache matches a token PREFIX that starts at the system instruction. Two rules follow, and they are the reason a small request ("change the hero text") is cheap:

The composition contract (stated in full at the top of src/rules/compose.ts). build<Agent>System(ctx) may depend only on (sid, locales, design.version, mode flags). Anything volatile in there throws the cached prefix — tool declarations and conversation history included — away the moment it changes, which for an editor agent is every turn it does its job. Everything else is:

• a derived, bounded fact line — designVocabularyBlock() gives the token vocabulary (TOKEN_VAR_DOC + roles/fonts/scheme read off the CSS) instead of the stylesheet; the template outline gives sections + bindings instead of the markup;
• a per-turn block on the user content — buildWebBuilderTurnContext() (document fields/data/outline) and the code agent's focus file. The web-builder collapses its earlier blocks by their marker on replay, so stale document data neither misleads the model nor grows the session;
• or behind a read tool — readWebTemplate (the source, when a layout change actually needs it; writeWebTemplate is refused until it has been called this turn), readStoreDocument.

The rule budget (src/rules/rule_index.ts). A fragment is inlined only when most turns need it and it is small; the rest are named in a ~1 KB RULES INDEX and fetched with readRule. Deferring costs one loop step, so it wins only while T > ~3,700 × p/(1−p) tokens — defer forms (2.2k tokens, <5% of turns), inline schema-fields (1k, most turns). Never slim below the implicit -cache floor: a prefix under it is never cached at all, which costs far more than it saves.

Tool declarations are slimmed for agents (tools/slim.ts) — the shared declarations carry MCP-grade prose because there the description is the client's only briefing; an agent already has that material in its prompt. slimDeclarations() returns copies, so the MCP server keeps the full text.

Mechanised check: src/rules/compose.test.ts asserts buildCodeAgentSystem is byte-identical for contexts that differ only in focus / contexts / surface / capabilities (the per-turn blocks — capabilities, the user's role + capability map, renders as the Your user: line right after the Surface: line, so the model refuses a write the role cannot make before spending the turn's reads on it), and that no surface wording leaked into the prompt. The live signal is usage.steps[0].cached / usage.steps[0].in on the second turn of a session — the first step's cache share, which ≈ 1 when the previous turn's prefix was reused (metadata.firstStepCached on generative/{sid}/log/* records it per turn). The former .dev/prompt-budget.mjs harness no longer exists.