What is Prompt Caching?
Reusing priced prefixes of a prompt across requests so repeated system context bills at the reduced cached rate. Long builder sessions benefit most, since stable instructions and history prefixes repeat every turn.
Example
BYOB model configs price cached input far below live input, with a small write cost to store the prefix — sessions that keep system prompts stable pay the cheap rate on every repeat turn.
What people get wrong
Expecting caching to help wildly varying prompts. Only stable shared prefixes hit the cache; rewriting instructions each turn pays full price plus write costs.
Frequently asked questions
What breaks the cache most often?
Reordering content, appending timestamps to the prefix, or swapping models — all three turn a cache hit into a full-price call plus a fresh write.
Related terms
Token Metering
Per-leg usage accounting that turns raw token counts into billable credits using each model price table plus markup. BYOB meters input, output, cached, and thinking tokens separately, with a minimum charge flooring tiny calls.
Context Threshold
The input-size line where a provider switches a model to higher long-context pricing. BYOB model configs store the threshold with above-threshold rates, so sessions crossing it pay more per token without changing models.
Model Context Window
How much conversation, code, and history a model can consider at once. Long builds need compaction and stable model selection so decisions survive across sessions.
System Prompt
The persistent instruction block prepended to every request that sets role, rules, and output format. It outranks user text in well-built stacks, which is why builders version it like code, not copy.
Temperature
Sampling randomness for generation: low values make output focused and repeatable, high values make it varied and surprising. BYOB chat configs default near 0.5 while tool calls use 0.3 for steadier structured actions.
Thinking Budget
The token allowance reserved for internal model reasoning before answering. Larger budgets suit hard multi-step problems; BYOB prices thinking tokens separately so runaway reasoning shows up in metering.