Why AI coding agents slow down in long sessions, and 5 prompt-caching mistakes to avoid
This is a shortened version. Read the full article on my site: https://tabishapps.com/blog/prompt-caching-coding-agents/
If you use Claude Code, Codex or an open model through Ollama, you've probably seen an agent get slower and more expensive as a session goes on. Often the reason isn't the model. It's the prompt cache being thrown away, sometimes by something as small as a timestamp.
This week Ollama shared three lessons on this (https://x.com/ollama/status/2122531964154712541), from a longer write-up by Together AI and Ollama (https://www.together.ai/blog/making-open-models-work-for-long-horizon-agentic-workloads). I read it, then tested the main claim on my own Mac.
Every turn, the agent sends everything again
On every step, the harness sends the model the whole conversation again: the system prompt, every tool definition, every file read and every command run. So for agents, the expensive part is reading the input, not writing the answer.
The one rule of prefix caching
Inference engines keep the work they did on a prompt for a while. If the next request starts with exactly the same tokens, they reuse that work and only process the new part.
The cache matches from the very first token up to the first token that's different, and stops there. Adding text at the end is cheap. Changing anything earlier, even one character, means everything after it is processed again.
The test
I used Apple's MLX and Gemma 3 12B (4-bit) on a MacBook Pro with an M3 Pro chip. The script builds a fake agent session and times how long the model takes to read the prompt for the next turn:
- No cache: read 10,275 tokens, about 2 to 4 minutes before the first word.
- Cache kept, new turn added at the end: read 2,513 tokens, about 1 minute.
- Cache kept, but a new timestamp in the system prompt: only the first 32 tokens matched, so it read 10,244 tokens, about 2 to 4 minutes.
One changed timestamp near the top made the cache almost useless. A cloud model is much faster than a laptop, but the shape is the same.
5 things that break the cache
- Anything in the system prompt that changes each turn, like the time, git status or current folder. Put changing information in a new message at the end.
- Changing reasoning effort mid-session. Some chat templates write it into the first system block. Pick it at the start and leave it.
- Adding tools near the top. Avoid changing the tool list during a session.
- Huge tool results. A file the agent reads is carried in every request after it. Search first, and read line ranges.
- Cutting output without saying so. Add a line like "showing lines 1-100 of 640", so the model knows there's more.
The short version: keep the start of the prompt identical for the whole session, and put anything that changes at the end.
The full article has the complete checklist and the test script you can run yourself: https://tabishapps.com/blog/prompt-caching-coding-agents/
This breakdown of prefix caching behavior in LLM coding agents is extremely practical. Placing dynamic variables like timestamps at the end of the context window rather than the system prompt makes a huge difference for latency.