RTK reports token savings, but our cost benchmarks disagree

What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.

Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.

You can find the project here: https://github.com/ory/lumen

  rtk command-that-prints-100k-tokens | tail -5

Worse, since rtk defaults to persisting that savings stat, it breaks sandboxing. Prefixing with rtk leads to random auto-mode denials from time to time too (this is independent of disabling savings stat persistence).

Honestly have no idea why anyone who knows the first thing about CLIs would take rtk gain seriously. I guess clueless vibe coders who has hardly ever worked in a terminal before will look at the stat and feel good about it?

That said, rtk is still mildly useful for compressing repeated test run outputs and stuff, but you should only ever use it on whitelisted commands; wrapping everything like they suggest you to do is just stupid.

Here are other cases demonstrating the exact same issues with these kinds of tools:

https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-... https://brandonbarker.me/writing/headroom-fewer-tokens-bigge...

  Average cost per attempt, without → with RTK:
  
  Claude/Fable: $1.72 → $1.64 (~5% cheaper)
  DeepSeek: $0.115 → $0.121 (~5% more expensive)

  Almost all Claude savings came from a single task.
  Excluding it, savings were under 1%.

That's also how I feel about skills/plugins. While some provide important context for specific projects/environments, I am very skeptical about (over)generalized skills like "writing JS tests" or "creating a spec". There are dozens of these skills internally at my company, but I haven't seen a single benchmark that shows any of those are better than just plain, single sentence prompts in a meaningful way (aka statistically significant).

https://mroczek.dev/articles/the-token-compression-illusion-...

Have not benchmarked it, the intent is mostly to save time, not necessarily tokens. I noticed that the models need a lot of toolcalls to ground themselves, and often have trouble with getting an overview.

I tested rtk among these and it was actually a net negative in both CPU time and accuracy, the latter would throw LLMs way off and make it hard to recover. If you are building a coding agent, I'd hard pass on rtk.

   ~ $ time grep Return * 2> /dev/null | wc -l
   966
   grep Return * 2> /dev/null 0.36s user 0.02s system 98% cpu 0.382 total
   wc -l 0.00s user 0.00s system 1% cpu 0.380 total


   ~ $ time rtk grep Return * 2> /dev/null | wc -l
   260
   rtk grep Return * 2> /dev/null 4.10s user 17.10s system 92% cpu 23.008 total
   wc -l 0.00s user 0.00s system 0% cpu 23.007 total

So yeah I am still on the lookout for a credible CLI wrapper, do let me know if you have any in mind.

[1] https://dirac.run/

If the tool won’t be upfront about those things, they are not worth looking into any further. It’s used car salesman strategy.

"Cap large/unknown command output: `COMMAND 2>&1 | head -c 4000`. Never stream full logs, tests, or large files."

I use that instead of RTK. Empirically, I found RTK makes my agents run longer to complete similar tasks.

Ponytail and Caveman seem to help somewhat.

This is not a good thing of course, but I also feel that acting surprised that this is going on is a little unnecessary.

Having said that: most tools are not helpful

It is possible to reduce token usage. It’s just much harder than the basic approach.

If my agent runs a shell command, show my agent the output of that shell command. I don't have a problem with automatically pruning or paginating large outputs, as long as the agent still has some form of access to the original output (e.g., by searching).

But removing verbosity from an 'ls' command? That's ridiculous! If my agent runs `ls -la`, do not drop the owner and date from the output, because THAT'S AN EXPECTED PART OF THE OUTPUT. Nothing should be trying to predict the agent's intent and interfering, unless you want your agent to get confused and fail.

These LLMs were trained on predictable shell behavior, and RTK deliberately subverts the model's expectations. There's no way that isn't degrading the model's capability.