All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.

What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.

Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.

You can find the project here: https://github.com/ory/lumen

▲ Whitespace 5 hours ago | parent | next [-]

I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?

    Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec.
    Releases v0.0.42 Latest last month

Why should I trust that what you're peddling isn't snakeoil?

▲

aeneas_ory 4 hours ago | parent | next [-]

I literally say you should take benchmarks with a grain of salt :)

> Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.

And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (caveat: it was on older models, Opus 4.6 I believe).

Also I explain WHY it saves tokens - because the model doesn’t have to brute force different terms until it finds the match it needs, but uses semantic „distance“ so the embedding does it for the model.

	▲	lucaprata 4 hours ago \| parent [-]
		[flagged]

▲

icantevenhold 5 hours ago | parent | prev | next [-]

Only way to find out is to do some testing yourself i think.

I’m using less tokens with Lumen but I also use a bunch of other tokens hacks/skills; it’s hard to measure the impact exactly but it feels significant

▲

4 hours ago | parent | prev | next [-]

[deleted]

▲

huflungdung 4 hours ago | parent | prev [-]

[dead]

▲ esperent 5 hours ago | parent | prev | next [-]

This sounds quite similar to dirac which made a stir a few months ago:

https://github.com/dirac-run/dirac

I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.

If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.

	▲	nevon 4 hours ago \| parent [-]
		I did the same with another project that does the same thing, called ck, and wasn't able to wring out any improved performance over just plain grep.

▲ gslepak an hour ago | parent | prev | next [-]

How does lumen compare to semble?

https://github.com/MinishLab/semble

▲ lopatin 3 hours ago | parent | prev | next [-]

I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)

	▲	bunderbunder 3 hours ago \| parent [-]
		I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days. But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity. The caveman gimmick makes sense to me as a clever hack. Caveman talk is a longstanding meme that's presumably well-represented in the models' training data. So just asking it to do that is just an ultra-concise way to tell the LLM to be ultra-concise. Which, in turn, is theoretically good for accuracy because putting too many instructions in the prompt is bad for task performance.

▲ woadwarrior01 an hour ago | parent | prev | next [-]

TBF, what difference does it make when the very harnesses you're using these "hacks" in are themselves vibe-coded?

▲ Bridged7756 5 hours ago | parent | prev | next [-]

Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.

	▲	SJMG 5 hours ago \| parent [-]
		You would think, except in their own testing they found no benefit. Some measures it was even worse on. https://blog.jetbrains.com/ai/2026/05/what-happens-when-you-...

▲ cassianoleal 5 hours ago | parent | prev | next [-]

> One of: Claude Code, Cursor, Codex, or OpenCode

What makes it incompatible with Pi, Zed or any other harness?

	▲	aeneas_ory 4 hours ago \| parent [-]
		Only the amount of free time I have to work on it - nothing fundamentally prevents it. PRs welcomed!

▲ ramon156 4 hours ago | parent | prev | next [-]

Lumen is pretty cool, it's just local RAG, but it's a realistic approach at RAG. I would still keep tool-calling in some places though.

▲ cosmojg 2 hours ago | parent | prev | next [-]

How does this compare to ripgrep and fzf?

▲ simianwords an hour ago | parent | prev | next [-]

> Whether it's caveman

the kinda guy who honestly thinks caveman.md reduces costs, actually adds it to his system prompt and is painstakingly reading the terse output

▲ rvz 2 hours ago | parent | prev | next [-]

So we replace all those "hacks" and snake-oil with even more pseudoscientific hacks and snake-oil?

	▲	aeneas_ory 2 hours ago \| parent [-]
		See my other comment on the person that commented the same thing you did!

▲ liam_ilands 3 hours ago | parent | prev [-]

[flagged]