Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5

stpedgwdgfhgdd an hour ago | parent | next [-]

I just dont get why people choose Python and not e.g. Go for high performance problems.

	▲	larme 9 minutes ago \| parent \| next [-]
		Go is not high performance enough. Like what others said, you implement the high performance part in C++ and use python to glue them.
	▲	Yoric 36 minutes ago \| parent \| prev \| next [-]
		Go is pretty good at performance, but pretty bad at expressing domain-specific logics. Python is the opposite, but once you have isolated the parts that need to be optimized, it's quite easy to rewrite them in a native language (in particular, the Rust-Python bindings are really good, although in this project, it's C++).
	▲	sigmoid10 an hour ago \| parent \| prev [-]
		Python is a very convenient skeleton for gluing together high performance modules that were written in C or cuda. Writing boilerplate code in those to adapt them to your project is much more inconvenient.

▲

hexnuts 2 hours ago | parent | prev | next [-]

Bad site design, if I can't scroll to see the next slide, that's just broken.

▲

x0ruman an hour ago | parent | prev | next [-]

The functionality is impressive, but the website needs some work

▲

sakex an hour ago | parent | prev | next [-]

Is this based on paged attention with hashing of the pages?

▲

pythongiant 3 hours ago | parent | prev [-]

KVBoost is a chunk-level KV cache reuse library for HuggingFace models (pip install kvboost). It supports two recompute strategies (selective boundary and CacheBlend), int8/int4 KV quantization for 2–4x RAM reduction, disk-backed cold storage, and 11 architectures including Llama, Qwen, Gemma, Mistral, and Phi. On Qwen2.5-3B we measured 47.9x TTFT speedup on an 8-turn conversation, 21x on code context reuse, 100–743x faster than MLX, and 3–41x faster than vLLM-MLX — including interior chunk reuse where vLLM gets zero hits. Outputs are token-for-token identical to baseline under greedy decoding. Works best on 3B+ models with 500+ token shared context. GitHub: https://github.com/pythongiant/KVBoost

▲

snovv_crash 2 hours ago | parent | next [-]

Even the things that should be normal dashes are em-dashes

	▲	mrob an hour ago \| parent [-]
		En-dashes are not em-dashes, and they're standard typography for numeric ranges. https://en.wikipedia.org/wiki/Dash#Ranges_of_values

▲

arjie 2 hours ago | parent | prev | next [-]

I don't get it. The output of the CacheBlend paper is in LMCache. Did you compare against vLLM with LMCache? This is confusing.

▲

pferdone 2 hours ago | parent | prev [-]

slop