Some of the most exciting engineering work is happening in the DS4 repo - and I'm watching it almost like a sports game.

When the DSpark paper came out[1] the next day we had folks attempting to implement, working together, validating their failures. Eventually their work being synthesized into a PR[1] that admits performance is not ideal. Something antirez alluded to in one of his videos (speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference).

There's recent work into "directional steering"[3] that has made it's way into per-session directional steering overrides thanks to audreyt[4].

There's support for the new Hy3[5] model also thanks to audreyt[6].

There's Pre-M5 optimizations[7] in the queue thanks to ivanfioravanti who also helped with some of the initial M5 optimizations.

I haven't watched a repo like this since llama.cpp and whisper.cpp in the early days (though llama.cpp is pretty exciting right now with the SYCL improvements that are flowing in for the new Intel GPUs).

The DS4 repo is a really interesting place to watch folks who heavily code with agents collaborate together in a way that seems pretty effective. I've been really enjoying it.

[1]: https://arxiv.org/abs/2607.05147

[2]: https://github.com/antirez/ds4/pull/502

[3]: https://arxiv.org/html/2406.00045v2

[4]: https://github.com/antirez/ds4/pull/148

[5]: https://hy.tencent.com/research/hy3

[6]: https://github.com/antirez/ds4/pull/523

[7]: https://github.com/antirez/ds4/pull/555

> speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference

This is somewhat of a nitpicking point, but AIUI speculative decoding is worthwhile if (1) you have viable unexploited parallelism that can speed up the "verify" step compared to plain decode, and (2) it's profitable for you to load complete model layers into memory in bulk, not just a tiny fraction of active parameters. (2) tends to be true for large hosters, but not only, e.g. if you're running inference on a large dense model (think Mistral Medium) with SSD streaming and a single session (no concurrent batching), speculative decoding can be quite great.

It's also a spectrum, e.g. some MoE models are not very sparse and comparatively few concurrent requests might already span most of the experts, such that loading a complete set of layer parameters and doing speculative decoding to increase compute intensity actually becomes worthwhile.

The ability to use steering vectors seems really powerful. I know you can build your own, but are there any collections of prebuilt steering vectors? Or even just mentions of what kinds of steering vectors people are building for their own use?

Thanks! And sorry for not yet merging many of those. The problem is, I'm dealing with tensor parallelism for the CUDA and Metal-RDMA fork right now, so was not albe to care about PR / issues for a lot of time.