(2025)

As it's 9 months old and they just had a major model release

throwa356262 5 hours ago | parent | next [-]

For K3 read this instead: https://arxiv.org/abs/2607.24653

The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.

▲

mcbuilder 5 hours ago | parent | next [-]

Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.

	▲	throwa356262 5 hours ago \| parent [-]
		Same with DeepSeek papers, they are a joy to read.

▲

senko 4 hours ago | parent | prev [-]

Not an expert, but looks like they did a lot more work on the RL part (9 expert models, full sandbox access for agentic tasks, etc)?

	▲	verdverm 4 hours ago \| parent [-]
		most new effort in training comes in the late phase with RL techniques the pretraining (slurping the internet) only goes so far, the new data being used is from human preferences and agent traces (designed and/or distilled)

▲

cptcobalt 5 hours ago | parent | prev | next [-]

Rather under-discussed back then: https://news.ycombinator.com/item?id=45766937

▲

GaggiX 6 hours ago | parent | prev [-]

I believe OP posted it because the new Kimi K3 has 69 KDA layers (the rest are 24 Gated MLA), I think previous large Kimi models had only MLA layers.

▲

yorwba 5 hours ago | parent [-]

It's not the same KDA as used in Kimi Linear, though.

▲

GaggiX 5 hours ago | parent [-]

What's the difference? They are both called Kimi Delta Attention.

	▲	yorwba an hour ago \| parent [-]
		The differences are explained in section 2.1.1 of the Kimi K3 technical report: https://arxiv.org/pdf/2607.24653#page=4