It's funny that we've finally returned to tanh activation functions, time is a circle.

▲ rhdunn 4 hours ago | parent | next [-]

From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but:

    f_gate(b,x) = b * tanh(x / b) * sigmoid(x)

    f_up(b,x) = b * tanh(x / b)

Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).

▲ whimsicalism 4 hours ago | parent | next [-]

Still more of a tanh than I've seen in years

▲ lostmsu 3 hours ago | parent | prev [-]

Which part of that is doing gating? I thought gate generally looks like

    f_gate(x, y) = f(x) * y

for some f. In your case b is a constant hyperparameter though, e.g.

    f_gate(x) = 4 * tanh(x / 4) * sigmoid(x)

So where's the gate?

	▲	rhdunn an hour ago \| parent [-]
		In the linked "Kimi-K3 Technical Report [pdf]" paper, section 2.3 (Stable LatentMoE, p6) has the table with those equations on (top of p7, using β_1 for the gate branch and β_2 for the up branch). They talk specifically about the function in section 2.3.2 (Sigmoid Tanh Unit GLU, bottom of p7). Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question. Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.

▲ pinkmuffinere 5 hours ago | parent | prev [-]

Wow this is fascinating enough that I’m actually going to read tfa lol