TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1

NitpickLawyer 2 hours ago | parent | next [-]

There's two variants of this (or, as the joke goes, for very big values of bit):

Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.

1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

▲

PcChip an hour ago | parent [-]

this is a really dumb question, but how is -1 represented?

is it a float? if so, how many bits is the float?

I've never heard of a bit ever having more than two possible values

▲

edflsafoiewq 4 minutes ago | parent | next [-]

It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128.

Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.

It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.

▲

petu 40 minutes ago | parent | prev | next [-]

packing multiple trits together

e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing

▲

zawaideh 44 minutes ago | parent | prev [-]

It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.

	▲	throwawayffffas 7 minutes ago \| parent [-]
		I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer. The way they do it is packing like the other comment says. Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.

▲

bensyverson 2 hours ago | parent | prev [-]

Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.