You can take 8192 bytes of information (1024 x 32 bit floats) and reduce that to 128 bytes (1024 bits, a 64x reduction in size!) and still get results that are about 95% as good.

I find that cool and surprising.

▲

sa-code 3 days ago | parent | next [-]

I'm with you, it's very satisfying to see a simple technique work well. It's impressive

▲

computably 3 days ago | parent | prev [-]

1024 bits for a hash is pretty roomy. The embedding "just" has to be well-distributed across enough of the dimensions.

▲

ImPostingOnHN 3 days ago | parent [-]

Yeah, that's what I was thinking: Did we think 32 bits across each of the 1024 dimensions would be necessary? Maybe 32768 bits is adding unnecessary precision to what is ~1024 bits of information in the first place.

	▲	FuckButtons 2 days ago \| parent [-]
		That’s a much more interesting question, I wonder if there is a way to put a lower bound on the number of bits you could use?