Fun fact - single bit neural networks are decision trees.

In theory, this means you can 'compile' most neural networks into chains of if-else statements but it's not well understood when this sort of approach works well.

▲

smokel 9 hours ago | parent | next [-]

> single bit neural networks are decision trees.

I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge. And I get the idea that the paper is stretching the concept of decision trees a bit.

Also, I still don't know exactly what you mean, so would you care to elaborate a bit? :)

[1] https://arxiv.org/pdf/2210.05189

▲

lioeters 8 hours ago | parent | next [-]

Closest thing I found was:

Single Bit Neural Nets Did Not Work - https://fpga.mit.edu/videos/2023/team04/report.pdf

> We originally planned to make and train a neural network with single bit activations, weights, and gradients, but unfortunately the neural network did not train very well. We were left with a peculiar looking CPU that we tried adapting to mine bitcoin and run Brainfuck.

▲

fooker 4 hours ago | parent | prev [-]

> I still don't know exactly what you mean

Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though.

> The trees that correspond to the neural networks are huge.

Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential with this kind of approach.

	▲	fc417fc802 2 hours ago \| parent \| next [-]
		> Training a one bit neural network from scratch is apparently an unsolved problem though. I don't think it's correct to call it unsolved. The established methods are much less efficient than those for "regular" neural nets but they do exist. Also note that the usual approach when going binary is to make the units stochastic. https://en.wikipedia.org/wiki/Boltzmann_machine#Deep_Boltzma...
	▲	cubefox 2 hours ago \| parent \| prev [-]
		> Training a one bit neural network from scratch is apparently an unsolved problem though. It was until recently, but there is a new method which trains them directly without any floating point math, using "Boolean variation" instead of Newton/Leibniz differentiation: https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...

▲

Almondsetat 11 hours ago | parent | prev | next [-]

Do you know of any software that does this? Or any papers on the matter? It could be a fun weekend project

	▲	tomashubelbauer 11 hours ago \| parent \| next [-]
		Made me think of https://github.com/xoreaxeaxeax/movfuscator. Would be definitely cool to see it realized even if it would be incredibly impractical (probably).
	▲	fooker 4 hours ago \| parent \| prev [-]
		I think any quantization approach should work, not an expert on this.

▲

11 hours ago | parent | prev [-]

[deleted]