Not without attention or something approximating it.

The fact that gzip is relatively fast should be your first clue that something important is missing.

Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.

If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.

▲

Retr0id 5 hours ago | parent | next [-]

> Gzip scales linearly. I can gzip a one terabyte file.

In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.

▲

bob1029 4 hours ago | parent | next [-]

I'll concede the window part, but Gzip runs within the physical confines of a single cpu core and is typically entirely resident in local caches. The point is not just the quadratic scaling but also what it scales with.

Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.

	▲	pishpash 3 hours ago \| parent [-]
		It won't move as fast as gzip provided the same resources, but to say that LLM's can never run at 300mbps on any hardware is ... a bold bet.

▲

Sesse__ 5 hours ago | parent | prev [-]

Match-finding does not need to be quadratic. However, truly optimal gzip block splitting is very slow, indeed.

▲

amelius 5 hours ago | parent | prev | next [-]

Perhaps a better question is if LLMs are used as compressors, how well is that expected to work.

▲

magicalhippo 5 hours ago | parent | next [-]

> if LLMs are used as compressors, how well is that expected to work

Quite well. This project[1], by Fabrice Bellard of ffmpeg fame, is quite old in AI years and uses an ancient LLM, but still beats xz by a solid margin.

[1]: https://bellard.org/ts_zip/

▲

amelius 4 hours ago | parent [-]

Makes me wonder if compression ratio can be used as a measure for intelligence. Any benchmarks using it?

	▲	magicalhippo 3 hours ago \| parent [-]
		> Any benchmarks using it? A challenge as I understand it is reproducibility. Normal LLM runtimes aren't typically fully reproducible even with same random seeds for distribution sampling, due to floating-point numbers, batching and such. Though averaging over many runs could alleviate that I suppose. While it would measure some aspects of intelligence, I'd argue it fails to capture other, more creative aspects.

▲

5 hours ago | parent | prev | next [-]

[deleted]

▲

Retr0id 5 hours ago | parent | prev [-]

Extremely well, aside from speed.

▲

bob1029 5 hours ago | parent [-]

> aside from speed.

And energy consumption.

	▲	K0balt 3 hours ago \| parent [-]
		I wonder if, on an information theoretical level, compression radiates heat, and decompression absorbs it?

▲

fedeb95 4 hours ago | parent | prev [-]

I agree, but also equating LLMs with intelligence is wrong.