| ▲ | Lio an hour ago | |
RAM compression sounds very useful but the thing I really need extra RAM for is local inference. Since the models are effectively highly compressed data already my assumption is that RAM compression won't help improve that much (I'm assuming unified memory here as per the Framework Desktop). A bit like trying to compress a JPEG in a zip file. Could anyone comment on that? | ||
| ▲ | kalcode 8 minutes ago | parent | next [-] | |
I don't know at all. But I've been playing with oMlx and Splash. Using Pi there are compression extensions like the billion-context. I suspect your right. But all seem to have massive issue boards. What I did see was oMlx release clustering so you can string 3 macs and have 3 times the memory or use one for prefilling accelerating. Which I thought was neat and what I'll likely poke at instead of getting more ran. | ||
| ▲ | rft 31 minutes ago | parent | prev [-] | |
I haven't tried it in RAM, but your reasoning matches what I saw when I looked at compressing weights for archiving them. The weights don't compress nicely with standard, especially fast, algorithms. Some more background in this repo [1] including example compression ratios. I would also worry about decompression speed, that gives you a hard bound on the throughput. If the numbers in this repo are accurate, you likely hit the throughput limit of decompression way before you hit the memory limit, even considering the space savings and thus higher effective throughput. | ||