| ▲ | tarpitt 6 hours ago |
| I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2 I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth. >SSD Wear Warning > Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health. Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only?
Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk? |
|
| ▲ | vforno 5 hours ago | parent | next [-] |
| I have a small laptop.
If you have more disks available, you could really do some testing.
When you have some benchmarks, submit a pull request or issue so we can maybe work on them.
We are really happy for contribute! |
| |
| ▲ | tarpitt 5 hours ago | parent [-] | | I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused. I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU. If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item. I'm going to benchmark this right now with what I have and I'll get back to you on github. | | |
|
|
| ▲ | valicord 6 hours ago | parent | prev | next [-] |
| > OS page cache can generate writes Is this a hallucination? What am I missing? Why would heavy reads generate writes? |
| |
| ▲ | 5 hours ago | parent | next [-] | | [deleted] | |
| ▲ | 5 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | fallingbananna 5 hours ago | parent | prev | next [-] | | Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD. | |
| ▲ | TacticalCoder 5 hours ago | parent | prev | next [-] | | > Is this a hallucination? What am I missing? Why would heavy reads generate writes? I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached? I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"? | |
| ▲ | onlyrealcuzzo 5 hours ago | parent | prev [-] | | Spilling | | |
|
|
| ▲ | vforno 6 hours ago | parent | prev [-] |
| That's possibly a good idea! We can work on it! |
| |
| ▲ | tarpitt 6 hours ago | parent [-] | | I also just edited my comment with more ideas in the beginning, sorry |
|
