| ▲ | Reverse-engineering Nvidia's CUDA-checkpoint for faster cold starts(blog.doubleword.ai) |
| 20 points by ilreb a day ago | 3 comments |
| |
|
| ▲ | zoobab a day ago | parent [-] |
| Would it be able to swap models on demand with this? |
| |
| ▲ | fergus00 a day ago | parent [-] | | https://blog.doubleword.ai/fast-sglang-starts yeah, this is part of the idea. If you get cold starts down to seconds or subseconds, then you can run many models multiplexed on the same GPUs | | |
| ▲ | zoobab a day ago | parent [-] | | I have read a Linkedin post in french recently with a university lab (one big machine serving hundreds of users) mentioning that they could swap models on demand. Will try to find back the link. |
|
|
