Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing.

- Bonsai Qwen 3.8 27B

- Strata

- Pulsar

- This...

I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.

I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.

So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.

I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.

At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.

▲

yjftsjthsd-h 29 minutes ago | parent | next [-]

> which is a ridiculously impractical number to review with a few volunteers.

How would that get fixed? I'm skeptical of lowering the standards of review. I suppose if you have some source of trusted volunteers...?

	▲	SillyUsername 10 minutes ago \| parent [-]
		Scan for exploits, spyware and viruses, actually automatically format the PRs if they don't match and warn the submitter, automatically upmerge main/master into branches under consideration where possible and ensure all new functionality has test cases (not the same as code coverage). When if comes to the human reviewer unless an issue is structural, don't care so much about the formatting and keep to one human approval. For simple changes, like documentation, comments etc, these can be approved by bots themselves as any fault is non functional. If you consider that if bugs are noticed they will be fixed by other PRs, it makes less sense now to try to fix all issues in each PR, especially as there's no guarantee this would cover all of them anyway with so many ongoing changes.

▲

thatsabadlook 43 minutes ago | parent | prev [-]

I wouldn't judge a project by not slamming in 1.7k prs a week. What a nightmare. How many of those prs are pure slop or not even worth looking at? I would wager 95% at a minimum

	▲	SillyUsername 41 minutes ago \| parent [-]
		I'm not saying 1.7K a week, but they aren't keeping on top of the PRs being raised so the number just keeps increasing, unless bots just close them because of inactivity. At some point they will have to cut their losses, strip back to a week's worth of PRs and close everything else to get back on top (perhaps this is actually what is happening, it just takes the bots a while to chug through them). Edit: just checked and it's now 1.9K PRs.