| ▲ | alex7o 11 hours ago | |
There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed. Also you can ship 9b fp4 model but it all just depends | ||
| ▲ | _heimdall 11 hours ago | parent | next [-] | |
Knowing what's installed would have to be an OS API. If LLMs provide a standard API surface to the OS, likely including metadata related to feature support. | ||
| ▲ | LPisGood 11 hours ago | parent | prev [-] | |
You can know what the user has installed if the OS developer offers something. | ||