| ▲ | draxen 3 hours ago | |
I really appreciate this. I hope LLM inference providers and Chinese LLM labs start using this latest vLLM version ASAP. I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024. If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024. | ||
| ▲ | jacquesm 3 hours ago | parent [-] | |
> I do things that require ~300 tool calls per prompt I have multiples of that with some regularity, what is the hangup? | ||