new | show | ask | jobs Github

vultour 2 hours ago

Which self-hosted one would you recommend?

▲

SOLAR_FIELDS 2 hours ago | parent [-]

LiteLLM is one of the most popular solutions. You would self host the gateway

▲

sothatsit an hour ago | parent [-]

We use LiteLLM and it is a bit of a dumpster fire of enterprise features and bugs. I can't even update the budget on keys in the UI (enterprise feature, although it may be a bug that it is marked as such). I can still update budgets through the API, but the API is a bit of a mess as well. Then we've ran into a lot of bugs like the UI DDOSing itself when the retry mechanism broke and it just started spamming API requests. And then basic features like the cleanup of old logs is an enterprise feature.

We are actively looking to switch away from it, so it was nice to stumble on a post like this. Something so simple as a proxy with budgeting for keys should not be such a tangled mess.

▲

NeutralCrane 42 minutes ago | parent | next [-]

Are there other alternatives you have been looking at? I’m just getting started looking at these LLM gateways. I was under the impression that LiteLLM was pretty popular but you are not the only one here with negative things to say about it.

	▲	sothatsit 13 minutes ago \| parent [-]
		I am planning to try any-llm-gateway that this post is about. We don't need anything fancy, so it seems that this might cover our needs.

▲

jetbalsa an hour ago | parent | prev [-]

I'm currently using apisix its ai rate limits are fine and the webui is a little json heavy but got me going on load balancing a bunch of models across ollama installs