litellm in the news again

What's interesting here is that this will only affect those who did not migrate off LiteLLM after it was attacked the first time.

I'm not saying everyong should migrate after there's a vuln, but I'm definitely saying that. Unless it's something huge and foundational like a programming language (PHP gets a second chance), just migrate within the next couple of weeks.

I didn't even have it installed the first time, so the ideal programmers, (read, your competition), actually suffered 0 vulns from this dep, if you suffered 1, that's already pretty bad, being burnt twice should be a wake up call. If you get hit 3 times by the same dependency, it's a lost case.

▲

wongarsu an hour ago | parent | next [-]

As someone who does not use litellm but instead a barely maintained underpowered alternative: what did everyone migrate to?

If it matters, I am mostly interested in the aspect of having a proxy that unifies multiple ollama and openai-compatible endpoints with model-based routing and failover. Optionally with api-keys, reporting, budgets, etc. I do not particularly care about the MCP gateway, Agent-To-Agent thingy, and whatever else LiteLLM bolted on over time

▲

dannyboland 41 minutes ago | parent | next [-]

After the last LiteLLM compromise, I took my requirement (an OpenAI compatible endpoint routing across multiple providers based on latency) and built that as a small static Rust binary - https://github.com/sturnus-dev/sturnus

Now that AI makes it cheaper to build such tools in-house, the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker.

▲

tkz1312 an hour ago | parent | prev | next [-]

Claude will write you something custom that fits your exact requirements with zero supply chain risk in an afternoon.

▲

Cu3PO42 3 minutes ago | parent | next [-]

Even if it plays out that way (which I doubt, we have a ton of requirements around spend tracking), the problem I'm seeing is long term maintenance. I don't want to fudge around with my proxy every time one of the model providers comes up with a new parameter that works slightly differently on Bedrock than it does on Azure.

That said, I am looking into alternatives. Two times in one year is really a terrible look.

▲

spike021 an hour ago | parent | prev [-]

how is there zero supply risk? claude will still package dependencies into a homemade solution at some level...

	▲	tkz1312 an hour ago \| parent [-]
		sure zero was a bit strong, but I personally feel pretty good about depending on only the go stdlib, compared to pulling in litellm.

▲

connorbrinton an hour ago | parent | prev | next [-]

Pydantic AI has a great set of abstractions over LLM providers. The documentation tends to focus on its agent loop more, but I find the LLM abstractions quite nice even on their own

▲

TZubiri 7 minutes ago | parent | prev [-]

As with many dependencies, I wholeheartedly question the requirement and relentlessly cut them out without replacement.

Taking the definition from another user on this comment-chain:

"an OpenAI compatible endpoint routing across multiple providers based on latency"

If the endpoint is truly OpenAI compatible, you can just switch out the vendor by switching out the HTTP address, we are talking potentially about a single line change.

- LLM_HOST = "api.openai.com" + LLM_HOST = "api.competitor.com"

It sounds silly, but remember that software is a winner-takes all game, 99% of devs will lose, so if you are seeing 99% of devs do the same thing, it isn't surprising that it's a grossly wrong choice. It takes skill and guts to be in the 1%, reject the mainstream trends, and just write code without grasping at external imports to do even the most basic things.

I'd like to take this opportunity to sell out, I'm open for contracts if you are looking for a remote 1% dev. Early stage startups only, email in userpage.

▲

john_strinlai an hour ago | parent | prev [-]

>PHP gets a second chance

second? i haven't kept count, but we have to be somewhere in the hundreds of chances now