Control LLM Spend and Access with Any-LLM-Gateway

Postedabout 2 months agoActiveabout 2 months ago

aittalam

59 points

26 comments

blog.mozilla.aiTechstory

calmpositive

Debate

0/100

Large Language ModelsArtificial IntelligenceAPI Gateway

Key topics

Large Language Models

Artificial Intelligence

API Gateway

The post introduces any-LLM-gateway, a tool to control LLM spend and access, with no discussion or comments provided.

Snapshot generated from the HN discussion

Discussion Activity

Very active discussion

First comment

Peak period

168-180h

Avg / period

Comment distribution25 data points

Loading chart...

Based on 25 loaded comments

Key moments

01Story posted
Nov 12, 2025 at 1:06 PM EST
about 2 months ago
Step 01
02First comment
Nov 19, 2025 at 2:01 PM EST
7d after posting
Step 02
03Peak activity
25 comments in 168-180h
Hottest window of the conversation
Step 03
04Latest activity
Nov 20, 2025 at 12:12 AM EST
about 2 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (26 comments)

Showing 25 comments of 26

bravura

about 2 months ago

2 replies

Thoughts on any-llm-gateway versus litellm-proxy?

litellm is a great library, but one team using litellm-proxy reported having many issues with it to me. I haven't tried it yet.

cowmix

about 2 months ago

Yeah, I wonder what gaps in Litellm Proxy made Mozilla want to even do this.

dbish

about 2 months ago

What were the problems? I've been trying it out and haven't hit issues yet, but not using it at scale yet so I'm curious what to watch out for. I figure it's open source (MIT) so I can make changes as needed if there was anything particulary annoying.

verdverm

about 2 months ago

1 reply

This service (llm proxy to all providers) are a dime-a-dozen

This one has very little on monitoring and no reference to OTEL in the docs

vultour

about 2 months ago

1 reply

Which self-hosted one would you recommend?

SOLAR_FIELDS

about 2 months ago

1 reply

LiteLLM is one of the most popular solutions. You would self host the gateway

sothatsit

about 2 months ago

3 replies

We use LiteLLM and it is a bit of a dumpster fire of enterprise features and bugs. I can't even update the budget on keys in the UI (enterprise feature, although it may be a bug that it is marked as such). I can still update budgets through the API, but the API is a bit of a mess as well. Then we've ran into a lot of bugs like the UI DDOSing itself when the retry mechanism broke and it just started spamming API requests. And then basic features like the cleanup of old logs is an enterprise feature.

We are actively looking to switch away from it, so it was nice to stumble on a post like this. Something so simple as a proxy with budgeting for keys should not be such a tangled mess.

Manouchehri

about 2 months ago

LiteLLM was good in the early days. I ran into more features than bugs. Sadly in the past year or so, I run into more bugs than features.

jetbalsa

about 2 months ago

I'm currently using apisix its ai rate limits are fine and the webui is a little json heavy but got me going on load balancing a bunch of models across ollama installs

NeutralCrane

about 2 months ago

Are there other alternatives you have been looking at? I’m just getting started looking at these LLM gateways. I was under the impression that LiteLLM was pretty popular but you are not the only one here with negative things to say about it.

smcleod

about 2 months ago

1 reply

Interested to see how this stacks up against Bifrost (fast but many features paywalled) and LiteLLM Proxy (featureful but garbage code quality). Especially if it gets a web admin / reporting frontend and high availability.

NeutralCrane

about 2 months ago

3 replies

We are just now looking into LLM Gateways and LiteLLM was one I was considering looking into. I’m curious to hear more about what makes the code quality garbage.

smcleod

about 2 months ago

1 reply

I've deployed LiteLLM proxy in a number of locations and we're looking to swap it out (probably to Bifrost), we've seen many bugs with it that never should have made it to a release. Most stem from poor code quality or what I'd classify as poor development practises. It's also slow, it doesn't scale well and adds a lot of latency.

Bugs include but are not limited to multiple ways budget limits aren't enforced, parameter handling issues, configuration / state mismatches etc...

What makes this worse is if you come to the devs with the problem, a solution and even a PR it's very difficult to get them to understand or action it - let alone see critical things like major budget blowouts as a priority.

dbish

about 2 months ago

1 reply

What about forking it for your own use? Not worth it for the bugs you had fixes for?

smcleod

about 2 months ago

Not worth the technical debt and architecture of the codebase. To be honest I'd sooner completely rewrite it in Golang/Rust or otherwise.

SOLAR_FIELDS

about 2 months ago

I personally had no issues using the client libs, my only complaint was that they only offer official Python ones would love to see them publish a typescript one

everlier

about 2 months ago

How do you like bugs where tools are not working, but only for Ollama provider and only when streaming is enabled? This is one of the real instances I had to debug with LiteLLM.

bitpush

about 2 months ago

2 replies

I'm conflicted on what Mozilla is doing here. On the one hand, it is nice that they are getting involved but com'on, dont you all have Firefox to work on?

This is a classic case of an over enthusiastic engineer who says yes / raises hand to everything, but doesnt do any one thing properly. At some point, you have to sit down and tell them to focus on one thing and do it properly.

benatkin

about 2 months ago

1 reply

They're dumping competition on two other open source python libraries LiteLLM and simonw's llm. Unlike these two, Mozilla's any-llm doesn't have to make money. I'm sure simonw will be welcoming because he's a friendly kind of guy, but it might seem frustrating to LiteLLM which has a paid offering, for which they'd prefer organic competition rather than whatever magic 8 ball Mozilla uses.

tomComb

about 2 months ago

1 reply

I don’t think those are comparable. simonw's llm Has a python SDK, but it’s very much CLI first. Light LLM is very much about the SDK. You can wrap some agent SDK’s around it, like Gemini, but that’s for agents not work flows. I can’t really think of them as in the same category.

benatkin

about 2 months ago

llm is a lot about the Python API (or SDK) as well: https://llm.datasette.io/en/stable/python-api.html

It shows how to use it async or sync, and even handles using async in a sync context.

It's hard to write a good CLI without also writing most of a Python API, and llm went the rest of the way by documenting it. I think llm has the best docs of the Python API of the three.

ekr____

about 2 months ago

Mozilla spun up a whole new entity (Mozilla.ai) to do AI stuff, so doing AI stuff outside of Firefox is already baked into the equation, whatever you think of this particular thing.

ouk

about 2 months ago

There is also PydanticAI Gateway (https://ai.pydantic.dev/gateway/). I use it with the PydanticAI framework and it's quite nice.

Emen15

about 2 months ago

Feels like this is carving out a middle layer, simpler than other gateways out there, but way more practical than just a unified client library.

mfrye0

about 2 months ago

I was looking for a version of a proxy that could maximize throughput to each LLM based on its limits. Basically max requests and input/output tokens per second.

I couldn't find something, so I rolled a version together based on redis and job queues. It works decently well, but I'd prefer to use something better if it exists.

Does anyone know of something like this that isn't completely over engineered / abstracted?

1 more comments available on Hacker News

View full discussion on Hacker News

ID: 45903485Type: storyLast synced: 11/20/2025, 7:50:24 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN