Client-Side GPU Load Balancing with Redis and Lua

Postedabout 1 month agoActive25 days ago

lneiman

47 points

10 comments

galileo.aiTech Discussionstory

informativepositive

Debate

20/100

GPU AllocationRedisLua Scripting

Key topics

GPU Allocation

Redis

Lua Scripting

Discussion Activity

Light discussion

First comment

18s

Peak period

132-144h

Avg / period

2.2

Key moments

01Story posted
Dec 2, 2025 at 12:39 PM EST
about 1 month ago
Step 01
02First comment
Dec 2, 2025 at 12:39 PM EST
18s after posting
Step 02
03Peak activity
4 comments in 132-144h
Hottest window of the conversation
Step 03
04Latest activity
Dec 9, 2025 at 2:41 AM EST
25 days ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (10 comments)

Showing 11 comments

lneimanAuthor

about 1 month ago

1 reply

Author here. We were hitting tail latency and low GPU utilization issues serving SLMs via Triton.

I built a scrappy client-side router using Redis and Lua to track real-time GPU load. It boosted utilization by ~40% and improved latencies.

Happy to hear feedback on the implementation or thoughts on better ways to do this!

pbrumm

27 days ago

Have you tried switching it to a job queue where the GPU instances try to keep themselves busy. That way you can auto scale the gpus based on utilization. I find it easier to tune and you can monitor latency and backlogs easier. It does require some async mechanisms to the client but I have found it easier to maintain

artyom

25 days ago

1 reply

If I understand the article correctly, any sufficiently capable attacker can:

- Know the global state of your GPU cluster via the client.

- Target the most struggling GPU instances specifically since the client decides which one to hit.

You offer a free tier which means anyone can get an account and try to do it (e.g. you can have one "harmless, mostly inactive" free account with the only purpose of retrieving GPU cluster status, and a bunch of burner accounts to overload struggling instances.

I may be completely wrong, but this sounds like DDoS served on a silver plate to me.

singron

25 days ago

1 reply

They run these clients themselves and the redis instance isn't publically exposed.

It would indeed be very strange to hope your random users coordinate with your client side load balancer. You wouldn't even have to send real traffic. You could just manipulate redis directly to force all the real traffic to go to a single node. DoSing redis itself is also pretty easy.

artyom

25 days ago

1 reply

I don't think the article implied that the client was for some sort of internal server-to-server communication, or that the Redis instance was directly exposed to the internet.

So no, I don't think they run these clients themselves. If the code runs out there, it's open to inspection.

tpurves

25 days ago

Either way, you are right to point out that it important to only a try a pattern like this if your clients are highly trusted (or/and have additional compensating controls against DDOS threats). It would be beneficial if the OP made more explicit what their client/server relationships and also flagged the risk you mentioned for general audiences not to go implementing such a solution in the wrong places.

PunchyHamster

25 days ago

2 replies

I'm gonna guess just switching from round-robin to leastconn (most balancers offer that option) would solve that just fine. You can then go to dynamically tune server weights if you have servers of unequal size or some other issues.

kgeist

25 days ago

Yeah, that can work. Just yesterday I benchmarked load balancing of LLM workloads across 2 GPUs using a simple least_conn from nginx. The total token/sec scaled as expected (2 GPUs => 2x token/sec), and GPU utilization reached 100% on both, as I increased concurrency from 1 to 128 simultaneous generations.

gorkish

25 days ago

Yeah I really don't understand why they went this direction as it builds considerable additional complexity directly into the application to solve a problem with an external component

I would have probably approached this by implementing a fix for the misbehaving part of k8s, though since there isnt a default LoadBalancer in k8s, I can't really can't speculate further as to the root cause of the initial problem. But most CNI or cloud providers that implement LB do have a way to take feedback from an external metric. I'd be curious why doing it this way wasn't considered, at least.

bnr4u

25 days ago

Very cool work! Did you investigate using the Power of two random choices method for your load balancing algorithm ?

https://brooker.co.za/blog/2012/01/17/two-random.html https://medium.com/the-intuition-project/load-balancing-the-...

maknee

25 days ago

It seems like the load_score serves a proxy for how much needs to be done. Is there a real value that could be used instead? The solution requires syncing with all of the GPU nodes anyways.

View full discussion on Hacker News

ID: 46123948Type: storyLast synced: 12/8/2025, 11:25:21 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN