Product Launch

anonymous

123 points

17 comments

Posted4 months agoActive4 months ago

Show HN: Run Qwen3-Next-80B on 8GB GPU at 1tok/2s throughput

github.com

LLM optimizationGPU limitationsAI model deployment

Discussion (17 comments)

Showing 17 comments

addandsubtract

4 months ago

2 replies

Great work! Can this technique also be used to run image diffusion models on lower VRAM GPUs?

anuarsh

4 months ago

Thanks! I don't have much experience with diffusion models, but technically any multi-layer model could benefit from loading weights one by one

GTP

4 months ago

Not an expert in machine learning, but AFAIK diffusion models use a completely different architecture, therefore you can't use the same code to run optimized versions of both. But maybe the core ideas can be adapted to diffusion somehow.

cahaya

4 months ago

5 replies

Nice. Seems like i cannot run this on my Apple silicon M chips right?

poorman

4 months ago

1 reply

If you have 64 GB of RAM you should be able to run the 4-bit quantized mlx models, which are specifically for the Apple silicon M chips. https://huggingface.co/collections/mlx-community/qwen3-next-...

cahaya

4 months ago

Got 32GB so was hoping I could use ollm to offload it to my SSD. Slower but making it possible to run bigger models (in emergencies)

jasonjmcghee

4 months ago

Depends how much ram yours has. Get a 4bit quant and it'll fit in ~40-50GB depending on context window.

And it'll run at like 40t/s depending on which one you have

mhuffman

4 months ago

This particular one may not work on M chips, but the model itself does. I just tested a different sized version of the same model in LM Studio on a Macbook Pro, 64GB M2 Max with 12 cores, just to see.

Prompt: Create a solar system simulation in a single self-contained HTML file.

qwen3-next-80b (MLX format, 44.86 GB), 4bit 42.56 tok/sec , 2523 tokens, 12.79s to first token

- note: looked like ass, simulation broken, didn't work at all.

Then as a comparison for a model with a similar size, I tried GLM.

GLM-4-32B-0414-8bit (MLX format, 36.66 GB), 9.31 tok/sec, 2936 tokens, 4.77s to first token

- note: looked fantastic for a first try, everything worked as expected.

Not a fair comparison 4bit vs 8bit but some data. The tok/sec on Mac is pretty good depending on the models you use.

tripplyons

4 months ago

I have can host it on my M3 laptop somewhere around 30-40 tokens per second using mlx_lm's server command:

mlx_lm.server --model mlx-community/Qwen3-Next-80B-A3B-Instruct-4bit --trust-remote-code --port 4444

I'm not sure if there is support for Qwen3-Next in any releases yet, but when I set up the python environment I had to install mlx_lm from source.

anuarsh

4 months ago

I haven't tested on Apple machines yet, but gpt-oss and qwen3-next should work I assume. Llama3 versions use cuda specific loading logic for speed boost, so it won't work for sure

mendeza

4 months ago

1 reply

what is the throughput for gpt-oss, 1 token every 2 seconds is really slow, but understandable because you are moving cache to disk

anuarsh

4 months ago

1tok/2s is the best I got on my PC, thanks to MoE architecture of qwen3-next-80B. gpt-oss-20B is slower because I load all single layer experts to GPU and unpack weights (4bit -> bf16) each time. While with qwen3-next I load only active experts (normally 150 out of 512). Probably I could do the same with gpt-oss.

aappleby

4 months ago

1 reply

Why even bother with the GPU at that point? CPU would be just as fast if you're bottlenecked on SSD bandwidth.

anuarsh

4 months ago

CPU is much slower than GPU. You can actually use both by offloading some layers to CPU as o.offload_layers_to_cpu(layers_num=12). It is faster to load from RAM than from SSD.

ydlr

4 months ago

1 reply

How dramatically does this shorten lifespan of SSDs?

anuarsh

4 months ago

Good question, need to research this one

anuarsh

4 months ago

There's one more exciting thing about Qwen3-next (except, efficient MoE architecture and fast linear attention) - MTP (Multi token prediction). It is the additional layer that allows generating more tokens without the need to go through the model again. I'm trying to make it work, but unsuccesfully yet. Maybe someone could help me with it - https://github.com/Mega4alik/ollm/blob/dev/src/ollm/qwen3_ne... (dev brunch). Take a look

Resources