Show HN: docker/model-runner – an open-source tool for local LLMs
github.comA few things that could make it even more powerful (maybe some are out of your scope):
- Persistent model settings (context size, temperature, etc.) across restarts — right now it always resets to 4k, which breaks multi-turn agents. - HTTP/gRPC interface to let tools and frameworks talk to DMR directly, not only through the CLI. (Here the issue is on Docker MCP side, right?) - Simple config management (`docker model set` or `docker model config`) so we can tweak GPU, threads, precision, etc. predictably. (there are at least a couple of issues on this topic already...)
TBH, I love how fast the discussion evolved today.
Congrats and good luck with this. I'll try to help, promised!
We're rebooting the model-runner community and wanted to share what we've been up to and where we're headed.
When we first built this, the idea was simple: make running local models as easy as running containers. You get a consistent interface to download and run models from different backends (llama.cpp being a key one) and can even transport them using familiar OCI registries like Docker Hub.
Recently, we've invested a lot of effort into making it a true community project. A few highlights:
- The project is now a monorepo, making it much easier for new contributors to find their way around.
- We've added Vulkan support to open things up for AMD and other non-NVIDIA GPUs.
- We made sure we have day-0 support for the latest NVIDIA DGX hardware.