GPT-Oss Reinforcement Learning

The primary goal of the release and our notebook https://colab.research.google.com/github/unslothai/notebooks... was actually to showcase how to mitigate reward hacking in reinforcement learning - for example when RL learns to cheat and output global variables instead like editing the timer to cheat on benchmarking and others. You can edit the notebook to do rl on other powerful models like Qwen, Llama etc automatically with Unsloth as well via our automatic compiler! We also made sink attention and moe inference super optimized for training - note flash attention 3 doesn't have sink backwards support so you'll have to use unsloth.

Gpt-oss tbh in our tests is a truly powerful model, especially the 120b variant - it's extremely popular in western enterprises since yes it's from openai but also because reasoning mode high and the censored nature and its reasoning capabilities are attractive. A big underutilized feature is its web search and internal intermediate tool calling which it can do as part of its reasoning chain just like o3 or gpt5.

RL yes isn't an all powerful hammer, but it can solve so many more new problems. For a financial institution, you can make automatic trading strategies via RL. For an intelligence agency, decryption via RL. For a legal startup, possibly case breakthroughs via RL, automatic drug candidates etc. And yes, big labs want to automate all tasks via massive RL for eg being able to play pokemon and all other games as one example. RL opened so many doors since you don't need any data, just one prompt like "make fast matrix multiplications kernels", and reward functions - it can allow many more interesting use cases where data is a constraint!

I’ve also seen a lot of enterprises suddenly invest in fine-tuning their own models, even though there is absolutely no reason they should be doing that. But because there’s a huge “we need to do something with AI” directive from the top. Eg “if we fine tune an AI model to understand our custom query language it will prevent our data scientists from taking down production databases”. This is an actual example I encountered just last week.

So if there’s a point, it’s probably not that it’s the best idea, but rather than enterprises are willing to buy it.

I currently have Qwen3 Coder 30B A3B on prem for developers to use and it's pretty good for that: fits within two Nvidia L4 cards (with quantization), has about 60 tok/s performance and can even use tools with something like RooCode or OpenWebUI.

However, if anyone asks it stuff in Latvian, it does mess up more often than not, like someone who has half-learned the language: more or less uses the words that represent the correct concepts, but not always the best pick and really often missing or wrongly used diacritics (ā, č, ē, ģ, ī, ķ, ļ, ņ, š, ū, ž). In a word, some basis of the language is there, but sadly in practice it's still unusable.

So far working with Latvian text I need to maintain EuroLLM running in parallel, which has great Latvian knowledge, but at the same time just knows less (not a model that's good for programming) and doesn't know how to call tools as well and isn't really meant for long contexts: https://huggingface.co/collections/utter-project/eurollm-66b...

So my ideal model (for this use case) would be something along the lines of: around 30B since can't fit anything much bigger into the VRAM for now, MoE, supports tool calling, okay programming knowledge, has a long enough context support, can converse in Latvian so I don't need to run 2 models (which means that the context sizes that can fit within VRAM are way too small for either of the models).

Without finetuning, I just have to sit and wait for someone to release that and it feels unlikely that it'll just pop into existence (even the bigger EuroLLM model is nowhere to be seen, TildeOpen works bad). With finetuning, maybe I have a chance to take the Latvian Wikipedia or whatever other samples of the language being used I can get, filter down to topics I care about, maybe use EuroLLM to generate question/answer pairs for training from that input data and then just run Unsloth over a weekend or something (who knows, maybe that'd be enough to bring Qwen3 up to speed).

Do most people care about that stuff? Probably not, but when you have to deal with a language that's less represented in the training data but will never have the budget to train something from scratch, finetuning is nice to have. RAG can't fix it not knowing how to use language.

There was a post the other day on HN where a dev spent $2k doing RL on an open model and beat the frontier models on Web Voyager.

Research is packed with examples like this yet we keep hearing from the community that training doesn’t work.

Then consider voice/vision modalities where context engineering doesn’t work well at all. I just spent 2 years on the computer use problem and can tell you with absolute certainty that context won’t get you there (we tried and tried). The only thing that meaningfully moved the needle was RL.

I really wish this idea that only big labs can train models would die. It’s really hurting the ecosystem at this point.