What GPT-Oss Leaks About Openai's Training Data

Unfortunately the article glances over some of practices of uncovering such patterns in the training data. It goes very straitghfully to the point, no lube needed. It didn't land well for me.

Could it instead be the case that these tokens were initialized at some mean value across the dataset (plus a little noise), and then never changed because they were never seen in training? Not sure if that is state of the art anymore but e.g. in Karpathy's videos he uses a trick like this to avoid the "sharp hockey stick" drop in loss in the early gradient descent steps, which can result in undesirably big weight updates.

Yes.

https://arxiv.org/abs/2403.06634

https://arxiv.org/abs/2311.17035

(I just have these ones off the top of my head because I'm a Nicholas Carlini fan and we interviewed him about these attacks.)

> Do biases go away completely or they just get suppressed down deep in the model's "mind"?

Bias is a human term, and couching the conversation in that context does nothing to address the issue here, because it gets into the quagmire of social context.

Let's say LLM's had taken off 15 years ago at the point system d launched. All the answers given are going to weight toward the old init system simply because there is a lack of information.

LLM's are only repeating the data they are given, and it's cheaper to remove the data after the fact than it is to try to scrub it out of the training data.

Theyre saying if you find references to a very specific set of phrases that were probably included accidentally on github then github is likely part of the training data.

Wouldn't it be best for them to strip that out of the training data for moderation reasons?

FWIW, I didn't get that sense.

And it's nothing new.

https://github.com/jiangyy/gpt-tokens

People found these adult-site-related Chinese phrases in Gpt-4o. The OP is more than one year late.

Hi, thanks! If someone posts better translations I will update them.

s/birth/berth :)

Informed layman warning.

The tokenizer covers the entire dataset. It's basically just a fixed-size Huffman code, grouping together common fragments of letters- for instance, the 100 most common English words are probably all single tokens.

During learning, the model proceeds in roughly the same way a child would: it starts by grouping tokens together, learning the deep regularities of language such as "news[paper]" being more likely than "news[q77.bfe]". Then it incrementally assembles these fragments into larger and larger chains. Similarly , it first learns thematic groupings, such as "word" being more likely somewhere after "dictionary" rather than "stop what I was reading to get the dictionary out every time I encountered a banana assault hungry". Then it starts to pick up "patterns": "as a [baby|child|kid] I had no [idea|concept|clue]". At some point in this process it naturally abstracts concepts from languages: "as a child" starts being internally represented by the same neurons as "als ich ein Kind war".

Then some magic happens that we don't understand, and out pops a neural network that you can talk to and that can write programs and use tools. To be clear, this is the case before RL: probably these patterns are now widespread in the training data, so that the model already understands how to "complete the pattern" on its own. RL then does some magic on top of that to bring it from 20% benchmarks to 80% and presto, AI assistant.

The LLM training process doesn't operate at that conceptual level. What it's doing is closer to examining a large number of possible meanings, seeing which fit the most, and moving its "understanding" in that direction. Repeat enough times, and it develops an association between the new word and the context in which it's used.

New words will usually be combinations of existing tokens, but at the beginning of training a new model, it doesn't "know" what any of the tokens mean. And there's no reason you can't treat every UTF-8 byte as a separate token, but that would require a larger model before you got results that look to a layperson like intelligence, understanding, or knowledge. Tokenisation lets you use a system like word2vec to assign each token a semantic embedding in a vector space, giving the model a bit of a leg up.

---

Response to the sibling comment https://news.ycombinator.com/item?id=45485439, since I've hit the rate limit:

> During learning, the model […] starts by grouping tokens together

You probably could design a ML system that works like this, and it'd probably be more efficient to train than a hundred-billion parameter GPT model, but that's not how GPT model training works. Instead, it attempts all of those things in parallel (although I would expect the solutions to the earlier, easier parts to settle down before the solutions to the later parts do), and the same process is responsible for all of the behaviour in a straightforward fashion.

We do understand the "magic": it's just that it produces a really complicated system that we can't characterise the iterative behaviour of. (For comparison, the iterative function f_c(z) = z² + c, iterated starting at 0, produces the Mandelbrot set.) To use an analogy: imagine the training data is a landscape, and the behaviour of the GPT model trained on it is a weather system. (The parameter count is the amount of atmosphere, or something.) There's nothing magical going on in the weather, but it's just too complicated to predict ahead of time, and tiny gaps in our understanding can magnify into extremely inaccurate long-term predictions. We can, despite this, make some blanket statements about the possible capabilities of a GPT model, of the form "a GPT model will never be able to do X unless you cheat".

The RL magic is, I believe, well understood, but I don't personally understand it. (I know what it does, since RL always does the same thing, but I don't know what it's doing to the model to achieve that.)

> "as a child" starts being internally represented by the same neurons as "als ich ein Kind war"

Yes and no. For a few reasons, including that this kind of association can occur without the same "neurons" getting involved until past the point where that representation exists, it's better to say that they're embedded in nearby regions of a vector space. The actual nodes of the neural network are an implementation detail.

I think it could infer the meaning of words composed out of tokens it has already seen before, same way that you might be able to infer the meaning of an unknown word based on its prefix/suffix, country of origin, context, etc.

For an entire token that it hasn't seen before, it would have to rely only on context. Presumably it could do this, since that is after all the case in the early phases of training.

Glitch tokens should be tokenizer-specific. Gemini uses a different tokenizer from the OpenAI models.

The origins of the OpenAI glitch tokens are pretty interesting: the trained an early tokenizer on common strings in their early training data but it turns out popular subreddits caused some weird tokens to be common enough to get assigned an integer, like davidjl - a frequent poster in the https://reddit.com/r/counting subreddit. More on that here: https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-...

Isn't that exactly what some of these models that have 30b params but only activate 3b at a time

This is addressed at the end of the blogpost

Chinese adult site ads are everywhere in repackaged free and pirate content, which are distributed thru sites including but not limited to github, shadow libraries and youtube.

for same reason, whisper some blank audio will output those ads.

I really hate that it was brought up as a bullet point at all. Like who fucking cares, why is this guy bringing up that boogieman.

I think that is what the article is saying, just with a very misleading phrasing. The phrases are from adult websites, and likely entered the training data via Github (lists for content blockers etc). Just like "to be or not to be" is from Hamlet, but you just read it on HN. At no point does the article actually say that adult websites were in the training data

Why would it require a constitutional amendment?

There's nothing new about being able to copyright something that's a transformation of another work. And they definitely aren't exclusively trained on public data.

I'd settle with them being held in a public trust for public benefit

> I wish we had a constitutional amendment that opensourced all AI commercial AI models and requires documentation and links to all training data and base prompts.

> They are trained on public data at our expense so We The People should own them.

The people who appear to have been trained off for the interesting parts of the blog post are mostly, like me, not American.

> AI should be free. Overhyped and Overpriced. I would love this setup for privacy and security.

Also, this entire blog post only exists because they're curious about a specific free open-weights model.

The "source" being ~"the internet", which we've got as much access to as most of the model makers (i.e. where you don't, they've got explicit licensing rights anyway), and possibly also some explicitly* pirated content (I've not been keeping track of which model makers have or have not done that).

* as in: not just incidentally

What you are describing is more-or-less a planned economy, the polar opposite of America's market economy. The government has the power to appropriate things for the common good because it's perceived that private enterprise isn't a necessary force. Sometimes it works, sometimes it doesn't; only certain countries can "moneyball" their way through economics like that, though. America has long since passed the point of even trying.

Your heart is in the right place here (I agree about FOSS), but there is a snowball's chance in hell that any of this ever happens in the USA. We'll be lucky if AI doesn't resemble cable TV by 2030.

> They are trained on public data

this is questionable, but okay...

> at our expense

?

> so We The People should own them.

in addition to training data, it is my understanding that a model's architecture also largely determines its efficacy. Why should we own the architecture?

Wouldn’t the same argument then be applied to all scraped data?

Unfortunately very unlikely in our forseeable future with the U.S. having a "U.S. against the world" mentality to the AI race. Would love to see this but this would get shot down immediately.

You can imagine LLM fingerprinting to be part of future pentest workflows where they identify the model and know it's weaknesses and vulnerabilities etc...

Isn't the only reason we can do that because we have access to the tokenizer? Do we have the Claude and Gemini tokens? I mean if they didn't publish that, would it defeat this attack?

Yes I thought so too.

I wonder if it will mean more or less revealing of the models that are running agentic flows (we currently abstract with Fast/Smart)

It is also possible that the first model calls other models, and you could reverse engineer the tool call structure by seeing when glitches occur based on different branches of the tool calling

One reason to do this is to test what the underlying model is for a stealth model.

Importantly, though, the token is "\\xadder" which looks a bit like an escaped hex code. That actually suggests a different origin of the token. `\xad` is the Unicode soft-hyphen (U+00AD). The soft hyphen is used to suggest where it makes sense to hyphenate a word if a line-break is needed. This shows up fairly frequently (2.9k occurrences) on GitHub in web-scraping datasets, which suggests that a model trained on data scraped from the web might see a fair number of these.

Basically, OpenAI is trained on web data that has a number of words where splitting on -der makes sense (e.g., mur-der, un-derstanding, won-derful; although the most common occurrence in a GitHub search for "\\xadder" is what appears to be an incorrectly encoded string "L\xc3\xadder", probably from the Portuguese and Spanish "Lí-der").

Anyways, using the o200k tokenizer `mur\xadder` yields two tokens (88762 and 179582). 88762 encodes "mur" and 179582 encodes "\xadder".

This guy adults.