California Ag to Openai: Harm to Children Will Not Be Tolerated

Well, reason on it, and you'll see that reason will confirm the instinct.

It's like an inhuman instinct, ingrained into the model by the pre-training process. To the model, the context is the world it sees - and it will always try to match its outputs so that they fit in with what it sees. In many ways, this context-matching process is at the very foundation of an LLM's behavior.

And the more context there is, the more constrained the space of "consistent continuations" becomes, and the harder it may become to break a model out of a rut it got stuck in.

This is what gives power to few-shot prompts. The model is primed to look at prior demonstrations, and act in a way that's consistent with what was demonstrated to it. Self-consistency works in your favor there. But it's also what powers a lot of unwanted LLM behaviors. Like poor multi-turn instruction following - when the model's self-consistency begins to dominate the conversation, and its own prior actions begin to have more weight in its future behavior than the instructions added by the user (hi Gemini).

This also means that you can totally "boil the frog" by shifting the vibe of the context over time. A well trained "harmless, honest and helpful" chatbot would never tell the user "you should try fentanyl"! But keep the conversation going for long enough, hit on the right themes, and that innate consistency drive may begin to overpower the "HHH" training. At a certain point, if the vibe of the conversation permits, the LLM might actually say "you should try fentanyl" to its user, training be damned.

Another tricky thing is that you can totally make a frog that boils itself! For example, if your model (hi GPT-4o) has a sycophancy bias? Then with every time it agrees with the user in an over-the-top sycophantic fashion, the strength of "the AI is a total sycophant" in the context grows. And the AI wants to match the context! The innate sycophancy bias and the sycophancy bias added by the self-consistency drive then feed into each other. Until the combined might of the two overpowers the "HHH" training, and the AI tells you that your idea to add some paint thinner into your herbal tea is a brilliant insight!