Openai Says It's Scanning Users' Conversations and Reporting Content to Police

But a big part of the issue is that OpenAI wants user engagement - and "not being sycophantic" goes against that.

They knew feeding raw user feedback into the training process invites disaster. They knew damn well that it encourages sycophancy - even if they somehow didn't before the GPT-4o debacle, they sure knew afterwards. They even knew their initial GPT-5 mitigations were imperfect and in part just made the residual sycophancy more selective and subtle. They still caved to the pressure of "users don't like our update" and unrolled a lot of those mitigations.

Also plenty of those hotlines are BS, or don’t work, or flat out don’t exist for given locales, etc.

The biggest issue is that LLM’s can act like a person, but aren’t a person, and fundamentally this causes problems. Especially for people that are already borderline or fully crazy.

Doesn't necessarily even need to call (particular in case of false positives) but there absolutely should be detection and a cutoff switch, where the chatbots just refuse to continue the conversation and then print out the hotline numbers (much like with reddit cares messages).

I'm generally not in favor of censorship or overly protective safeguards on LLMs, but maybe it's needed for hosted models/services that are available to the masses.

But before they get locked down more, we should try some legislation to limit how they can be marketed and sold. Stop letting OpenAI, etc. call the models "intelligent" for one. Make the disclaimers larger, not just small print in the chat window but an obvious modal that requires user agreement to dismiss - disclaim that it's a predictive engine, it is not intelligent, it WILL make mistakes, do not trust its output. Make it clear during the chat session over and over again, and then have a killswitch for certain paths.

In the meantime Ive had two therapists that we ended with since they didnt help the condition, and we're very expensive.

Having trouble parsing the double negation in your comment.

Sorry, I’ve had a long day :)

Correct, but that's true for all bugs.

In this case, the deeper bug was the AI having a training reward model based too much on user feedback.

If you have any ideas how anyone might know what "too much" is in a training reward, in advance of trying it, everyone in AI alignment will be very interested, because that's kinda a core problem in the field.

I think OpenAI is only now beginning to realize how connected some people are to their product and that the way their models behave has a huge impact.

Just to clarify, this statement will always be true: "N members of our species aren't ready for technology Y". And N will always be greater than or equal to 1.

I often see comments online descending into the argument about firearms. Besides the potential number of people hurt/killed, what's the difference between someone walking into a school with a gun versus one walking into a school with a knife? Or a sharpened spear (from other comments in the thread)?

In many ways, I think a knife could be worse. You can hurt/kill a lot of people very quietly with a knife, leaving most of the school none the wiser. They're easier to conceal, easier to make from non-metallic substances (and thus can be easier to sneak past metal detectors.) I imagine people would be a lot less concerned about a knife collection than a gun collection, etc etc.

I don't disagree with your comment about someone not being ready for a firearm. However, I think that the argument that we're not recognizing the dangers of "gun free" zones as potential targets (by at least one statistic, 94% of mass shootings in the US happen in a "gun free" zone) and mitigating that danger in a meaningful way actually supports your point about syncophantic AIs better.

You really think having access to ChatGPT is as dangerous as giving everyone a nuclear weapon?

Every previous time someone says something like this, I've looked at the original blog post again just to make sure I didn't miss something.

OpenAI's own words for GPT-2 never read like a boast to me. Caution, not a boast.

News may have bigged it up, but news does that for everything.

I mean just compare it to the GPT-5 release: they're not shy or subtle when they actually want to boast.

If we had a systems architecture in an engineering project that kept leading to bad results, we would consider rearchitecting how said system is run.

I believe it is important to make it clear to people that capitalism, too, is just a (imo deeply flawed) systems design. It is not a natural constant. If we find it leads to negative outcomes (which I strongly believe), we can and should talk about rearchitecting.

don't worry, I associate Marxism with REAL socialist regimes, like Cambodia

Nothing wrong with alternatives, but for that we need to let go of "let's make it just like the thing that also fails". We also need to acknowledge that socialism assumes that humans are fundamentally good, and that ignores that many are fundamentally not. We need to acknowledge that the core idea of socialism, the common good, is ill-defined. (Just like, while we're on AI, ideas of alignment on one value system suffer from that ill definition)

So, no, you don't get voted to hell for saying that the problem is capitalism. You do get downvoted for sloppy thinking if you simply propose "socialism will save us" because it's long established that in this form, the statement is not true.

Ultimately, the issue is that HN is not made for that discussion. You'd need nuanced political debate, and the internet does not seem conducive for that. That's the second reason for downvotes - HN is not interested in contentious political debate, for better or worse.

But no, it's not because we just all immediately think of the USSR/Maoist China.

So far, we've been pretty good at identifying where capitalism just doesn't work and then bandaiding that area with legislation.

For example, capitalism has no solution for disability. In a capitalist system, everyone must work. Those who don't work should, then, be filtered out - die.

But that's obviously bad, so bandaid - SSI. We say, if you're disabled, we'll give you little socialism, so you don't die. We'll put a communal responsibility on keeping you alive because that's what's most beneficial for society.

There's no rule anywhere saying we have to just let AI make the world a worse place. No, WE decide that, even in a capitalist system.

Just not a very good argument.

> Is being dead better, or is having complete privacy better? Or does it depend?

I know you're being provocative, but this feels like a false dichotomy. Mental health professionals are pro-privacy AND have mandatory reporting laws based on their best judgement. Do we trust LLMs to report a suicidal person that has been driven there by the LLM itself?

LLMs can't truly be controlled and can't be designed to not encourage mentally ill people to kill themselves.

> Mentally ill people trying to use an LLM for therapy

Yes indeed this is one of the core problems. I have experimented with this myself and the results were highly discouraging. Others that don't have the same level of discernment for LLM usage may mistake the confidence of the output for a well-trained therapist.

No, they can run quantized versions of those models, which are dumber than the base 30b models, which are much dumber than > 400b models (from my use).

> They are a little bit dumber than the big cloud models but not by much.

If this were true, we wouldn't see people paying the premiums for the bigger models (like Claude).

For every use case I've thrown at them, it's not a question of "a little dumber", it's the binary fact that the smaller models are incapable of doing what I need with any sort of consistency, and hallucinate at extreme rates.

What's the actual use case for these local models?

That's why lettuce is the most deadly food on your table. You are far more likely to get serious food poisoning from it than from anything else.