New Benchmark Shows Top Llms Struggle in Real Mental Health Care

Yeah, initially I wrote this test/benchmark harness because I wanted to compare multiple different prompts for the same tasks and the same model, but obviously eventually grew out from there. But it still has the prompts at core, and I re-run everything whenever something changes, or I add new models to it.

> How many times did you run each prompt?

It's structured in a way of Category > Task > Case and that's mixed with a list of Prompts for each Task, then each Case runs with each of the Prompts. So I guess you could say that each prompt gets "exercised" the number of existing cases that exists for the Task they're in.

> Did you use the same rubric to score each experiment?

I'm not sure if you mean something specific by "rubric" (I'm not from academia), but they're all pretty much binary "passed" or "not passed". The coding ones are backed by unit tests that were failing, and after test case must pass without being changed, translation ones backed by (mostly) simple string checking, and so on. I don't have any tasks or cases that are "Rate this solution from 0-10" or similar.

LLMs have room for improvement (we show that their scores are medium-low on several dimensions).

Maybe the average human also has lots of room for improvement. One thing does not necessarily depend on the other.

the same way we can say that LLMs still have room for improvement on a specific task (lets say mathematics) but the average human is also bad at mathematics...

We don't do any claims about human therapists. Just that LLMs have room for improvement on several dimensions if we want them to be good at therapy. Showing this is the first step to improve them

Posted 6 months after the post and all the rest of the comments. It's some kind of SEO manipulation. That reddit thread ranked highly in my Google search about Betterhelp being bad, so they're probably trying to piggyback on it.

    127. A technological advance that appears not to threaten freedom often turns out to threaten it very seriously later on. For example, consider motorized transport. A walking man formerly could go where he pleased, go at his own pace without observing any traffic regulations, and was independent of technological support-systems. When motor vehicles were introduced they appeared to increase man’s freedom. They took no freedom away from the walking man, no one had to have an automobile if he didn’t want one, and anyone who did choose to buy an automobile could travel much faster and farther than a walking man. But the introduction of motorized transport soon changed society in such a way as to restrict greatly man’s freedom of locomotion. When automobiles became numerous, it became necessary to regulate their use extensively. In a car, especially in densely populated areas, one cannot just go where one likes at one’s own pace one’s movement is governed by the flow of traffic and by various traffic laws. One is tied down by various obligations: license requirements, driver test, renewing registration, insurance, maintenance required for safety, monthly payments on purchase price. Moreover, the use of motorized transport is no longer optional. Since the introduction of motorized transport the arrangement of our cities has changed in such a way that the majority of people no longer live within walking distance of their place of employment, shopping areas and recreational opportunities, so that they HAVE TO depend on the automobile for transportation. Or else they must use public transportation, in which case they have even less control over their own movement than when driving a car. Even the walker’s freedom is now greatly restricted. In the city he continually has to stop to wait for traffic lights that are designed mainly to serve auto traffic. In the country, motor traffic makes it dangerous and unpleasant to walk along the highway. (Note this important point that we have just illustrated with the case of motorized transport: When a new item of technology is introduced as an option that an individual can accept or not as he chooses, it does not necessarily REMAIN optional. In many cases the new technology changes society in such a way that people eventually find themselves FORCED to use it.)

    128. While technological progress AS A WHOLE continually narrows our sphere of freedom, each new technical advance CONSIDERED BY ITSELF appears to be desirable. Electricity, indoor plumbing, rapid long-distance communications ... how could one argue against any of these things, or against any other of the innumerable technical advances that have made modern society? It would have been absurd to resist the introduction of the telephone, for example. It offered many advantages and no disadvantages. Yet, as we explained in paragraphs 59-76, all these technical advances taken together have created a world in which the average man’s fate is no longer in his own hands or in the hands of his neighbors and friends, but in those of politicians, corporation executives and remote, anonymous technicians and bureaucrats whom he as an individual has no power to influence. [21] The same process will continue in the future. Take genetic engineering, for example. Few people will resist the introduction of a genetic technique that eliminates a hereditary disease. It does no apparent harm and prevents much suffering. Yet a large number of genetic improvements taken together will make the human being into an engineered product rather than a free creation of chance (or of God, or whatever, depending on your religious beliefs).

    129. Another reason why technology is such a powerful social force is that, within the context of a given society, technological progress marches in only one direction; it can never be reversed. Once a technical innovation has been introduced, people usually become dependent on it, so that they can never again do without it, unless it is replaced by some still more advanced innovation. Not only do people become dependent as individuals on a new item of technology, but, even more, the system as a whole becomes dependent on it. (Imagine what would happen to the system today if computers, for example, were eliminated.) Thus the system can move in only one direction, toward greater technologization. Technology repeatedly forces freedom to take a step back, but technology can never take a step back—short of the overthrow of the whole technological system.

    130. Technology advances with great rapidity and threatens freedom at many different points at the same time (crowding, rules and regulations, increasing dependence of individuals on large organizations, propaganda and other psychological techniques, genetic engineering, invasion of privacy through surveillance devices and computers, etc.). To hold back any ONE of the threats to freedom would require a long and difficult social struggle. Those who want to protect freedom are overwhelmed by the sheer number of new attacks and the rapidity with which they develop, hence they become apathetic and no longer resist. To fight each of the threats separately would be futile. Success can be hoped for only by fighting the technological system as a whole; but that is revolution, not reform.

I've heard people say the same thing ("LLMs don't need to be as good as an expert to be useful, they just need to be better than your best available expert"), and I also know that some people assume that LLMs are, by default, better than nothing. Hence my comment.

I'm citing a medical doctor and clinical psychologist with decades of experience who has recorded a hundred hours of training podcasts, and linking actual therapy sessions that you can listen to, and you're saying "no it isn't" with nothing to back that up except "you reckon it isn't".

> "the idea that therapy is always easy "someone gives you a bit of information, like a switch was flipped, done": No, it isn't. Habits that are learned over decades aren't that easily changed."

Nobody said it was always easy. Yes they are. People try to quit smoking cold turkey three times a week for five years. Then they read Alan Carr's "The Easy Way to Quit Smoking" and then they don't want to smoke anymore and there's no talk of "quitting" because they aren't smokers and non-smokers don't need to quit. With the right understanding, the viewpoint flips and the mind is changed. Same with overweight people who try dieting for years and then have a health scare and sometimes that switches it so they change instantly (and sometimes it doesn't). Most things won't easily change a habit, like most changes in code won't fix a specific bug. But some changes can, and we should look for them.

> "Like a gym session, if it's always easy then you're not doing the work."

This is some Puritanical suffering-culture, or some one-upmanship manliness culture. This is the reason I mentioned the int32 to int64, sometimes it might require searching to find insight, but there's no points for searching harder and trying harder, if you can have the same insight in two hours instead of two years, that's good not bad. The Universe doesn't give points for "doing the work" (I suspect one of your beliefs does).

> "Reassurance can work, but IMHO you'll be back soon enough, as the root cause hasn't been addressed"

This is strawmanning or not understanding; this is addressing the root cause and not reassurance; the step of "paradoxical agenda setting" gets to the heart of why reassurance doesn't work. Someone who says "I lost my job, I didn't work hard enough, I am a loser" doesn't get helped by reassuring them that they are not a loser. It might be that they have a deep-seated value that "hard work is good" and they are getting into a human race condition where "reassurance that you aren't a loser" goes to "if I can think I'm a winner even when I don't try, then laziness can be winning, and I don't want that. I won't go there. So I reject the reassurance and return to my belief that if I am a loser".

The fix is trace that, and find a working technique to unjam it. Which is case-by-case individual, but somewhere like "I understand that feeling like a loser is the flip-side of my belief that hard work is good. I actually want to keep the feeling because that's one of the things which guides me and pushes me to work hard, and I value that. I can't get rid of one without the other. What I've done is try to grab tightly to one side of this (hard work is good) and push away the other side (I'm a loser because I didn't work hard) but they're the same thing, so grabbing it hard is pulling it back. Until it's dialled up to 11 and the brain is shouting "LOSER" all the time while the person is pushing it away. And it doesn't make sense to judge a whole self as a winner or loser, people have lots of components. It doesn't make sense to say "I didn't work hard" as there were times at work when I did work hard. So actually I want to keep the feeling "I am a loser if I don't work hard" because it encourages me to work harder (which I value). I want it dialled down to 2 instead and just focused on small areas of work and life, not judging all of me all the time".

and with understanding, finally listening to the thought that's been running around, accepting it as a thing you asked for, that reminds you of something else, it 'suddenly' calms down. Acknowledged. Never to come back.

> "if chugging some simple agreeable affirmations are all that you need, by all means listen to the LLM. The sycophancy machine can do that."

Can you see this as the typical HN cynicaler-than-thou putdown? Maybe the reader will think you're a really tough C++ programmer who only values science and muscles, instead of a woke hippy gullible loser? But you don't look tough for changing "therapy skills developed over decades" into "simple agreeable affirmations" you just look like you don't understand and are embarrassed.