Famous Cognitive Psychology Experiments That Failed to Replicate

I think one would wish the famous ones to be more often replicable.

Thanks for providing the reference, that's useful context. Those are awful replication rates, worse than a coin flip. Sounds like the OP can add their own introduction to their list. From the introduction:

> Most results in the field do actually replicate and are robust[citation needed]

How interesting would it be if every PhD thesis had to have a "replication" section, where they tried to replicate some famous paper's results.

Sure, dump it on the lowest level employee, who has the least training and the most to lose. Punish them for someone else's bad research. Grad school already takes too long, pays too little, and involves too much risk of not finishing. And it doesn't solve the problem of people having to generate copious quantities of research in order to sustain their careers.

Disclosure: Physics PhD.

>some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair.

I don't like Giancotti's claims. He wrote: >This post is a compact reference list of the most (in)famous cognitive science results that failed to replicate and should, for the time being, be considered false.

I don't agree with Giancotti's epistemological claims but today I will not bloviate at length about the epistemology of science. I will try to be brief.

If I understand Marco Giancotti correctly, one particular point is that Giancotti seems to be saying that Hagger et al. have impressively debunked Baumeister et al.

The ego depletion "debunking" is not really what I would call a refutation. It says, "Results from the current multilab registered replication of the ego-depletion effect provide evidence that, if there is any effect, it is close to zero. ... Although the current analysis provides robust evidence that questions the strength of the ego-depletion effect and its replicability, it may be premature to reject the ego-depletion effect altogether based on these data alone."

Maybe Baumeister's protocol was fundamentally flawed, but the counter-argument from Hagger et al. does not convince me. I wasn't thrilled with Baumeister's claims when they came out, but now I am somehow even less thrilled with the claims of Hagger et al., and I absolutely don't trust Giancotti's assessment. I could believe that Hagger executed Baumeister's protocol correctly, but I can't believe Giancotti has a grasp of what scientific claims "should" be "believed."

The idea isn't that it is easier to do things when not tired. It is that you specifically get tired exercising self control.

I think that can be subtly confused by people thinking you can't get better at self control with practice? That is, I would think a deliberate practice of doing more and more self control every day should build up your ability to do more self control. And it would be easy to think that that means you have a stamina for self control that depletes in the same way that aerobic fitness can work. But, those don't necessarily follow each other.

If you were to meet a "normal" person, would you interpret that as meaning "perpendicular" or as meaning "the kind of person that doesn't look at everything like it's a mathematical expression"?

> in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing.

The normal distribution predates the general factor model of IQ by hundreds of years.[0]

You can try other distributions yourself, it's going to be hard to find one that better fits the existing IQ data than the normal (bell curve) distribution.

[0] https://en.wikipedia.org/wiki/Normal_distribution#History

I took an IQ test as a high school student, and one of the subtests involved placing a stack of shuffled pictures in chronological order. I had one series in the incorrect order, because I had no understanding of the typical behavior of snowfall. The test proctor said almost everyone she tested mixed that one up, because it doesn't snow in the area where I live.

I have no doubt that IQ tests reproducibly measure the test takers ability to pass tests, as well as to perform in a society that the tests are based on.

I think it's disingenuous to attribute IQ to intelligence as a whole though, and it is better understood as an indicator of cultural intelligence.

I would expect that, for cultures who's members score below average on IQ tests from the US, an equivalent IQ test created within that culture would show average members of that culture scoring higher than average members of US culture.

What exactly are they meant to replicate other than other IQ tests? They don't make a statement about anything that is falsifiable, other than that if you give somebody who scores high on a test carefully designed and tested to match the results of previously given IQ tests when given to the same people, they'll tend to match the results that those people will get on other tests that were calibrated in the exact same way.

If you're trying to say they replicate over the lifetime of the same person, I've had a 15 point swing between tests, out of the few I've taken. What did stay constant for me from age 10 to age 40 was my Myers-Briggs test (my dad was a metrics obsessive), and that's obvious horseshit. Consistency doesn't mean you're measuring what you claim to be measuring.

edit: if it matters, scores were between 137 and 152, so exactly an entire standard deviation. That's like the difference between sub-Saharan and European that racists are always crowing about, but in the same person. IQ doesn't even personally replicate for me.

Survivorship bias. You can easily make someone's IQ test not replicate. (Hit them on the head really hard.)

From working in industry and rubbing shoulders with CS people who prioritize writing papers over writing working software I’m sure that in a high fraction of papers people didn’t implement the algorithm they thought they implemented.

Nowadays everyone publishes their code. There’s typically a project page on github.io, a paper on arxiv.org, and a public repo.

And most medical studies. It's just as bad as social psych, if not worse, because there's real money at stake in churning out new drugs.

I'm still amazed that wikipedia doesn't have redirect away from its mobile site

I thought you were pointing out some bias by comparing the research to previous research from the same authors. It took me far too long to realize that the experiment was from 100 years ago, and you were pointing out that the names were coincidentally the same.

Um, actually I’d say it is the responsibility of all scientists, both professional and amateur, to point out falsehoods when they’re uttered, and not an act of smugness.

As someone who's part of a startup (hrpotentials.com) trying to bring truly scientifically valid psychological testing into HR processes .... yeah. We've been at it for almost 7 years, and we're finally at a point where we can say we have something that actually makes scientific sense - and we're not inventing anything new, just commercializing the science! It only took an electrical engineer (not me) with a strong grasp of statistics working for years with a competent professor of psychology to separate the wheat from the chaff. There's some good science there it's just ... not used much.

"they are AWFUL at stats."

SF author Michael Flynn was a process control engineer as his day job; he wrote about how designing statistically valid experiments is incredibly difficult, and the potential for fooling yourself is high, even when you really do know what you are doing and you have nearly perfect control over the measurement setup.

And on top of it you're trying to measure the behavior of people not widgets; and people change their behavior based on the context and what they think you're measuring.

There was a lab set up to do "experimental economics" at Caltech back in the late 80's/early 90's. Trouble is, people make different economic decisions when they are working with play money rather than real money.

https://en.wikipedia.org/wiki/Stereotype_threat shows up in this list as not replicated, however, it is one of the most studied phenomena in psychology.

>meta-analyses and systematic reviews have shown significant evidence for the effects of stereotype threat, though the phenomenon defies over-simplistic characterization.[22][23][24][25][26][27][28][9]

Failing to reproduce an effect doesn't prove it isn't real. Mythbusters would do this all the time.

On the other hand, some empires are built on publication malpractice.

One of the worst that I know is John Gottman. Marriage counselling based on 'thin slicing'/microexpressions/'Horsemen of the Apocalypse'. His studies had been exposed as fundamentally flawed, and training based on his principles performed worse than prior offerings, before he was further popularized by Malcolm Gladwell in Blink.

This type of intellectual dishonesty underlies both of their careers.

https://en.wikipedia.org/wiki/Cascade_Model_of_Relational_Di...

https://en.wikipedia.org/wiki/The_Seven_Principles_for_Makin...

https://www.gottman.com/blog/this-one-thing-is-the-biggest-p...

> And it isn't a skill issue as much as a cultural one. They teach it wrong

It's definitely a skill issue then

I was very surprised at how many statistical methods are taught in undergraduate psychology. Far more statistics than I ever touched in engineering for sure. Yet the undergrads really treated statistics as a cookbook, where they just wanted to be told the recipe and they'd follow it. Honestly they'd have been better off just eyeballing data and collaborating with statisticians for the analysis.

The problem with a lot of the “higher free variable” sciences like psychology, ecology and sociology etc

Is they are the ones who need to be at the bleeding edge of statistics but often aren’t

They absolutely need Bayesian competitive hypothesis testing but are often the least likely to use it

I read in a study that it takes 10,000 hours to become proficient in statistics /s

There’s surely irony here

Reminds me of Feynman’s Cargo Cult Science speech:

https://people.cs.uchicago.edu/~ravenben/cargocult.html

We rack up quite a lot of awfulness with eugenics, phrenology, the "science" that influenced Stalin's disastrous agriculture policies in the early USSR, overpopulation scares leading to China's one-child policy, etc. Although one could argue these were back-justifications for the awfulness that people wanted to do anyway.

i'm struggling to imagine many negative effects on society caused by the specific papers in this list

The most obvious one is the breakdown of trust in scientific research. A frequent discussion I would have with another statistics friend of mine was that that anti-vax crowd really isn't as off base as they are more popularly portrayed and if anything, the "trust the science!" rhetoric is more clearly incorrect.

Science should never be taught as dogmatic, but the reproducibility crisis has ultimately fostered a culture where one should not question "established" results (Kahneman famously proclaimed that one "must" accept the results of the unbelievable priming results in his famous book), especially if that one is interested in a long academic career.

The trouble is that some trust is necessary in communicating scientific observations and hypothesis to the general public. It's easy to blame the failure of the public to unify around Covid as based around cultural divides, but the truth is that skepticism around high stakes, hastily done science is well warranted. The trouble is that even when you can step through the research and see the conclusions are sound, the skepticism remains.

However, as someone that has spent a long career using data to understand the world, I suspect the harm directly caused by the wrong conclusions being reached is more minimal than one would think. This is largely because, despite lip service to "data driven decision making", science and statistics very rarely are the prime driver of any policy decision.

I imagine it's comparable to the damage done when policies are set that are not based on studies.

Let's be candid: Most policies have no backing in science whatsoever. The fact that some were backed by poor science is not an indictment of much.

I once did a corporate internal management course that was filled with pseudoscience bullshit. I imagine the impact of that course on the company's productivity was net negative. I'm sure lots of orgs have similar courses.

Learning styles have also been debunked for decades though they continue to be used in education. I saw an amusing line in an article that said 90% of teachers were happy to continue using them even after accepting they're nonsense.

And that's just theories that have been debunked (i.e. proven wrong).

In many cases--longitudinal studies are an example, but not the only one--that's not feasible. And it's often expensive--who would pay for it?

Agreed, and the independent lab should be chosen by the publisher, and be kept secret until results are in.

Yeah, the marshmallow one taught me to have patience and look for the long returns on investments of personal effort.

I think there may be something to a few of these, and more may need considering regarding how these are conducted.

Let’s leave open our credulities for the inquest of time.

You'd think it's so far in the past that it isn't even considered, but Zimbardo was elected president of the American Psychological Association in 2002, which wasn't all that long ago.

I have a draft of a blog post on this. Originally I was going to write about how cortisol isn't always bad, or good, it's just a chemical in us. But then I started noticing the pattern you point out here where I'm not sure anyone uses the cortisol argument in good faith. Everyone who brings up cortisol is usually trying to sell you something

Has anyone tried to reproduce it? Good luck convincing an ethics review board to let you try that again.

Meanwhile, it’s been reproduced “in vitro” in numerous episodes of atrocity, e.g. Abu Ghraib…

Yes, the APA certainly has a lot to answer for in their history. The Guantanamo Prison torture scandal is still fresh in my memory.

https://www.democracynow.org/2007/8/20/apa_members_hold_fier...

I recently read the lifework book of a nobel prize winning psychologist and it was full of these disproven experiments. As a non-psychologist my trust in the experts is extremely low.

It’s also self-referential because there is no objective measure of “racism,” so how can you even measure whether someone is “more racist” based on reaction time to stereotypical stimuli?

“No objective measure” pretty much sums up the whole field, to be honest. I started on a CS & Psych double major, did about eight psych courses, and then decided it was mostly a joke once I got to the quantitative portions. But those courses were very useful for general life skills. Developmental psychology in particular was packed with dense lessons about how we learn as children… social psych was a good overview of all the “well-known” experiments… etc.