The 'Decline Effect' and Scientific Truth

Friday, May 13, 2011


Surprising and exciting scientific findings capture our attention and captivate the press.  But what if, at some point after a finding has been soundly established, it starts to disappear?  In a special collaboration with Radiolab we look at the 'decline effect' when more data tells us less, not more, about scientific truth.

Correction: An earlier version of this short incorrectly stated that Jonathan Schooler saw the effect size of his study fall by 30% on two different occasions. In fact, he saw it fall by that amount the first time he repeated the study and saw a general downward trend thereafter. The audio has been adjusted to reflect this fact.


Correction: An earlier version of this short incorrectly attributed a statement to Jonathan Schooler’s advisor. The statement was actually made by his colleague. The audio has been adjusted to reflect this fact.

Comments [21]

Quentin Hammonds

Wow I wasn't paying any attention to what she was saying becaues of the nice beat that was playing while she was talking. Well I already know what people are calling the truth is not the truth. You call it scientific, I call it BULLSHIT. We are living in a world where we are bios. Some people believe what they here and others believe what they see. To be honest I don't care becaues I can't believe what people are saying because they could be playing me and I can't believe what I see because it could be a magic trick.

May. 19 2011 09:00 AM
JS from Washington, DC

The decline effect I'd put my money on is that most of your listeners are a little dumber now than they were last week. Although I do share several previous commenters' surprise that the Uncertainty Principle wasn't mentioned; lay people can rarely resist the urge to loudly and publicly misunderstand quantum physics. Perhaps if you had interviewed a Kabbalist, as due diligence would suggest....

May. 18 2011 11:50 AM
Wayne Robey from Lafayette Indiana, USA

I would offer a hypothesis for this decline in effect. It would have to be tested by reference to the experiments and the experimenters. I am surprised that it has not been offered here or in the original report. The experimenter is part of the experiment. While double blind methods are intended to remove this effect, the psychic effect of the desire of the originator of the experiment still reaches it's performance even when clearly recognized physical means have been excluded. The existence of various types of psychic ability have been conclusively demonstrated. In this case, I propose that the intensity of the desire of those involved in the experiment to reach a specific outcome is the influencing factor. Did the first person to look at the symmetry of bird tale feathers as an indication of desirability of the bird as a mate expect the result found? If he did not, he probably would not have done the experiment. Then the next person may have doubted the first result rather than being excited by it. If he was excited by it, his emotional intensity was probably less than the first. This general pattern of desired outcome would give the reversion phenomena reported. If the Heisenberg's Uncertainty Principal is involved, it could provide the amount of uncertainty available for psychically transferred desire to act, but this is pure speculation on my part.

May. 17 2011 06:14 PM
Barry Zorman from United States

The proposed idea that changes in fundamental physics over time could explain inconsistencies in the results of human and animal behavior studies seems very unlikely. Observations on simple systems, such as hydrogen atoms on Earth and across the time delays of distant objects in space, have placed narrow limits on such changes.

May. 17 2011 02:56 PM
Danielle Babiarz from Chicago

I too immediately thought of Heisenberg's Uncertainty Principal (aka Observer Effect) and was surprised there was no mention of it. Additionally, at the end there was a statement to the effect of "we know nothing." Isn't this the basis of Socratic or any Subjective philosophy? Again no mention of that connection. The sound effects are distracting and seem cheesy, but I'll admit, I found the piece fascinating.

May. 17 2011 01:29 PM
L. Edgar Otto from Eau Claire, WI

Yes, there is a lot of philosophy here. But considering the latest science, the new physics really, has reached a lot of foundational problems and unexpected directions- this author has made some brilliant deductions that seem to have relevance to our daily lives and honest experiences.

Those who say this is not science from my perspective are declining into an old school and are getting rather obsolete. I would as the author this: If my roommate takes Prozac and it has diminishing effects and he has a mental problem with diminishing effects then, although things can be cured and not grow worse it take infinite time to do so.

The author btw made no claims this was a formal scientific theory. But deeper questions have been presented and looked at by scientists who should know better than to show decline on what they think they know is not there when they feel they know everything.

May. 17 2011 10:12 AM

I am sure electrons have a collective consciousness. Too bad OTM decided not to have available an expert on methodological issues and preferred to spread FUD on science as a whole. So called scientific truths are the current accepted theories. They replaced previous theories and will be replaced by better ones. If you don't understand this basic aspect of science as work in progress, you are bound to miss its lasting value. In the Millikan's experiment case as well in the coin toss example that was mentioned in the show, in the long run we are approaching the true mass of the electron, the true probability of heads. And that's a wonderful thing that can't be said about many other forms of knowledge, be them opinion, religion, myth, whatever. It's having a workable notion of progress and a (mostly) non-violent way to achieve consensus, in the long run.

May. 16 2011 04:52 PM
Stuart Filler from Detroit

If the phenom is complex, the stuff described in James Gleick's book CHAOS: MAKING A NEW SCIENCE seems to bear on it. The description and explanation by the scientist theyinterviewed on the program have my full confidence; I take his reference shifting reality in the spirit of pure speculation in which he offers it. He's describing a feeling and what it makes him think of and he all but shouts out that the number of variables might be closer to eighty than to eight. (Freud's little essay titled "The Uncanny [unheimlich]" might be worth a read at the same time.) But Gleick's description of a kind of analytic revolution in the 60s-70s-80s, which to my lousy recollection speaks of fluid motion and weather and genetics and I suspect society and mentation, seems to have everything to do with it. The nonlinearity of the problem stares out at us. Heisenberg on observing the microcosm gives us a metaphor rather than a description.

May. 16 2011 12:19 PM
joel Matlock

There are three types of lies

Damn lies

God Damn lies

And statistics

May. 16 2011 08:45 AM
Kenneth BraIterman from Concord NH

As the only wire editor at a small newspaper, all the "study" stories came through me. If the study came from a reputable journal or institution, and made a provocative headline and lead, I used it as a story or a brief. I had ho time or training to evaluate the methodology or seek other comments. So we had studies that showed children never lie about child abuse, and we were having an epidemic of missing, presumably abducted kids.

A federal study, with policy implications, showed that there was no evidence that new, expensive anti-psychotic drugs were more effective than old inexpensive ones. The Washington Post's medical reporter wrote a nuanced 20-inch story about it with expert comments. AP sent out a 10-inch dumbed down version. Local TV reporter, "A new federal study shows anti-psychotic drugs don't work." End of story.


May. 15 2011 04:13 PM
Andy from Atlanta

As I was listening to this report the Heisenberg Uncertainty Principle immediately jumped into my head. So I was absolutely shocked to hear it said that there was nothing like that being discussed in physics. That the act of observation affects the observed is a fundamental principle of modern physics - a lesson I learned back in high school and was more fully developed in college courses.

What I'd really like to hear would be a panel discussion on this topic between quantum physicists, the social scientists from this story, and (believe it or not) perhaps a few theologians, especially Kabbalists.

(Real Kabbalists, not the pseudo, "pop" Kabbalah, please. Why them? I've read a few essays that indicate some Kabbalistic writings are very similar to quantum physics, only without the scientific vocabulary. I'd love to hear their insights on this.)

May. 15 2011 03:06 PM
David from King of Prussia, PA

Robert Millikan. The charge of the electron. Nobel Prize 1923. Millikan's original number was low (although within experimental error of the now-accepted number.) Over time, as other physicists repeat the experiment, the number slowly approaches its now-accepted value.

The fact that it slowly approached the now-accepted value is strange. One would think that later experiments would have results on either side of the now-accepted value, with equal chance.

The fact that they didn't is now generally seen in the scientific community as a series of cases of confirmation bias. In other words, knowing Millikan's value, the next experimenter chooses to discard data points that are too far from Millikan's value, thinking that some sort of error occurred, thus skewing the data toward a lower value, yet still higher than Millikan's. And so on with each succeeding experimenter until the now-accepted value is arrived at.

Nothing spooky about it.

May. 15 2011 12:54 PM

Far-fetched conclusions, spurious logic, tacky sound effects...what is this, talk radio?

May. 15 2011 12:01 PM
Karen Jeffrey from Chicago

A possible explanation of the issue described in your piece is bias by the researchers. A researcher's job is to make new discoveries and many would like to make that big discovery. A big discovery is not only beneficial for the individual, but also for their whole field. Even in precise disciplines like Math, unintentional mistakes and oversights are found all the time during the review process. Some of the the errors and oversights are a consequence of the researcher's enthusiasm over their accomplishment. This is just human nature. In fields where it is more difficult to check the researcher's work, there will be mistakes due to enthusiasm that make it through the review process. Over time, there will be less enthusiasm over the results and therefore less bias towards proving the results correct.

May. 15 2011 08:30 AM
Mark Reagn from Iowa City

A much less spooky explanation for the "decline effect" is that experimental design improves because of knowledge gained during each iteration of a particular type of study.

May. 15 2011 08:08 AM
Tim Grant from New Hampshire

Casting Pearls...

May. 14 2011 04:04 PM
Miles Fidelman from Newton, MA

What utter lack of understanding of basic physics and mathematical logic.

First off, the Heisenberg Uncertainty Principle makes it absolutely clear that observation DOES change what's observed, and at atomic levels tells us precisely how much.

Second, Godel demonstrated that any logical system has inherent paradoxes.

Between the two, they establish limits to our knowledge, and the mutability of knowledge. Given that the story talks about memory and psychological effects, and what's known about the placebo effect, observer bias, and the proven unreliability of eye witness testimony - the result is a story that's utter garbage.

May. 14 2011 03:15 PM
Toby Saunders from Georgia

What utterly perverted nonsense... it's stupid work like this piece which keeps me from donating to NPR. Those Radio Lab guys look for stories with some 'spooky' result, which is actually entirely reasonable and to be expected, and gets some crack pot excuse for an expert to say, "maybe it's the ether leaking" and the guys laugh, throw in some cheap sound effects and encourage people to be cynical about truth. The effect described happened in a few studies while in others there is no such effect... Jad even admitted in the end of the piece that they were simply filtering out the truth in search of 'spooky' and 'interesting' stories. That is what Radio Lab is all about after all: making a joke out of science and encouraging stupidity. Thanks to this bogus piece, a huge portion of the listeners are now saying, 'well, global warming is myth, evolution is myth... it's fine for men to abuse women... slavery was fine at one point... plate tectonics is bogus, there is no truth.' -Pernicious.

May. 14 2011 03:05 PM
Larry Curcio from Pittsburgh, PA

As Laurence Drell pointed out, patterns of publication bias may very well account for the slow regression to the mean. Such a pattern has already been implicated in the studies on antidepressants that were referenced.

There is also an effect I call the "Idiot effect" . You see it on internet forums a lot. Someone notes that the common sense answer to some question is A, but clever analysis demonstrates that it is actually B. Discussion occurs and somewhere along the line, a latecomer cluslessly protests the result is really A, as anyone with common sense would know.

At this point, explanation occurs, and the clueless latecomer becomes clueful. Just about then, another clueless latecomer appears. After an iteration or two, the clueless prevail, and the insight is defeated.

The same process can be found in research studies - particularly those whis subtelties associated with experimental procedure.

Your commentator's own results? Can't explain them.

May. 14 2011 02:06 PM
Jamie Scott from NYC

What a fascinating program! Thanks for providing.

In a brain numbed by pollen, my neurons are firing furiously at thoughts of the "ether" and the affect we have on each other, ourselves, and all living (and perhaps even non-living) beings simply by observing or (perhaps) merely considering them, or something possibly related to them, or something they never thought of at all.

May. 14 2011 12:07 PM
Laurence Drell, MD from Washington, DC

There is another piece of information about the skewing of meta analysis. There are fewer published reports of negative results. We have a tendency to report positive new changes.... if the test "isn't working out" often it may not be completed or the researcher has doubts about his method or thinks it is of less value and does not publish it when in fact more information is needed. Its just how to pick and choose what is important information.

Someone famous said: Genius is knowing what NOT to pay attention to.

(It was also said that some research showed that older people are better at this… but it wasn’t tested repeatedly. So I am just going to believe that there is something positive about aging and not just those declining functions.)

Laurence Drell, MD

May. 14 2011 08:37 AM

Leave a Comment

Email addresses are required but never displayed.