Can Psychology Studies Always be Trusted? What Replicating Research Found

After a few high-profile cases were reported of questionable data in psychology related studies, a year long effort has been made to reproduce the results of some of the most popular published studies.

Included in reports of bogus data reporting was a well-known and often reported psychologist from Tilburg University, who was found to be falsifying data and entire experiments, as reported by the New York Times in 2011.  After a few more stories like this were widely reported, the Reproducibility Project began.

Coordinated by the Center for Open Science, The University of Virginia and a team of more than 250 researchers who were mostly volunteers, the project identified 100 studies from 2008 and redid the experiments to see if the results could be repeated.  The replicated studies were done in close collaboration with the original study authors.

As reported in the journal Science, the project’s results show no evidence of outright fraud or false data, but that the evidence used for most published studies was not as strong as originally claimed.  Overall, more than 60 of the 100 studies did not exactly reproduce results when tested by the reproducibility team.

As reported in the  New York Times, examples of studies that did not have replicated results include:

• A study on free will
• A study on the effect of physical distance on emotional closeness
• A study on the mate preferences of women

The team sites possible explanation for the rising discrepancy as the hypercompetitive culture in science, with the goal being a splashy result that gets reported widely with possible viral results, without incentive to replicate findings.   Also, given the increase of scientific reporting used in the media, results may also be over-simplified for easier reporting to laymen readers.

The Center for Open Science has indicated this may be a problem across all the sciences, not just psychology, and they have begun a similar reproducibility project in the field of Cancer biology and research.

What experts in the field and professionals at medical journals hope to stress is that while this may be long overdue, this is now the beginning of a focus on reproducibility for the entire industry from universities to the journals reporting results – but much more research needs to be done and standards in replication should be set.

The team of researchers involved in the reproducibility project also acknowledge other differences between the original and replication studies could explain the different or weaker findings – including differences in design and context.