Cognitive Sciences Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Usually an article goes through a peer-review process before it is published, and from what I here it is quite common that the reviewers "demand" some changes.

  • Are there guidelines for peer- reviewers, especially as the methodological side of things is concerned?
  • Are  "methodology specialists" incorporated in the process? 
share|improve this question
up vote 3 down vote accepted

I get the impression that good journal editors will get at least one reviewer who is skilled in the methodology used in the paper. The importance of this reviewer role would presumably vary with the statistical or other methodological complexity of the paper.

That said, reviewing is well known to be imperfect particularly when it comes to checking all the possible smaller errors that can be made. Errors in statistical analysis and reporting are wide-spread in published articles in psychology. For example, Bakker and Wicherts (2011) did a review, where to quote the abstract

we checked the consistency of reported test statistics, degrees of freedom, and p values in a random sample of high- and low-impact psychology journals... On the basis of 281 articles ... [we estimate that] around 18% of statistical results in the psychological literature are incorrectly reported. Inconsistencies were more common in low-impact journals than in high-impact journals. Moreover, around 15% of the articles contained at least one statistical conclusion that proved, upon recalculation, to be incorrect; that is, recalculation rendered the previously significant result insignificant, or vice versa.


  • Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666-678. FULL-TEXT
share|improve this answer

This does not really answer your question (because it doesn't deal in detail with methodological issues), but there is a nice journal article that deals with problems of the peer review process in general by evaluating "great scientific works of the past" from the perspective of current social sciences:

The reference list has some articles on statistical/mathematical methodology questions.

In another interesting article, Gerd Gigerenzer shows how p-values are misinterpreted in current research and null hypothesis testing is made a false requirement for the acceptance of papers by major journals:

A Google search for "peer review guidelines" brings many hits from publishers and institutions. Maybe some of them deal with methodological issues in more detail.

share|improve this answer
Another interesting paper on the same topic: Niewenhius et al (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature Neuroscience 14, 1105–1107, – H.Muster Aug 22 '13 at 9:24

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.