3
$\begingroup$

Usually an article goes through a peer-review process before it is published, and from what I here it is quite common that the reviewers "demand" some changes.

  • Are there guidelines for peer- reviewers, especially as the methodological side of things is concerned?
  • Are  "methodology specialists" incorporated in the process? 
$\endgroup$
0

2 Answers 2

3
$\begingroup$

I get the impression that good journal editors will get at least one reviewer who is skilled in the methodology used in the paper. The importance of this reviewer role would presumably vary with the statistical or other methodological complexity of the paper.

That said, reviewing is well known to be imperfect particularly when it comes to checking all the possible smaller errors that can be made. Errors in statistical analysis and reporting are wide-spread in published articles in psychology. For example, Bakker and Wicherts (2011) did a review, where to quote the abstract

we checked the consistency of reported test statistics, degrees of freedom, and p values in a random sample of high- and low-impact psychology journals... On the basis of 281 articles ... [we estimate that] around 18% of statistical results in the psychological literature are incorrectly reported. Inconsistencies were more common in low-impact journals than in high-impact journals. Moreover, around 15% of the articles contained at least one statistical conclusion that proved, upon recalculation, to be incorrect; that is, recalculation rendered the previously significant result insignificant, or vice versa.

References

  • Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666-678. FULL-TEXT
$\endgroup$
3
$\begingroup$

This does not really answer your question (because it doesn't deal in detail with methodological issues), but there is a nice journal article that deals with problems of the peer review process in general by evaluating "great scientific works of the past" from the perspective of current social sciences:

The reference list has some articles on statistical/mathematical methodology questions.


In another interesting article, Gerd Gigerenzer shows how p-values are misinterpreted in current research and null hypothesis testing is made a false requirement for the acceptance of papers by major journals:


A Google search for "peer review guidelines" brings many hits from publishers and institutions. Maybe some of them deal with methodological issues in more detail.

$\endgroup$
1
  • 2
    $\begingroup$ Another interesting paper on the same topic: Niewenhius et al (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature Neuroscience 14, 1105–1107, nature.com/neuro/journal/v14/n9/abs/nn.2886.html $\endgroup$
    – H.Muster
    Aug 22, 2013 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.