Take the 2-minute tour ×
Cognitive Sciences Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. It's 100% free, no registration required.

I was sent an email by a researcher with the following question:

You wrote that difference scores are okay to use if measurement error is minimised.

  • Does that imply that biological data are more reliable for statistically analysing change than say a self-report personality scale?
  • For example, I've measured fear via cortisol and also self-report scale, does this mean that I can look at hormone measurements in a difference score but not on the self-report scales?
share|improve this question
The following paper might be interesting: Tomarken, A.J. (1995). A psychometric perspective on psychophysiological measures. Psychological Assessment, 7 (3), 387-395. –  Gala Jun 30 '13 at 18:59

1 Answer 1

I think the relevant reliability would be the expected test-retest reliability of the measure at the two time points which are otherwise being used to examine difference scores. Thus, if you were looking at difference scores derived from an experiment where measures are taken one day apart where an intervention is applied on one of those days, then you would want to know the correlation between those two days in the absence of any intervention.

In general, I would question your assumption that hormone measures are more reliable than self-report personality measures. I'm not experienced with hormone reliability data, but a good personality scale should have test-retest reliability around .80 or 90. My intuition is that hormone measures have much lower reliability than this, but I'd be very interested in anyone who can cite test-retest reliability measures for various hormone measures.

I found this discussion of cortisol test-retest reliability (Nicolson):

Test-retest reliability of acute stress response measures appears to be low, probably because of both the underestimated noise introduced by spontaneous pulsatile activity (Young, Abelson, & Lightman, 2004) and the tendency of the cortisol response to habituate following repeated exposures. Low reliability of cortisol outcome measures means that laboratory stress experiments are particularly vulnerable to Type 2 error. This issue is important in all studies, but especially needs to be taken into account in intervention studies, where stress reactivity is compared pre- and postintervention. More analyses are needed to determine how many cortisol measures per session and how many repeated sessions are necessary to obtain reliable measures of stress reactivity for different subject populations and stressors (Gunnar & Talge, 2007; Hruschka et al., 2005).


share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.