Take the 2-minute tour ×
Cognitive Sciences Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. It's 100% free, no registration required.

Context: When reading research articles in psychology, you often encounter new non-aptitude self-report psychological tests (e.g., measures of personality, well-being, psychopathology, learning style, stress, goal orientation, etc.). Most of these tests involve obtaining the sum or the mean of a set of items for one or more scales or subscales. Each item typically involves item text and a consistent set of response options (e.g., Likert-type response scales, yes-no response scales, etc.). There are thousands of such scales, some which are frequently used; others are used only once.

A lot has been written about how to assess test validity. However, here I wanted to focus particularly on how to assess what a particular individual or group score means. Specifically, I'm thinking of the situation where you are reading a journal article and you encounter a scale that you have not seen before, such as:

  • A 10-item measure of self-esteem on 1 to 5 scale; group mean is 3.2. Is the sample generally high, low, or moderate in self-esteem?
  • A 30-item check-list of good eating habits with a mean of 20.7. Does the sample have good eating habits?

For well developed tests, there is a test manual. Other times there is a website or key journal article with information. Finally, in some cases, very little information is readily available. In some cases useful information is given in the methods section, but typically, on its own, this is inadequate. Thus, the process of trying to locate information such as item content and normative comparisons begin.

A lot has been said about normative and criterion referenced test interpretation. If you are using a test in clinical practice or you are using a test in your own research, you should invest time in understanding what different scores on the test actually mean. However, when casually reading a journal article you can encounter many tests. In such cases, the article is one of many, and you might just want to get a quick sense of what a given score means.

Question

  • What is a good way to get both a normative and absolute understanding of the meaning of a group or individual non-aptitude test score?
  • What is an efficient way of obtaining the information required to make such a judgement?

Note the use of the term "non-aptitude". I'd like to exclude discussion of intelligence testing, ability testing, and knowledge testing, and so on.

Update: To clarify, I didn't mean to imply in the question that item information was not available anywhere. I guess the question is referring to the general situation where there probably are previous uses of the measure in the literature, and there might be a key article that describes the measure somewhere. I guess I'm thinking of a good general strategy for securing relevant information and then using it to form an understanding of the scale properties.

share|improve this question

1 Answer 1

Tough question.

I'm going to assume that the question pertains to measures which lack all the information you've outlined: test manual, validity studies, website and key articles, item content and normative comparisons. If these assumptions are correct, then what you are a describing is an insufficiently explained measure. I will argue that we necessarily cannot know what such a scale means in any scientific sense.

A scale is a measure. A scientific measure is a process which generates evidence, i.e., observations that discriminate between different possible answers to a specific question. (Data may constitute evidence with respect to one question and not another.) The outcome of this discrimination, or analysis, is what the scale "means", i.e., the logically sound interpretation we may make on the basis of analyzing the evidence.

To determine whether a method does, in fact, generate evidence, we need to be able to replicate it independently. But the premises imply that we don't have enough information to do so. Because we cannot independently verify which hypotheses the evidence generated by the method discriminates between, if any, there is no way to know what it means.

share|improve this answer
    
First time I did a pure methodology answer, so call it a test. I don't feel too good about the lack of references for the evidence definition, but I don't even know where I'd start tracking them down. Thoughts, comments, downvotes welcome. –  Christian Hummeluhr May 9 at 17:24
    
I tend to agree with the conclusion that a normative or absolute understanding of the test is impossible given the premises that are in the question. I think the best solution -- it's not a very good one -- is to collect the data yourself. –  Josh May 9 at 20:43
    
Thanks for coming back to this question. To clarify, I didn't mean to imply in the question that item information was not available anywhere. I guess the question is referring to the general situation where there probably are previous uses of the measure in the literature, and there might be a key article that describes the measure somewhere. I guess I'm thinking of a good general strategy for securing relevant information and then using it to form an understanding of the scale properties. –  Jeromy Anglim May 10 at 7:05
    
@JeromyAnglim So, just to clarify ... something like (1) What information do we need to interpret a scale? (2) How can we most effectively get that information? –  Christian Hummeluhr May 10 at 7:33
    
@ChristianHummeluhr Yep :-) –  Jeromy Anglim May 10 at 7:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.