5
$\begingroup$

Certainly, you have reading comprehension passages on the SAT, GRE, and LSAT. But beyond that point, is it really possible to measure reading comprehension through multiple-choice tests? There are multiple motivations for why an author may make argument X, and multiple interpretations of his exact argument, which may be affected by the immediate circumstances that made him write that argument at that particular point in time (and while we may know a bit about the society that he lived in, chances are that we'll probably never know who he intended the book to be written for - hell - some people probably intended to write a book to impress some potential lover [and hell, maybe some passages in the book are motivated to attract the sympathy of that lover] even though they pretended to write it for the greater good of society).

I mean, all you have to do is to look at courtrooms. There, people pretty much argue about what people (in a case with precedent) actually meant, and there's usually no concrete (and correct) answer. And people who study the life of a particular philosopher may have multiple interpretations of why philosopher X wrote passage Y, and what X really meant in writing passage Y.

Plus, it's oftentimes very interesting to read the writing of someone you know REALLY well because you know their life so well that you can dream up hypotheses of what prompted them to write passage X at time Y (and you also know his social network so you could imagine what audience he had in mind, and what audience he might have secretly hoped to impress without mentioning it). And perhaps because of this, reading comprehension is not a skill with a single scale, since people are better at understanding some people than they are at understanding others.

I mean, I guess the ultimate test of reading comprehension would be to test someone on one of Immanuel Kant's books, or on some recent postmodernist book (made artificially difficult to read/parse), and then ask him "well, what is Kant trying to get at"? But is that really a valid test? Or maybe one where you have to dissect the entire structure of a book where X is pretending to be Y pretending to be Y pretending to be X (of course, such books are rarely written, but theoretically possible).

Alternatively, wouldn't someone with exceptional reading comprehension be better at reading crappy writing? (because they're probably more able to process multiple thought-streams at once - which includes possible fragmentary/jumbled ideas?) A common assumption in reading comprehension tests is that the writing is "perfect", but this is frequently not the case, and people (especially older people) can get attention lapses when writing

$\endgroup$
6
  • 3
    $\begingroup$ Interesting musings, but I think this might be better suited to the philosophy SE. This reads more like an epistemology question, but that's just my interpretation ;) $\endgroup$
    – zergylord
    Jan 19, 2012 at 3:28
  • 2
    $\begingroup$ Heh, I don't think that the Philosophy SE would appreciate this type of question. :) But yes - I'm interested in the cognitive factors behind reading comprehension, and whether or not there is a "ceiling effect" with respect to improvements in reading comprehension ability, and if this "ceiling effect" has to do with understanding psychology. $\endgroup$ Jan 19, 2012 at 3:57
  • 2
    $\begingroup$ I think it's okay to have provide musings as context for a question, but ultimately there should be a clear articulation of the question. I think you might improve this question by adding at the end perhaps in dot points the 2 or 3 explicit single-sentence questions that you have. $\endgroup$ Jan 20, 2012 at 1:25
  • $\begingroup$ Okay - I've bolded the 2 explicit questions that I have. I need to attend to my other questions now, but please comment if you'd like it improved any further. $\endgroup$ Jan 20, 2012 at 2:40
  • $\begingroup$ Is this the question? Amongst individuals who are around or above the average level of verbal intelligence, are differences in verbal comprehension more explainable by a variable/construct known as 'psychological intuitiveness'? $\endgroup$ Feb 28, 2012 at 14:04

1 Answer 1

5
$\begingroup$

The assessment of reading comprehension is not my main area of interest, but some of the many articles on the topic of assessment of reading comprehension may be reflevant. See "measurement of reading comprehension" on Google Scholar (e.g., a couple of the pdfs of reviews here, and here).

This entry in the literacy encyclopedia on assessment of reading comprehension might be a good starting place. The article has a the following quote of a definition of reading comprehension:

The National Assessment of Educational Progress (NAEP) 2009 Reading Framework Committee defines reading comprehension as ...“an active and complex process that involves understanding written text, developing and interpreting meaning, and using meaning as appropriate to type of text, purpose and situation (National Center for Educational Statistics, 2005, p. 2).

Here are a few general points:

  • Multiple choice testing has the benefit of being reliable given a sufficient number of items. Of course, questions could be raised over the breadth of what it measures (i.e., validity issues).
  • Familiarity with the content should increase reading comprehension. Thus, to build on your question, a philosophy student should comprehend more Kant, than a non-philosophy student holding general reading ability constant. A similar argument could be made where the reader has insider knowledge about what the author is trying to say.
  • Some texts are more difficult to comprehend in general, whether it be because the text is poorly written (e.g., grammar, spelling, stylistic issues) or because of the complexity of the material.

Implication of the Rasch model to questions of measurement:

A basic model of responding to multiple choice questions based on Item Response Theory (IRT) is the Rasch model. It basically says that the probability of an individual getting an item correct is a function of their ability and item difficulty.

So according to this model, increasing the difficulty of the reading comprehension items would affect all individuals roughly equally. As can be shown with IRT the reliability of measurement can be increased by setting the difficulty of items to a level that discriminates between individuals. I.e., if items are too difficult, everyone gets the item wrong, and we don't learn much; likewise, if the item is too easy, everyone gets the item right, and we don't learn much. Relating this to your question, including poorly worded writing or really difficult subject matters would only increase difficulty, which would not directly affect measurement of individual differences.

However, the Rasch model is generally simplification of the real world. In general, I think it takes you a long way in understanding the factors driving test performance. However, in some cases you might have reading items which some individuals understand better than others and this factor is unrelated to ability (e.g., a reading that embodies cultural assumptions shared by only some individuals). It is for this reason that reading comprehension tests should probably avoid content where a subgroup is likely to possess substantially more domain specific knowledge on the topic.

$\endgroup$
1
  • $\begingroup$ Very good points. And yeah - it's probably true - most reading comprehension passages are small, so they don't involve complex psychological issues that you'd see in, say, a longer book. $\endgroup$ Jan 21, 2012 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.