Usoh et al. (PDF) applied presence questionnaires (which are usually designed to measure one's feeling of presence in a virtual environment) in both a virtual and a real world environment in a between-subjects design and found no difference in people's sense of presence between the two environments.
While the authors only argue that presence questionnaires are not valid unless they pass a "reality check", i.e. participants self-report a higher feeling of presence in the real environment, in my opinion, results might have been very different if they had employed a within-subjects design, exposing participants to both environments and thereby arriving at a relative sense of presence score for each environment.
If this were true, it would imply that such questionnaires are essentially not applicable in between-subject designs, as people would only be able to reliably rate their sense of presence in a particular environment relative to another environment. Hence I've recently asked whether it is valid to treat repeated measures data as both within- and between-subject data and compare the analysis of both to see if there are differences.
But my underlying assumption was questioned by some of the answers, and I posted a follow-up question to ask if it ever makes sense for between- and within analyses of the same data to differ.
By way of a very contrived example, assume I get subjects to rate on a scale of 1 to 10 how much they like an offer of free beer. In one condition, people are offered one litre of beer for free, in the second condition, people are offered two litres of beer for free. If I ran this experiment with a between-subjects design, I would personally assume that there isn't any discernible difference between the two conditions, because - hey, free beer! But as a within-subjects design, I think I stand a reasonable chance of seeing an effect, because more free beer is better than less free beer.
One comment has asked for the question to be migrated to here so, to give the whole thing a more cog-sci spin, my intent here is to question my (admittedly intuitive) assumption that people can not relate their absolute sense of presence to a number from 1-7, but can relate their relative sense of presence compared to another environment. Is there any research other than Usoh's paper on presence in particular, or are there similar phenomenon in other measurement instruments?