Take the 2-minute tour ×
Cognitive Sciences Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. It's 100% free, no registration required.

Assume, I want to evaluate how effective two teachers are in teaching English to German children. Both teachers have been teaching at the same high school for twenty years, and both use a distinctly different pedagogical methodology. In fact a small competition has arisen between them: they have published and discussed their ideas and practise in journals relevant to their profession, and they have now called in a data analyst (you) to conduct this evaluation which, so they hope, will decide their contest and reconcile the former friends.

The school, where they both work, is the only school for its small town. When pupils enter this school, they are randomly assigned classes: one half of the children are assigned to one class (and one maths teacher), the other half to the other class (and the other maths teacher).

The two teachers are tired of not knowing which method is best. In the interest of their pupils they want to finally decide on the better one and both use this from now on. They hope, that you don't need to test one cohort of children when they finish elementary school, have them taught for the 8 years from 5th grade until they graduate from high school, and then measure their mathematical ability again, to come to a conclusion. Therefore they ask you:

Is it enough to compare the levels of the dependent variable post-intervention? Or do you need to measure it pre-intervention as well? Why?

share|improve this question
    
But you did measure depressivity: you know that your subjects have light depression. –  Ana Jun 12 '13 at 11:18
    
I changed my example to a case where we don't have the values pre-treatment. –  what Jun 12 '13 at 11:51
    
In your example, class assignment is definitely not random at all. First letter of last name depends on language and hence ethnicity/origin and many other relevant socio-economic variables in many societies. This one was easy to debunk and could lead to some large correlations but generally speaking this sort of procedures are not a good way to randomize at all. –  Gala Jun 24 '13 at 11:38
    
@GaëlLaurans Yes, okay, I edited my example. Simply assume that assignation is random. The focus of my question is elsewhere. –  what Jun 27 '13 at 9:22
    
Well, that's why it was merely a comment. I just thought it was funny you should be so confident when it was obvious to me that this would generate strong psychologically-relevant correlations. Paying attention to this kind of things is infinitely more important than statistical niceties or including a pre-test measure. Regarding the question itself, I think Jeromy already covered it quite well. You might also want to check stats.stackexchange.com/questions/3466/… which provides a lot of references on the issues involved. –  Gala Jun 27 '13 at 9:26
show 1 more comment

1 Answer 1

up vote 8 down vote accepted

Answer based on your original depression example

Note that this answer was originally written based on your initial example, where you asked:

Assume, I have developed a new intervention for people with light depression. I want to compare the effectiveness of this intervention (E) with an existing intervention (C). For this, I recruit test subjects from the local psychotherapeutic ambulance and randomly assign them to either the experimental (E) or control group (C). The interesting dependent variable is of course depressivity.

In general, you don't need to measure pre-treatment for the dependent variable. In the limit random assignment ensures that the groups are equal. Or to put it another way, random assignment ensures that the groups are not biased to be higher or lower on the dependent variable at baseline. A typical between-subjects t-test comparing post-treatement scores would typically provide an unbiased test of whether the intervention had an effect on dependent variable (i.e., depression) relative to control intervention.

That said, there are many benefits to including a baseline measure:

  • Including a baseline measure of depression will almost always give you more statistical power because you are able to control for much of the stable individual differences in the dependent variable (i.e., depression).
  • If you participants drop out during the intervention, it can be helpful to see whether this is related to baseline levels.
  • You can begin to assess individual differences in the effect of the intervention.
  • Where there are questions about whether random assignment was performed correctly, you can test for baseline differences.

Note there are several options for analysing pre-post treatment-control designs including ANCOVA, difference scores, and interaction effects. See this discussion for further ideas.

Note also that there are good reasons to assess an intervention by measuring more than two time points. For example, you might obtain (a) multiple baseline measures to get a sense of the stability pre-intervention (b) multiple you might measures during the intervention to assess depression during the intervention, and (c) multiple follow-up measures particularly to see both the immediate and the longer term effect of the intervention.

Updated points based on the teaching example

  • It is an empirical question whether first letter of last name is related to the effect of the teaching intervention or baseline differences. In general, it would be better to have a better form of randomisation of participants to groups.
  • Where there are questions about the randomisation procedure, the presence of a pre-test measure can be beneficial to check this.
  • There are several particular issues related to assessing the effectiveness of interventions relating to children in class rooms even when students have been randomly assigned to classrooms. (a) With just one classroom each, it is difficult to tease out what is the effect of the teacher and what is the effect of the curriculum; (b) there will typically be a lack of independent observations. Thus, for example, students within the classroom may influence each other. Thus, your effective sample size is not as great as it may seem.
share|improve this answer
    
Okay, I understand. Let's assume this is not a treatment for depression, but something that the general population experiences, like school education. So the population we are interested in is not a subpopulation which will be difficult to identify, but simply everyone. That way it will be easy to draw repeated random samples and be sure that all samples have the same normally distributed characteristics (e.g. final school grades). [contd.] –  what Jun 11 '13 at 12:41
    
[contd.] Would it be legitimate to measure the effects of the two "treatments" in two classes being taught by two different methods, and measuring the baseline with a third group of kids just entering school at the same time (e.g. June 11th, 2013), instead of measuring a group of first graders now and measuring the same kids again, post treatment, in ten years? (Assume the kids were randomly assigned to the experimental and controll classes etc. I'm just trying to understand the principles, the examples are made up as I go.) –  what Jun 11 '13 at 12:42
    
The key thing you mentioned in your question is "random assignment". In general, the choice of dependent variable does not change things. That said, when you get into a specific research context, different issues arise. For example, teaching interventions have their own features related to the mode of delivery in shared class rooms. –  Jeromy Anglim Jun 11 '13 at 13:19
1  
I would emphasize that while random assignment is in theory sufficient (as Jeromy already noted), in reality it is very hard to obtain random samples. E.G. you can't randomly assign students to a class. And even if you did, from that point on they are all in the same class and measures are not independent anymore. Hierarchical models were developed with this application in mind, but they are suited for a variety of contexts. So maybe this might be an option. –  Jens Kouros Jun 13 '13 at 8:53
1  
@what You are confusing randomization and random sampling. It's perfectly possible to randomly assign self-selected participants in an Internet survey to different conditions and validly make inferences about the effect of this manipulation. Generalizing your conclusions to a well-defined population (beyond “people ready to participate in my study”) is problematic but that's an entirely different problem. Also note that pre-test measures can be useful for a number of things but they do not replace randomization. –  Gala Jun 27 '13 at 8:50
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.