Imagine a population of people in an office environment. There is a small war going on about whether this brand of coffee machine is better than that one.
How would you go about testing which coffee maker is truly most preferred?
My current plan is to take 10 people out of that population and make them each drink two cups of coffee, one made by each machine. People will be given the choice of preparation styles for their individual trial, black, milk, sugar, or both. Both cups will be made in the exact same way for that individual trial. Whether a given cup will have coffee from machine "A" or "B" will be randomized to avoid people's preference for their first taste being taken into account. Neither the participants, nor the administrator of the test will know which cup was made with which machine, to avoid unconscious biases. Lastly, each participant will be independent of each other to avoid a bias forming from everyone else choosing a particular cup. (Though the randomized nature of the study should make this a moot issue, but still.)
Is there anything I'm forgetting to make this a more perfect study? I would love to have a larger sample than 10, but alas, that's not an option. Also, I'm worried that allowing people to choose how their coffee will be prepared will skew the results, but as long as both cups within a trial are made the same way, it shouldn't be a problem, right?