Take the 2-minute tour ×
Cognitive Sciences Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. It's 100% free, no registration required.

I saw this question on Twitter from Jay Van Bavel (@jayvanbavel), and thought it was a good fit for the site.

Does anyone find that multiple subjects from the same IP address in their #MTurk data? How do you deal with this issue? Citations welcome.

Thus, to rephrase:

  • What does it mean when you get two or more Mechanical Turk participants in a study with the same IP address?
  • What is a general strategy for dealing with such cases?
  • Are there are any references discussing the issue?
share|improve this question
    
This question is wildly off-topic –  Martin Källman Jul 27 '13 at 9:17
5  
@MartinKällman Why do you say so, this is definitely a critical part of modern research methodology in the cognitive sciences. –  Chuck Sherrington Jul 27 '13 at 9:24
1  
See stackoverflow.com/questions/2835915/… –  John Pick Jul 28 '13 at 6:14
add comment

2 Answers 2

up vote 4 down vote accepted

Since this is a relatively new problem for behavioral researchers, I don't know that there is a common consensus. I found two articles, one of which was a study that had used crowdsourcing for medical pictograms.

Their approach was as follows:

First, we checked for duplicate records. After sorting the data by participants’ IP addresses, we found three pairs of responses with the same IP address. In two pairs, the pictogram interpretations and the demographic survey answers were nearly identical, but the participation dates were different. We counted them as duplicate records and kept only the first record of each on file.

From:

Yu B, Willis M, Sun P, Wang J (2013) Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk J Med Internet Res, 15(6):e108 [FREE] [DOI]

In an article just posted this month, the subject is dealt with in more general terms, and some statistics are offered as to how frequently this might occur, and some of the reasons behind it.

Although workers may be able to have more than one concurrent MTurk account and, thus, more than one WorkerID, this is uncommon. Amazon actively works to identify and eliminate duplicate accounts. More important, requesters often restrict lucrative HITs to workers who have completed a large volume of high-quality work in the past

So, it seems that WorkerID can be used as a unique identifier and Amazon does actually screen for duplicate accounts, eliminating some of the risk of duplicate responders to the same study.

In terms of IP addresses, the article offers some idea of the extent of the problem:

Examinations of worker IP addresses typically reveal a small minority of workers (around 2.5 %; Berinsky et al.,2012**) who submit HITs from the same IP address, which may often result from workers being separate members of a single household.

From:

Chandler, J, Mueller, P, Paolacci, G (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, published online 9 July 2013 [DOI]

The first article pointed out the fact that dropping data from a study based upon duplicate IP addresses can be done without much of a cost penalty.

Other demographic factors can be used to discern whether these are different people, but the accuracy of these answers is not guaranteed, but can be bolstered by having dependencies between demographic categories (e.g., recording gender and last menstrual period should be consistent).

Results should be checked more carefully for users that share the same IP address, as even if they are different members of a household, they may be sharing a set of answers, but it seems like there is a low cost to including data from those with matching IP addresses.

** The citation for the Berinsky paper included in the Chandler 2013 work is as follows: Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368. [DOI] I did not examine this work

share|improve this answer
add comment

I'll address just the first of your three sub-questions, the others have been answered by Chuck Sherrington.

  • What does it mean when you get two or more Mechanical Turk participants in a study with the same IP address?

IP addresses are rarely "fixed" to an individual computer. Each provider has a range of IP addresses available and assigns them to computers as they connect to the network. Every time you boot your computer, every time you connect to a WLAN network, you'll probably have a different IP address. Try it with a tool such as http://whatismyipaddress.com

So of course the IP address you had yesterday will be assigned to someone else tomorrow. It only means that both computers were connected through the same provider. No other conclusions can be drawn from the IP address!

(Only the combination of IP address and connection time is unique to your computer. Your provider stores your connection data for a legally specified period, and during this time the police will be able to identify your computer if they know the time of the connection, but this data is not publicly available.)

Usually IP addresses remain stable through a session, but you cannot even rely on that. DSL connections are often reset once every 24 hours, so if you are online at night, you might experience a loss of your network connection, and after that you'll very likely have a different IP. Mobile connections are separated much more often, and your IP address will change as often. So, if you save a user's IP address at the beginning of a survey and again at the end, both might differ! This will be especially true for users surfing through anonymisation tools such as a TOR network which are programmed to switch IP addresses regularly.

You cannot reliably identify a single computer through an IP address. Use cookies for that or rely on user specified data.


See https://en.wikipedia.org/wiki/IP_address#IP_address_assignment

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.