# Open source software for running Internet psychological experiments that collect reaction time data

I've often used Inquisit to run psychological experiments online. The software enables delivery of stimuli (e.g., text, images, etc.) and collection of reaction times.

Obviously general purpose programming languages provide one avenue for delivery of online experiments. There's also a good listing of software for psychological experiments here. I've also seen PsyToolkit which is GNU licensed software for programming psychological experiments, but I'm not aware of any online option.

Are there any open source options for online delivery of psychological experiments, particularly ones that enable reaction time measurement?

-
I'm curious about this too. I use Adobe Flash to write psychological experiments -- the opposite of open source! – Andy DeSoto Feb 6 '12 at 22:27
HTML5 is really capable and could be used to construct reaction time experiments. While I have no data on response registration latency, I can't imagine that it is that bad. A plus with using HTML is that you can run pilots (or studies) on Amazon Mechanical turk, which is a very convenient way of rallying participants. See for example: Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3-5. doi:10.1177/1745691610393980 – Rasmus Bååth Feb 20 '12 at 8:43
I think the question (or answers?) might need a bit refinement in that response time measurements are easy in terms of accuracies in the order of seconds, but it becomes exceedingly difficult if tens-of-milliseconds-accuracy is necessary. – Christiaan Mar 9 '15 at 12:28
@Christiaan that's not accurate. Many web technologies are accurate on the scale of tens-of-milliseconds. – Josh Jun 7 at 21:40
@Josh - then why did I have to work for months to get my laptop running within the millisecond-range accuracy? Many applications may claim they do well, but it needs thorough validation. For example, a USB port imposes an 8-ms delay on anything you throw in there due to polling. At least on a Windows machine that is. – Christiaan Jun 7 at 21:42

For an open source JavaScript/HTML/CSS solution, check out jsPsych: http://www.jspsych.org. It can be used for reaction time measurement and interactive designs. An article describing the library was recently published in Behavior Research Methods.

A subsequent article investigated the properties of reaction time distributions collected with JavaScript compared to those collected with MATLAB and Psychtoolbox. The main result is that JavaScript was 10-40ms slower, but had equivalent variance across different experimental conditions and equal sensitivity to the experimental manipulation of set size in a visual search task.

There are several good answers to this related question about the validity and accuracy of response time measurements online. Many of those answers discuss findings that are relevant to JavaScript libraries like jsPsych.

de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47(1), 1-12.

de Leeuw, J. R., & Motz, B. A. (2015). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods, advance online publication.

-
thanks - this look really promising and also very current - last update on github github.com/jodeleeuw/jsPsych was only 2 weeks ago – jacanterbury Jun 5 '14 at 14:36

Another option is ScriptingRT. It's open source and fairly easy to use. The experiments are designed via a script language and then compiled into Flash apps.

ScriptingRT is designed to measure reaction time differences in the millisecond range. Schubert et al. (2013) report comparisons with other response time programs (e.g., DMDX, e-prime, Inquisit, see Studies 1 and 2).

The data suggest that measuring reaction times with ScriptingRT is associated with a little bit more noise than on lab computers. However it is sufficiently precise for the type of questions typically addressed in psychological experiments.

From Study 1:

the SDs of these responses stayed below 7 ms in all three browsers. That value is comparable to many regular keyboards and standard reaction time software. In addition, the constant added by measuring in ScriptingRT was about 60 ms. This result suggests that researchers using ScriptingRT should thus focus primarily on differences between RTs and be cautious when interpreting absolute latencies.

From Study 2:

ScriptingRT resulted in both longer response latencies and a larger standard deviation than all other packages except SuperLab and E-Prime in one configuration. Nevertheless, in absolute terms, the SD of 4.21 is comparable to what was standard for keyboards for a long time [16]. It is thus clear that any test with ScriptingRT should be well powered and used to assess primarily paradigms with a large effect size.

Schubert, T.W., Murteira, C., Collins, E.C., Lopes, D. (2013). ScriptingRT: A software library for collecting response latencies in online studies of cognition. PLoS ONE 8: e67769. doi:10.1371/journal.pone.0067769

-

We just released a beta version of Tatool Web based on JavaScript and HTML 5 which allows for running web experiments in the browser and measuring reaction times. You can check it out on www.tatool-web.com and of course it's open source.

http://www.tatool-web.com

-

Limesurvey is worth checking out (more suitable for questionnaire style tasks, but very flexible and with some coding it should be possible to, eg. record RTs)

Wextor could be another possibility - it allows building more complicated designs, has not been developed for a bit, though...

-

I'm not sure if it can measure reaction times but Tatool, developed at the University of Zurich, is an open source experiment platform that can be run from the web:

http://www.tatool.ch/

-
More recently, they posted themselves here, stating it does support reaction times. :) – Steven Jeuris May 22 at 13:31

WebExp is a free framework for developing web-based experiments, and the source code is freely available. The client side is a Java applet, so subjects must have Java installed on their computers. Perhaps this causes fewer drop-outs than having to install the executable generated by Inquisit Web. This paper discusses the timing accuracy of WebExp-based experiments.

-
from a quick look at their website it doesn't look like this product has been worked on since 2009 :( – jacanterbury Jun 5 '14 at 14:33

To complement the answers, there's also Just Another Tool for Online Studies: JATOS (disclaimer: I'm one of the authors). It's an open source tool that focusses on the server side. It will provide:

• a secure server
• a database (MySQL or H2)
• participant management (optionally prevents repeated access)
• a graphical user interface to access results
• run group studies in real-time
• a few etceteras.

Because JATOS focusses on the server side, it doesn't restrict in -almost- any way what you run in the browser (i.e., what you do in the client side). That means that it's actually complementary to a few of the tools that other answers suggested, that will mostly give you client-side solutions to present stimuli and measure reaction times in the browser. For the latter, I use the performance.now method in JavaScript that runs in most browsers and works pretty well. Probably not for careful priming or multisensory integration-type experiments, though.

-

I would highly recommend looking at http://www.qualtrics.com/. You can take a look at free trial accounts and see how its easy to use interface is still extremely sophisticated – e.g. automatic option to record response time – and can be augmented with javascript when you want to.

I'm not sure of pricing because many universities often buy a site license. But Qualtrics has revolutionized the way we do experiments.

-
Hi Joseph. Welcome to the site. It's great to have someone with your background and expertise involved on the site. I guess Qualtrics isn't open source. Is Qualtrics mainly for surveys or can you do experiments involving multimedia, reaction time, randomisation of stimuli, etc? – Jeromy Anglim Apr 27 '12 at 13:10
The questions clearly asks (in bold) for Open Source Options and Qualtrics is a commercial package. Also, although I don't think the builtin timing features will have the precision of psychology response time type experiments without writin gyour own Javscript. – jacanterbury Jun 5 '14 at 14:27

You should have a look at otree, "a Django-based framework for implementing multiplayer decision strategy games".

It is manly designed for experimental economics, but it is a very neat and versatile piece of software, and I am sure you could actually run most psychological experiment with it.

Once the experiment is coded, it only requires internet access and a browser on the part of the participant.

Big plus : otree comprises functionalities to connect with Amazon Mechanical Turk.

Caveat : I don't know if collecting reaction time data is a baseline option, but it should not be too hard to code into the framework.

-