Hot answers tagged methodology
16
My research group has gone pure python for coding experiments; we've been burned too many times by glitches and implicit behaviour in boxed experiment-building software to bother trusting it. Moving from a point-and-click experiment design interface to pure code does have a large learning curve, and you want to be careful to model your own code on well ...
12
I'd suggest checking out the Linear Ballistic Accumulator (Donkin et al., 2011) model for a scenario like this. While LBA can be used to model any number of alternatives in a speeded choice task, to model signal detection you'd want to model just two accumulators, one for the "signal" response and one for the "no signal" response. With this scenario, ...
11
PEBL is a free open source programming language to help construct psychological experiments. A wiki and developer community provide some documentation and support.
PEBL has a Psychological Test Battery of many common or simple psychological tests so you may be able to use it out of the box without constructing your own test. PEBL includes free versions of ...
10
MANOVA is definitely a bad idea given that one dv is continuous and the other is binomial. After exploring a number of different approaches to combining RT and accuracy data, I've come to conclude that the best current approach is to use linear ballistic accumulator model (e.g., see Donkin et al 2011).
The LBA is a simple (structurally and computationally) ...
9
A consensus within cognitive psychology is that there really is no such thing as "computation", though that depends how you define it. It seems all arithmetic ( + - * / ) is simply fact retrieval from long-term memory. More complex problems simply rely on the fact that we can break them down into simpler problems. For instance, when you learn to solve ...
8
In addition to Mike's suggestion, see the Ratcliff diffusion model and variants thereof. E.g.:
Ratcliff, R., & Rouder, J. N. (1998). Modeling response times for two–choice decisions. Psychological Science, 9, 347–356.
Ratcliff, R., & Tuerlinckx, F. (2002). Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction ...
7
If you come to this question from the bayesian tradition, then there is only one place where you can sneak in bias: your prior. This dovetails nicely with the wikipedia definition:
a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly ...
7
Bias can be quantified in many different ways. In human memory research carried out in the cognitive psychology tradition, there are simple ways to think about it. One basic measure of cognitive bias is merely called bias, and it's a measure of the absolute accuracy of an individual's probability judgments. You average probability judgments across a given ...
7
FingerFriendlySoft has created an app for all iOS devices (that is, iPod touch, iPhone and iPad) that is called N-back Suite. This is, as the name suggests, an app which lets you take the n-back test.
Included are both the single and the dual n-back test and you can chose different amounts of n (from 1 to 10), five different speeds, and different type of ...
7
Schroedingers Cat nailed a couple of important points and I need to expand it further.
I believe the important roadblock in the creation of an absolutely intelligent machine is the fact that human brain is still not decrypted. There isn't a holistic cognitive model that describes how the brain functions, reacts or take decisions. Even in case such a model ...
7
There are a variety of models solving accuracy and RT that have been pretty well tested and LBA is probably fine (I haven't used it). If you don't want to go that far there is a rather simple way to analyze data controlling for SAT that has much better mathematical properties than IE scores (which, as Mike said were named by me, but offhandedly proposed by ...
7
A paper comparing the performance of Inverse Efficiency Scores and diffusion models for the quantification of RT and accuracy can be found here.
Rach et al. (2011) "On quantifying multisensory interaction effects in reaction time and detection rate " Psychological Research
Volume 75, Number 2, 77-94, DOI , PDF
7
Procedures like this are often called "funnel debriefing" procedures, and they basically consist of an extended version of what you already had in mind.
You begin with some very general questions about the nature of the experiment (e.g., "Do you have any initial thoughts or reactions about this experiment?"), then move on to some questions that are slightly ...
7
Programs/packages for EEG analysis
There are decent MatLab toolboxes with good tutorials for for the analysis of EEG data. The EEGLAB toolbox (tutorial) can be operated by both GUI and command-line (and script). The fieldtrip toolbox (tutorial) is mainly operated by command line / script.
Of course there are also (commercial) software packages for EEG ...
6
Measures of arousal surely do play a role in subject performance on a wide variety of cognitive tasks. Generally, scientists can safely ignore this factor as it is assumed to introduce random noise between participants. From a hypothesis testing perspective, scientists are much more worried about factors that introduce systematic bias, or factors that skew ...
6
"how to build a system that can read natural language and ... understand it"
I am not sure you appreciate what you are wanting here. Just this one piece requires an understanding of the processes of the mind that I don't believe we are at yet. There is a lot of material about how we MIGHT achieve this, but you would need to appreciate the whole cognative ...
6
Diederich & Busemeyer (2003) presented a diffusion model for three choice alternatives (p. 314). The paper is a tutorial for calculating diffusion models with (discrete) matrix methods. The extension to three choice alternatives is reached by defining a two-dimensional diffusion process on a triangular plane (state space).
Recently, Wollschläger & ...
6
Instead of having a single integrator with two bounds for two choices (symmetric random walk model), you can have many competing integrators each with a bound (race model). For example, see Fig 2. of Gold and Shadlen 2007 and references therein.
As for the continuous choices case, it is important to understand a limit of discrete choices can be very ...
6
You should probably also check out:
Pleskac & Busemeyer (2010). Two-stage dynamic signal detection: A theory of choice, decision time, and confidence. Psychological Review.
Also, I believe Busemeyer has a dynamic signal detection theory paper but I don't know that it has been published. The Pleskac & Busemeyer paper probably draws on this ...
6
WebExp is a client/server based psychology/linguistics experiment creation/running system written in Java. It is freely available.
A subject types in the appropriate web address and they see the experiment pages that have been created; obviously you have to have access to a server on which the experiment software+configuration files are running. It ...
6
I use the FieldTrip toolbox in Matlab to analyze my own modified auditory MMN experiment :) But I use MEG, so I don't have that many software options. The toolbox is very powerful but it has a steep learning curve and I would recommend it only if you already have both Matlab and EEG data analysis experience.
I don't analyze my data in the classical MMN way ...
6
The Open Science Framework will do some of what you ask for. Additionally, it will allow you to preregister your hypotheses to properly distinguish confirmatory and exploratory research.
Features (quoted from the homepage).
Document and archive studies
Share and find materials, scripts, data
Detail individual contributions
Increase transparency
...
6
The basic approach that you are describing sounds like inverse efficiency scores (e.g., see Townsend and Ashby, 1978,1983), which are measured as
$$\frac{r}{1-e} = \frac{r}{c}$$
where $r$ is reaction time, $e$ is proportion error, and $c$ is proportion correct. John Christie provides a critique of inverse efficiency scores here or see the discussion in ...
6
Using Parameters Estimated from an Individual in a Group Analysis
In a way this is exactly what usually happens when we calculate the mean reaction time across all conditions for a group of participants. When we normally calculate mean reaction times we assume that some process (P) takes t milliseconds to complete plus some Gaussian distributed noise. We ...
6
There are many granulations of the objective-subjective distinction. The most objective would be the behaviorism which says it is meaningless to reason about anything except how environmental stimulus leads directly to observed behavior. This camp was popular in psychology during the time of B.F. Skinner, but now is seldom followed by people outside of ...
5
I would say there are no such tests/toolboxes, that would allow you to properly conduct any cognitive testing on iPhone, iPad or even using web-based applications. There are some games that attempt to do it, like the one suggested by @Speldosa, but nothing really serious.
At the moment there seem to be no way to control and record different variables (like ...
5
This kind of task is generally used when studying short-term memory. Typically, a subject is presented with some stimuli, followed by a probe item after a varying delay; subjects is then asked to recall (information retrieval) if the probe item was presented before (information encoding). A famous example is Sternberg experiment.
Such tasks can rely on ...
5
I recently had similar problem and I used inverse efficiency (IE) scores. These scores
were derived by dividing the response times by correct response rates separately for each condition, carried out in such a way that the higher the score was, the worse was the performance. So you get something like "corrected reaction time" scores. Here is example of paper ...
5
Another possible approach is using EZ-diffusion model suggested by Wagenmaker, van der Mass and Grasman (2007). Quoting Brown & Heathcote (2008; p. 4):
This model is extremely simple, with just one source of variability in
evidence accumulation—within-trial randomness—and simple linear
accumulation (although evidence for one response does count ...
5
It sounds like you're looking for a platform on which to implement computerized adaptive tests (since subsequent questions are contingent on prior responses).
I found Concerto, which is based in R and MySQL, but allows some flexibility in presentation (it says it uses HTML directly, but you could probably couple it with another language).
Only top voted, non community-wiki answers of a minimum length are eligible
