I like that O'Boyle and Aguinis (2012) highlight the importance of discussing the distribution of job performance. Their results clearly demonstrate that for some performance metrics a normal distribution is a very poor representation. They also present a persuasive argument for why this has practical implications for managing human resources. That said I think there are a couple of issues regarding the generalisation of their results.
Issue of what is the natural metric of performance
It is possible to transform a variable to change its distribution.
For example, it is common to apply a log or square root transformation to skewed variables in order to make the resulting variable approximately normal.
Thus, in order to speak about the distribution of performance one is confronted with the issue of what is the appropriate metric of performance for a given variable.
To take one concrete example measures of time to perform a task are often positively skewed. In some instances it is modelled with an Inverse Gaussian distribution (e.g., Baayen & Milin, 2010).
However, instead of using time to perform a task, you could measure productivity as the amount of times a task is performed in a unit time.
This will be a multiple of the inverse of time to perform the task (i.e., $c \times 1/y$ where $y$ is time to produce one unit and $c$ is the amount of time given to perform the task repeatedly). In addition to reversing the scale it will also substantially transform the distribution.
The example demonstrates that not only can any given performance measure take on a variety of distributions, but that often there is more than one natural metric of performance.
How might distribution vary across tasks and performance metrics?
The following summarises the performance measures used in the article which results showed were better fit by a Paretian than a normal distribution.
- Academic: Number of publications in top journals over 9.5 year period
- Creative: Number of award nominations; number of top 500 songs, etc.
- Political: Number of appearances in the legislature; time in office
- Sports: Home run count; number of wins; goals scored;
There are several common elements that describe these performance domains:
- Higher scores on these metrics are achieved by being close to the best in the domain. In most sporting contests prizes are not distributed evenly. Salaries and prizes are much higher for those at the top. Similar arguments can be made about political success.
- Higher performance often yields additional support which reinforces performance. If you are a successful writer, then you are likely to get more support from publishers in terms of promotion and production support.
- The tasks lack natural performance constraints and are often non-standardised. I imagine that the distribution on more standardised tasks would be more normal. For example, number of calls successfully handled in a call centre or number of widgets made on a production line amongst reasonably trained employees would vary but would be a lot more normally distributed.
In an article by Theodore Micceri (1989) he reviewed the distribution of test scores in a wide range of psychometric tests and found substantial variation in the degree to which normality was obtained. While Micceri (1989) is used as a critique of the ubiquity of the normal distribution, it also highlights that distributions can vary substantially across contexts and domains, some being normal, some not being so.
O'Boyle and Aguinis state in their discussion that
Our central finding is that the distribution of individual performance
does not follow a Gaussian distribution but a Paretian distribution.
This is helpful in encouraging those wanting to pursue further research on the distribution of performance. Nonetheless, the distribution of performance will vary as a function of the task and the metric of performance used. All the datasets used in O'Boyle and Aguinis share certain characteristics and leave out a large proportion of the performance space.
Thus, I think the question properly phrased should be: "Under what conditions is the distribution of task performance characterised by a Paretian distribution?" or to be more inclusive of the range of possible performance distributions, "what causes the distribution of performance to vary?"
My initial hypothesis is that performance distributions are related to issues of task standardisation, inherent performance constraints, and the degree to which the winner-takes-all. That said, I'd like to see a more comprehensive review of performance measures that systematically examined a wider range of tasks.
- Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156. PDF
- Baayen, R. H. & Milin, P. (2010). Analyzing reaction times. International Journal of Psychological Research, 3, 12-28. PDF