10
$\begingroup$

O'Boyle and Aguinis (2012) wrote a paper arguing that individual job performance follows a Paretian distribution rather than a normal distribution (put simply, a very long tail to the right, rather than a bell-shape). The abstract was as follows:

We revisit a long-held assumption in human resource management, organizational behavior, and industrial and organizational psychology that individual performance follows a Gaussian (normal) distribution. We conducted 5 studies involving 198 samples including 633,263 researchers, entertainers, politicians, and amateur and professional athletes. Results are remarkably consistent across industries, types of jobs, types of performance measures, and time frames and indicate that individual performance is not normally distributed—instead, it follows a Paretian (power law) distribution. Assuming normality of individual performance can lead to misspecified theories and misleading practices. Thus, our results have implications for all theories and applications that directly or indirectly address the performance of individual workers including performance measurement and management, utility analysis in preemployment testing and training and development, personnel selec- tion, leadership, and the prediction of performance, among others.

Question

  • Are they correct in claiming that most researchers assume that individual job performance is normally distributed?
  • Are they correct in claiming that the Paretian distribution provides a better fit to individual job performance than the normal distribution?

Reference

  • ERNEST O’BOYLE JR. HERMAN AGUINIS (2012). THE BEST AND THE REST: REVISITING THE NORM OF NORMALITY OF INDIVIDUAL PERFORMANCE. Personnel Psychology Volume 65, Issue 1, pages 79–119, Spring 2012 PUBLISHER SITE; also see UNIVERSITY PRESS RELEASE
$\endgroup$
7
  • $\begingroup$ Good question, but I am worried that you are asking "is this paper correct?" $\endgroup$ Apr 2, 2012 at 2:53
  • $\begingroup$ I see the two questions as basically the same: I am asking whether the central claims in O'Boyle and Aguinis (2012) are correct. I considered making the question title more specific (e.g., "are X's claims that ...") but I thought it was more useful to frame the question more broadly because the claims extend beyond any one paper. What are your concerns? $\endgroup$ Apr 2, 2012 at 3:12
  • $\begingroup$ It might be field specific (and I only have experience in my own field), but I find people are often cautious to question a paper's validity in a public forum, especially when it is the central finding and now some specific highly technical point. I still like the focused and technical nature of the question and your presentation, so I upvoted it. $\endgroup$ Apr 2, 2012 at 3:58
  • 1
    $\begingroup$ I think with both questions, arguments can be made both ways. E.g., you can show datasets that support pareto and you can show datasets that support other distributions. You can show researchers who assume normality and you can show those that don't. Thus, I imagine a good answer would try to integrate the evidence and would deal more with issues of generalisability and strength of argument, rather than black and white issues of correct/incorrect. More broadly, I think critical and respectful discussion of specific journal articles should be actively encouraged. $\endgroup$ Apr 2, 2012 at 4:08
  • 2
    $\begingroup$ There is a nice NPR piece on this research and G+ discussion for those that want a quick summary. I also find the connection between power-laws of degree distribution in network formation (via preferential attachment, etc) and these performance results too close for comfort. Is there a more meaningful connection between the two? $\endgroup$ May 7, 2012 at 2:09

2 Answers 2

6
$\begingroup$

An alert of a follow up paper ON THE DISTRIBUTION OF JOB PERFORMANCE: THE ROLE OF MEASUREMENT CHARACTERISTICS IN OBSERVED DEPARTURES FROM NORMALITY and subsequent search brought me to this discussion. I don't have full access to these articles unfortunately.

Of the two questions that posed, no, I wouldn't expect to find a normal distribution and no, I wouldn't expect to find a pareto distribution. As was mentioned in the G+ discussion that Artem linked to, what we tend to see is some skewed distribution such as a log-normal.

A good tool for exploring these types of situations, especially in a dynamic / systems-based situation is to use simulation. This is what I did in a 2010 blog post on 'forced ranking' which at it's heart frequently assumes normally distributed performance.

The post shows selection of any sort quickly produces a skewed population. This selection can be either on entry level selection or during a performance review cycle. What I didn't explore in that simple model, but again supports the outcome, is self-selection - i.e. the likelihood of an employee leaving is to some degree based on their assessment of fit.

I also explore what happens if we use a probability distribution not only for the individual, but also a distribution for the likely measurement error (because only in rare situations can be objectively measure performance.) For me this is the sort of instance where a Bayesian Hierarchical model makes sense.

In my day job working with senior HR people to help them understand their workforces I find simulation a powerful tool. We can make a few simple assumptions and show that over time unexpected results will occur. Many times it will show that normal distributions don't logically hold over time.

$\endgroup$
1
  • $\begingroup$ Thanks the Beck, Beatty, & Sackett article looks like an interesting critical complement to O'Boyle and Aguinis' article. $\endgroup$ Sep 18, 2013 at 3:58
3
$\begingroup$

I like that O'Boyle and Aguinis (2012) highlight the importance of discussing the distribution of job performance. Their results clearly demonstrate that for some performance metrics a normal distribution is a very poor representation. They also present a persuasive argument for why this has practical implications for managing human resources. That said I think there are a couple of issues regarding the generalisation of their results.

Issue of what is the natural metric of performance

It is possible to transform a variable to change its distribution. For example, it is common to apply a log or square root transformation to skewed variables in order to make the resulting variable approximately normal. Thus, in order to speak about the distribution of performance one is confronted with the issue of what is the appropriate metric of performance for a given variable.

To take one concrete example measures of time to perform a task are often positively skewed. In some instances it is modelled with an Inverse Gaussian distribution (e.g., Baayen & Milin, 2010). However, instead of using time to perform a task, you could measure productivity as the amount of times a task is performed in a unit time. This will be a multiple of the inverse of time to perform the task (i.e., $c \times 1/y$ where $y$ is time to produce one unit and $c$ is the amount of time given to perform the task repeatedly). In addition to reversing the scale it will also substantially transform the distribution.

The example demonstrates that not only can any given performance measure take on a variety of distributions, but that often there is more than one natural metric of performance.

How might distribution vary across tasks and performance metrics?

The following summarises the performance measures used in the article which results showed were better fit by a Paretian than a normal distribution.

  • Academic: Number of publications in top journals over 9.5 year period
  • Creative: Number of award nominations; number of top 500 songs, etc.
  • Political: Number of appearances in the legislature; time in office
  • Sports: Home run count; number of wins; goals scored;

There are several common elements that describe these performance domains:

  • Higher scores on these metrics are achieved by being close to the best in the domain. In most sporting contests prizes are not distributed evenly. Salaries and prizes are much higher for those at the top. Similar arguments can be made about political success.
  • Higher performance often yields additional support which reinforces performance. If you are a successful writer, then you are likely to get more support from publishers in terms of promotion and production support.
  • The tasks lack natural performance constraints and are often non-standardised. I imagine that the distribution on more standardised tasks would be more normal. For example, number of calls successfully handled in a call centre or number of widgets made on a production line amongst reasonably trained employees would vary but would be a lot more normally distributed.

In an article by Theodore Micceri (1989) he reviewed the distribution of test scores in a wide range of psychometric tests and found substantial variation in the degree to which normality was obtained. While Micceri (1989) is used as a critique of the ubiquity of the normal distribution, it also highlights that distributions can vary substantially across contexts and domains, some being normal, some not being so.

O'Boyle and Aguinis state in their discussion that

Our central finding is that the distribution of individual performance does not follow a Gaussian distribution but a Paretian distribution.

This is helpful in encouraging those wanting to pursue further research on the distribution of performance. Nonetheless, the distribution of performance will vary as a function of the task and the metric of performance used. All the datasets used in O'Boyle and Aguinis share certain characteristics and leave out a large proportion of the performance space.

Thus, I think the question properly phrased should be: "Under what conditions is the distribution of task performance characterised by a Paretian distribution?" or to be more inclusive of the range of possible performance distributions, "what causes the distribution of performance to vary?"

My initial hypothesis is that performance distributions are related to issues of task standardisation, inherent performance constraints, and the degree to which the winner-takes-all. That said, I'd like to see a more comprehensive review of performance measures that systematically examined a wider range of tasks.

References

  • Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156. PDF
  • Baayen, R. H. & Milin, P. (2010). Analyzing reaction times. International Journal of Psychological Research, 3, 12-28. PDF
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.