The first thing to note is that the study was not published as a peer-reviewed journal article. It was published as a blog post. Thus, it has not gone through a peer review process. While the reader can function as the reviewer, the article does not follow the structure of a journal article. As such, there is an inadequate description of the method in the blog post to facilitate a proper evaluation of the study, although if you follow a trail of references, there may be more information, such as in this related post comparing male and female attractiveness ratings.
Here is a critique of Kanazawa's blog post "Why black women are less physically attractive than other women":
- Subjectivity of ratings: Kanazawa states that "the interviewer rates the physical attractiveness of the respondent objectively". This is incorrect. Any rating by an individual of someone's attractiveness involves a degree of subjectivity. By averaging ratings over many raters we can obtain what we might call inter-subjective agreement about attractiveness. The more raters you have then the more reliable will be the individual-level measurement. That said, even such inter-subjective ratings will be influenced by the pool of raters used. For example, what is the gender, racial, composition of the raters. To what extent do these factors influence the ratings provided? Raters also come from a particular historical period where there are particular views about attractiveness. Thus, it's important to consider these factors when discussing any measure of attractiveness.
- Error in ratings: Kanazawa gets attractiveness ratings by averaging three ratings from three raters." From these three scores, I can compute the latent physical attractiveness score". This is not true. Three raters is better than one. But there would still be substantial unreliability in measuring "latent physical attractiveness". This is over and above the general issue of there being different definitions of latent physical attractiveness based on the pool of raters used and definitions applied. Kanazawa then goes on to make other points about factor analysis which are incorrect (e.g., claiming that factor analysis "eliminates all random measurement errors"). That said, some unreliability in measuring individual attractiveness is not fatal to the study. If anything, it would reduce observed group differences.
- Inadequate sample and rater description: There is no explicit reporting of sample sizes or sampling procedure. Thus, it is difficult to evaluate how robust the results are and whether the groups are comparable on other relevant factors such as age, dress, and so on. Furthermore, the lack of information about where the sample comes from and where the raters come from make it difficult to see to what extent any findings would generalise to other times, places, and locations.
- No standard deviations provided: There is no standard deviation provided which makes it difficult to assess the size of the group differences. If we assume that attractiveness ratings are z-scores (which is likely if derived from a factor analysis). Then the differences in attractiveness ratings between black and white women were about 0.25 standard deviations. Thus, whatever the ratings captured, if you assume SD=1, then the differences between black and white women is fairly small.
- Inadequate discussion of covariates: Kanazawa inadequately discusses covariates that might explain any observed differences. He briefly mentions BMI and intelligence as covariates. However, there are presumably many relevant covariates including age, socio-economic status, physical health and more. An adequate analysis would explore a wide range of covariates and how they relate to attractiveness ratings, and the degree to which group differences are altered by such measures. In particular, taking the perspective of evolutionary psychology requires that you first rule out short term cultural explanations such as poverty and health related explanations.
- Inadequate discussion of explanations: Kanazawa writes that "the only thing I can think of that might potentially explain [the differences] is testosterone". This seems quite naive when I can think of a wide range of reasons why any observed differences could emerge, such as socio-economic status, money spent on appearance, lifestyle factors related to health and exercise, and most importantly culture-specific values about attractiveness that the raters may be applying.