It is interesting and quite under-researched topic in psychology. What has been studied and definied extensively are different abnormal sexual behaviours, and exhibitionism is one of them. In the DSM-IV exhibitionism is defined as sexual arousal by revealing one's body or performing sexual acts in public and it's a form of paraphilia. Attraction to being watched by others during sexual intercourse is a form of exhibitionism called martymachlia. Presence of such sexual behaviours in DSM-IV is a clear indication that majority our society have deeply enrooted social and moral norms regarding inhibition sexual behaviour in public. If you think about it, majority of countries bans public sex and limits exhibitionism to designated places (nudist beaches and nudist colonies).
Following valid and constructive comments by @Piotr and @Preece I removed speculative part of my answer about cultural factors and expanded my answer arguing for evolutionary explanation.
Territorial mating behaviour in animals
Initially I should point out that you are not exactly right in saying that 'most species don't practice private sex'. It's true that many primates do that. But we can argue that some territorial behaviours in animal kingdom are a form of providing security during mating. Those behaviours are very common in animal world including lizards (Davis, 1980), birds (Brown, 1969; Greenwood, 1980) and mammals (Greenwood, 1980).
Evolutionary development of shame
@PiotrMigdal specified that he is mainly interested in the issue of "unwillingness to have sex in public". We can trace the origin of such 'unwillingness' to the sense of shame that could be explained from evolutionary standpoint. Darwin (1872) argued that shame represents what would be at the primitive level an instinctive seeking for cover, but his elaboration on this wasn't clear. MacCurdy (1930) took this idea further. He argued that prehistoric man sought concealment for activities which expose him to danger in a hostile environment, e.g., eating, sleeping, sexual intercourse and excretion (Maccurdy, 1930). Concealment was sought prior to the fulfilment of any act that would limit or prevent rapid self-defence. For example, Maccurdy (1930) boldly pointed out that postures during both sexual intercourse and excretion prevents people from rapid self-defence.
Malinowski (1927) writes that:
it is characteristic that sexual activities, sleep and excretion are
surrounded by protective taboos and mechanisms of concealment and
isolation in almost every society.
In this context, the sense of shame could have developed as a response to natural drive for self-protection (Dawrin, 1872; Maccurdy, 1930). Therefore, evolutionarily this would form the basis of unwillingness of having sexual intercourse in public.
Because personal security increased with the development of civilisation, the sense of shame has also transformed and became more sophisticated (Maccurdy, 1930). The shame exists only for specific situations (e.g. public sex), but it doesn't apply in other social conditions. Maccurdy (1930) gives examples with excretion, where it isn't uncommon for partners to urinate in each others presence or in the presence of other friends. We can also put nudist beach in this 'shame-exclusion' category. Overall, evolutionary account would highlight that out need for private sex is related with growth for security, the extension of shame, and the evolution of modesty. Along the same line, Malinowski (1927) also points out that sexual intercourse in public could excite jealousy and would be an indicator to invite rivals to seize that which is being enjoyed.
- American Psychiatric Association. (2000). "Diagnostic and statistical
manual of mental disorders" (4th ed., text rev.).
- Davis, J. (1980) "The times of mating and oviposition of the Western
fence lizard S.o.occidentalis", J. Herpetol. 14:102
- Brown, J. (1969) "Territorial behavior and population regulation in
birds: a review and re-evaluation". Wilson Bull. 81:293-329.
- Greenwood, P. J. (1980) "Mating systems, philopatry and dispersal in
birds and mammals." Anim. Beha. 28, 1140-1162.
- Darwin, C. (1872) "The Expression of the Emotions in Man and
Animals." London: John Murray.
- MacCurdy, J. (1930) "The Biological Significance of Blushing and
Shame." British Journal of Psychology, 21, 174-182.
- Malinowski, B. (1927) "Sex and Repression in Savage Society." London:
Kegan Paul, Trench, Trubner & Co.