Seems this is a newly discovered phenomenon! Tangen, Murphy, and Thompson (2011) describe this as a result of their method of presentation: alignment of the pupils and fast cycling through faces with different proportions. It is also important that the cycle of new images remain uninterrupted. They say "relative encoding seems to drive the effect," and list several other references for related phenomena.
Dvorsky (2013) offers (slightly) further explanation [emphasis added]:
The answer may have to do with the lack of information going to our visual cortex. Neuroscientist Mark Changizi explained it to me this way:
Because the faces are in the periphery, there's not enough information for the brain to realize it's a bunch of different people's faces appended one after another — something that's very non-natural. Instead, the brain presumes the reasonable: that it's a single person's face in the periphery, and then the face-changes are treated as this person's facial expression changes. Basically, each of those photographs has a different resting face expression, relative to the one before it, and these become drastic, cartoon-like, facial expressions on a single dynamic monstrously cartoon-like expressive creature.
I'm not entirely convinced of this explanation (much less satisfied), but it seems plausible enough, I suppose. More research may be necessary to explain this, as the authors themselves don't seem to fully comprehend what's going on here. Maybe more is out there already, but Google Scholar indicates only one citation at the time of this post, and it seems they only cited it for the materials used; no other substantive connection is apparent.
I would warmly welcome any better answers than this. I've taught intro psych classes and collected a pretty decent trove of optical illusions, because these are always popular instructional material. This one is definitely getting filed away with the rest of them, but strikes me as quite unique among them all. (Not to mention mysterious and disturbing!) Compare the Thatcher effect, which reduces distortion when presented abnormally; staring at it right-side-up makes the distortions more obvious!
Thatcher effect demonstration by Alex Dodge
· Dvorsky, G. (2013, September 19). An explanation for why your peripheral vision is completely borked. io9. Retrieved from http://io9.com/an-explanation-for-why-your-peripheral-vision-is-comple-1349170436.
· Tangen, J. M., Murphy, S. C., & Thompson, M. B. (2011). Flashed face distortion effect: Grotesque faces from relative spaces. Perception, 40(5), 628–630. Retrieved from http://www2.psy.uq.edu.au/~jtangen/TanMurTho2011.pdf.