Human Face Processing

Machine learning and machine vision are currently hot topics in engineering, and showing results comparable or better than humans in many tasks, such as face detection, recognition, expression identification, landmarking, trait modification, and so on.

However, the great majority of these impressive results rely on black-box techniques that achieve their goals without considering the nature of information representation and processing; correspondingly, scientific investigation of how humans process faces has remained largely qualitative and ad hoc.

Combining vision techniques with statistical modling

Right now there is a timely opportunity to combine the best machine learning/vision techniques with advanced statistical modeling, in order to gain a mathematically rigorous and theoretically grounded understanding of human face processing.

Such an understanding is important not only because face processing is essential for human social life, reproductive success, and even survival (e.g. discriminating a foe versus a friend), but may also usher in a whole new level of understanding by revisiting many classical cognitive functions (perception, attention, learning, memory, decision-making, social cognition) through the use of face stimuli as much more complex and higher-dimensional experimental probes than found in traditional experiments.

Having more complex behavioral experimental paradigms that are nevertheless computationally interpretable makes it possible to study the neural bases of cognitive functions in ways that were hitherto impossible with simpler, artificial experimental stimuli and paradigms. With our parametric, vector-space model of human psychological face space, we are able to have the cake and eat it too: retaining interpretability while probing the brain with high-dimensional, complex, ecologically important face stimuli.

So far, we have developed vector space models of face representation in the brain using both traditional computer vision models, such as the Active Appearance Model, and deep neural net-based models, such as convolutional neural nets (Guan et al, 2018; Ryali, Wang, Yu, 2020; Ryali, Goffin, Winkielman, Yu, 2020).

In addition, we are collecting human face similarity judgments online and using them to further fine-tune this vector space model of psychological representation (in preparation). We are using such models to probe, in a systematic and data-driven way, which facial features drive different kinds of facial judgments related to demographic traits (race, gender, age), emotional expressions (happy, sad, angry), or social impressions (attractive, trustworthy, intelligent), and how these different facial judgments are related to each other at a featural level; as well as assess novel faces for predicted psychological impressions and modify them to enhance/suppress certain impressions (Guan et al, 2018).

We are also modeling how contextual effects in facial attractiveness judgment is related to statistical typicality of faces and its modulation through both top-down attention and bottom-up saliency in a task-dependent manner, and how learning and memory modify the psychological face space (Ryali & Yu, 2018; Ryali et al, 2020).

Relatedly, we are examining how individual differences in face perception arise from demographic factors such as gender, race, age, and experiential factors such as previous exposure of a particular face or faces like it (in preparation).

In addition, we are uncovering specific and previously unsuspected alterations in social processing of faces in depression and anxiety (in preparation).

We are collaborating with neuroscientists to examine the neural representation of the features in the brain implicated by the models (Huang et al, 2019).

We are also beginning to model how different kinds of face processing impairments may afflict different subpopulations of psychiatric patients, with a view toward developing novel diagnostic, predictive, and therapeutic techniques (Yu et al, in preparation).

Over all, I expect that this novel area of research in my lab will yield a fuller representational and computational understanding of natural intelligence than has been hitherto possible.

Related Papers

  • Ryali, C, Yu, A J (2018). Beauty-in-averageness and its contextual modulations: A Bayesian statistical account, Advances in Neural Information Processing Systems. MIT Press, Cambridge, MA.
  • Guan, J, Ryali, C, Yu, A J (2018). Computational modeling of social face perception in humans: Leveraging the active appearance model, bioRxiv.
  • Huang, S J, Ryali, C, K, Liu J, Guo, D, Guan, J, Li, Y, Yu, A J (2019). A Model-Based Investigation of the Biological Origin of Human Social Perception of Faces. Proceedings of the Cognitive Science Society Conference.
  • Ryali, C, Wang, X, Yu, A J (2020). Leveraging computer vision face representation to understand human face representation. Proceedings of the Cognitive Science Society Conference.
  • Ryali, C K, Goffin, S, Winkielman, P, Yu, A J (2020). From likely to likable: The role of statistical typicality in human social assessment of faces. Proceedings of the National Academy of Sciences, 117 (47).