Elsevier

Cognitive Science

Volume 26, Issue 6, November–December 2002, Pages 797-815
Cognitive Science

Face recognition algorithms and the other-race effect: computational mechanisms for a developmental contact hypothesis

https://doi.org/10.1016/S0364-0213(02)00084-8Get rights and content

Abstract

People recognize faces of their own race more accurately than faces of other races. The “contact” hypothesis suggests that this “other-race effect” occurs as a result of the greater experience we have with own- versus other-race faces. The computational mechanisms that may underlie different versions of the contact hypothesis were explored in this study. We replicated the other-race effect with human participants and evaluated four classes of computational face recognition algorithms for the presence of an other-race effect. Consistent with the predictions of a developmental contact hypothesis, “experience-based models” demonstrated an other-race effect only when the representational system was developed through experience that warped the perceptual space in a way that was sensitive to the overall structure of the model’s experience with faces of different races. When the model’s representation relied on a feature set optimized to encode the information in the learned faces, experience-based algorithms recognized minority-race faces more accurately than majority-race faces. The results suggest a developmental learning process that warps the perceptual space to enhance the encoding of distinctions relevant for own-race faces. This feature space limits the quality of face representations for other-race faces.

Introduction

In everyday life, people interact socially with a variety of other people. Perception of faces is an important aspect of social interaction. Proficiency at processing faces of people from different categories (e.g., age, sex or race) can affect how these groups of individuals are perceived. It is well known anecdotally that people recognize faces of their own race more accurately than faces of other races (Feingold, 1914). This “other-race effect” has been supported more formally by a large body of psychological evidence (e.g., see meta-analyses in Bothwell, Brigham, & Malpass 1989; Shapiro & Penrod, 1986). In addition to the accuracy advantage we have in recognizing own- versus other-race faces, there also seems to be a perceptual component to this effect, captured in the commonly heard observation that other-race faces “all look alike to me.” This phenomenon suggests that we may have difficulty perceiving the uniqueness or individuality of other-race faces.

Despite the robustness of other-race findings in the psychological literature (Shapiro & Penrod, 1986), an underlying explanation for the phenomenon is less certain. Most hypotheses draw on the difference in “contact” or “experience” we have with own- versus other-race faces. At its most basic level, the contact hypothesis predicts a relationship between the amount of experience we have with other-race faces, and the size of the other-race effect. A handful of studies over the years has assessed the validity of this hypothesis, defining contact variously from simple questionnaires assessing previous exposure to members of other races (e.g., Malpass & Kravitz, 1969) to the experience of living in an integrated neighborhood (Feinman & Entwisle, 1976). As noted by Levin (2000), these studies have yielded inconsistent results, with some finding support for the contact hypothesis (Carroo, 1986; Chiroro & Valentine, 1995; Cross, Cross, & Daly, 1971; Feinman & Entwisle, 1976; Shepherd, Deregowski, & Ellis, 1974) and other studies failing to find support for this hypothesis (Brigham & Barkowitz, 1978; Lavarkas, Buri, & Mayzner, 1976; Malpass & Kravitz, 1969, Ng & Lindsay, 1994).

One reason for the lack of consistency among these studies might be linked to the diversity of the methods employed, and consequently, to the kinds of experience each may be measuring. Notably, most studies examining the contact hypothesis for the other-race effect predate important psychological findings and theory that differentiate learning that occurs developmentally and learning that occurs beyond an early “sensitive”/critical period. Though data on the special sensitivity of the developing brain to experience has been available for several decades, there has been an explosion of relevant neuroscience evidence in recent years (cf., for a number of example reviews that span various sensory systems, Gazzaniga, 2000). These findings support the idea that the behavioral effects of experience during development may differ markedly and qualitatively from the effects of experience later in life. In psychological terms, these ideas have been worked out most coherently in the context of early language development by Kuhl and co-workers (e.g., Kuhl, Williams, & Lacerdo, 1992; see also, Kuhl, 1999 for a review of the relevant work). This theory builds on data aimed at understanding how young infants discriminate speech sounds from their native language and from other languages (e.g., Werker, Gilbert, Humphrey, & Tees, 1981). These data indicate an early stage of development during which young infants (under 6 months of age) can discriminate sounds from all languages equally well. By about 6–12 months of age, however, infants begin to demonstrate a marked advantage for native language discriminations over non-native language discriminations.

The Native Language Magnet (NLM) theory proposed by Kuhl (1998) posits that early language experience warps the perceptual space to accommodate distinctions that are particularly relevant for sound discriminations in one’s native language (Kuhl, 1994, Kuhl, 1998). By this account, the earliest contact with language takes part in structuring the perceptual space in a way that maximizes the differences between similar/confusable sounds in one’s native language. Once structured, the resultant perceptual space affects the quality of the representations possible for sounds in all languages.

An analogous, albeit more slowly developing process, may account for the other-race effect for face perception and recognition (O’Toole, Deffenbacher, Abdi, & Bartlett, 1991; Shepherd, 1981). In reviewing evidence for the contact hypothesis many years ago, Shepherd (1981) noted that among the few studies testing children, and/or those defining “contact” with other-race faces developmentally (Cross et al., 1971; Feinman & Entwisle, 1976) more consistent evidence for the contact hypothesis is found. For example, Feinman and Entwisle (1976) tested the face recognition abilities of 288 African American and Caucasian children from segregated and integrated schools. The children were from grades 1–3 and 6 and were tested using a standard old/new face recognition task with photographs of African American and Caucasian children. The results showed a trend toward larger other-race effects for children in segregated schools than for children in integrated schools. When the integration status of the child’s neighborhood was also taken into account, the racial composition of the neighborhood proved highly significant. The magnitude of the other-race effect advantage was greater for children living in segregated neighborhoods. In a similar study, Cross et al. (1971) tested 120 African American and Caucasian adolescents and found that Caucasians from integrated neighborhoods showed a smaller other-race effect than their counterparts from segregated neighborhoods. In their study, African American adolescents recognized African Americans and Caucasians equally well.

Complementing these studies, Chance, Turner, and Goldstein (1982) charted the developmental course of the other-race effect by testing Caucasian participants between the ages of 6 and 20 years old on a memory task for Caucasian and Asian faces. They found that the youngest participants, 6 years olds, recognized faces of both races equally well. By 10 years of age, however, there was a recognition accuracy advantage for Caucasian faces, which became successively larger for the older participants. Combined, these studies suggest the possibility that not all “contact” is equally effective in reducing/preventing an other-race effect. Contact early in life may be related to the magnitude of the other-race effect, whereas contact later on appears to be less consistently related to recognition skills for other-race faces. It is worth noting that to the best of our knowledge, no additional developmental studies of own- versus other-race face recognition have appeared since the early 1980s.

The application of a theory like that proposed by Kuhl (1998) to the problem of learning faces would posit that early experience with faces warps the perceptual space to accommodate distinctions that are particularly relevant for discriminating among faces of one’s own race (Kuhl, 1994, Kuhl, 1998). By this account, the developmental component of contact with faces consists of structuring the perceptual space to maximize differences between similar/confusable faces of one’s own race. Once structured, the resultant perceptual space affects the quality of the representations possible for faces of all races.

The purpose of the present study was two-fold. First, we wished to explore the kinds of computational learning mechanisms that might underlie different versions of the contact hypothesis. Psychological manipulation and/or accurate gauging of the relevant variables (e.g., contact with other-race faces) is complicated for the other-race effect. This is because a number of social and attitudinal factors may play a role in the assessment of other-race contact and possibly in how observers approach the task (cf., Brigham & Malpass, 1985). It is further likely that the developmental time course of phoneme acquisition may be accelerated relative to face perception (e.g., Carey & Diamond, 1977). Computational models can therefore serve as a valuable tool for studying the learning mechanisms that may impact various processing stages, as they allow us to manipulate individual components of the algorithms and observe the effects of these manipulations on model recognition performance. This enables us to screen out learning mechanisms that do not reproduce human patterns of performance and to focus on more promising hypotheses for understanding the other-race effect.

A second purpose of the study was to evaluate the susceptibility of current computational face recognition algorithms to the other-race effect. There are both theoretical and practical reasons to study the other-race effect in the context of these engineering-based face recognition algorithms. For the former, face recognition algorithms make use of a diverse variety of training and testing paradigms that can be considered analogous to the psychological processes by which face representations are created, stored, and retrieved from human memory. The performance of different models may offer insight into the ways in which face race biases relate to the nature of the model choices for learning and retrieving faces from memory. More practically, many computational algorithms are being developed for security systems and for law enforcement applications. It is therefore worthwhile to know the extent to which accuracy varies for different races of faces as a function of the model implementations.

This paper is organized as follows. We begin with a brief report of a human recognition experiment, which lays the foundation for the evaluation of the computational models. We then present the background for interpreting the representation and retrieval stages of computational algorithms of face recognition in the context of the psychological experiment. Four kinds of algorithms are classified according to the principles of face representation they employ. The next step was to test individual models from each of these classes to determine whether or not the models show an advantage for recognizing faces from the “majority” race. Finally, we relate the representation categories of the models to their performance with majority- and minority-race faces.

Section snippets

Engineering-based computational models of face recognition

Before proceeding, we note that the source of both the stimulus sets and algorithms for this work is the Face Recognition Technology (FERET) program (Phillips, Moon, Rizvi, & Rauss, 2000). Between August 1994 and March 1997, the U.S. Government evaluated 18 state-of-the-art face recognition algorithms for the purpose of exploring the potential of each as an automated system. Thirteen of these algorithms, as implemented in the FERET test, were available to us. With these algorithms, we were able

Psychological experiment

We first carried out a standard human face recognition experiment with Asian and Caucasian observers recognizing Asian and Caucasian faces from the FERET database. Although the other-race effect has been reported many times, this experiment was necessary for two reasons. The first reason was to assure a replication of the basic effect with the present stimulus set. The second reason was to verify that the Asian and Caucasian faces used for the simulations were equally discriminable for human

Computational models of face recognition

Individual computational models of face recognition can vary in the way faces are represented and retrieved from memory. Representation describes the encoding of a face for input to a computational algorithm. At a level common to all computational models, a face representation can be thought of as a point in a multidimensional similarity space, or equivalently, as a vector from the origin (i.e., average face) of the space to the face location. The axes of this space can be interpreted as the

Face recognition algorithms and the other-race effect

We consider four types of representations. These representations differ in terms of the way the features or axes of the face space are determined. The four models comprise computational implementations of a generic contact hypothesis, a developmental contact hypothesis similar to that proposed by Kuhl, 1994, Kuhl, 1998, and two non-contact hypothesis control algorithms. A full description of the simulation methods, including the exact composition of the training sets and the performance

Procedure overview

The general procedure for comparing human and algorithm performance consisted of the following steps. In all cases, the algorithm performance measures to which we had access were based on a pre-determined set of 501 “old” individuals, chosen randomly from the FERET database. These old individuals were strongly biased for the inclusion of Caucasians, but also included Asians, African Americans, Indians, and Hispanics. The precise distribution is given in the “training set” section. For the

Results

The hit rate, false alarm rate, and A′s for Asian (minority race) and Caucasian (majority race) faces at the criterion which gives the maximum model A′ appear in Table 1, with bold-faced numbers indicating other-race effects (i.e., superior performance for majority-race faces).

For the generic contact hypothesis, which appear in the first eight rows of Table 1, seven out of eight models fared better with the Asian or minority-race faces for the A′ measure. All eight models favored the Asian

Discussion

The other-race effect for human face recognition is a problem that has implications for the way we individuate and recognize people of different races. The use of computational algorithms to aid or replace humans on this task has similarly important implications. Although the human accuracy advantage for recognizing faces of our own race over faces of other races is well documented, the underlying reasons for this advantage are less certain. The most common psychological hypothesis for this

Acknowledgements

This work was supported by grants from the National Institute of Justice administered through the National Institute of Standards and Technology to A. O’Toole and P.J. Phillips. We would also like to thank the three anonymous reviewers and Nils Penard for their comments on this manuscript.

References (34)

  • P.K. Kuhl

    Learning and representation in speech

    Current Opinion in Neurobiology

    (1994)
  • R.K. Bothwell et al.

    Cross-racial identification

    Personality & Social Psychology Bulletin

    (1989)
  • J.C. Brigham et al.

    Do “They all look alike?” The effects of race, sex, experience and attitudes on the ability to recognize faces

    Journal of Applied Social Psychology

    (1978)
  • J.C. Brigham et al.

    The role of experience and contact in the recognition of own- and other-race faces

    Journal of Social Issues

    (1985)
  • S. Carey et al.

    From piecemeal to configural representation of faces

    Science

    (1977)
  • A.W. Carroo

    Other-race face recognition: A comparison of Black American and African subjects

    Perceptual and Motor Skills

    (1986)
  • J.E. Chance et al.

    Development of differential recognition for own- and other-race faces

    Journal of Psychology

    (1982)
  • P. Chiroro et al.

    An investigation of the contact hypothesis of the own-race bias in face recognition

    Quarterly Journal of Experimental Psychology, A, Human Experimental Psychology

    (1995)
  • J.D. Cohen et al.

    Psyscope: A new graphic interactive environment for designing psychology experiments

    Behavior Research Methods, Instruments and Computers

    (1993)
  • J.F. Cross et al.

    Sex, race, age, and beauty as factors in recognition of faces

    Perception & Psychophysics

    (1971)
  • C.A. Feingold

    The influence of environment on the identification of persons and things

    Journal of Criminal Law and Police Science

    (1914)
  • S. Feinman et al.

    Children’s ability to recognize other children’s faces

    Child Development

    (1976)
  • Gazzaniga, M. (2000). The cognitive neurosciences. Cambridge,...
  • Kuhl, P. K. (1998). The development of speech and language. In T. J. Carew, R. Menzel, & C. J. Schatz (Eds.),...
  • Kuhl, P. K. (1999). Language, mind and brain: Experience alters perception. In M. Gazzaniga (Ed.), The cognitive...
  • P.K. Kuhl et al.

    Linguistic experience alters phonetic perception in infants by 6 months of age

    Science

    (1992)
  • P.J. Lavarkas et al.

    A perspective on the recognition of other-race faces

    Perception & Psychophysics

    (1976)
  • Cited by (0)

    1

    Tel.: +1-972-883-2486; fax: +1-972-883-2491.

    View full text