Inferring Emotion from Facial Expression in Social Contexts. A Role of Self-Construal?

Ana Maria Draghici, Raisa Petca and Hauke Hillebrandt¹

¹Jacobs University Bremen - Germany - adraghici@jacobs-alumni.de

Abstract

The study attempted to replicate the findings of Masuda and colleagues (2008), testing a modified hypothesis: when judging people’s emotions from facial expressions, interdependence-primed participants, in contrast to independence-primed participants, incorporate information from the social context, i.e. facial expressions of surrounding people. This was done in order to check if self construal could be the main variable influencing the cultural differences in emotion perception documented by Masuda and colleagues. Participants viewed cartoon images depicting a happy, sad, or neutral character in its facial expression, surrounded by other characters expressing graded congruent or incongruent facial expressions. The hypothesis was only (partially) confirmed for the emotional judgments of a neutral facial expression target. However, a closer look at the individual means indicated both assimilation and contrast effects, without a systematic manner in which the background characters' facial expressions would have been incorporated in the participants' judgments, for either of the priming groups. The results are discussed in terms of priming and priming success, and possible moderators other than self-construal for the effect found by Masuda and colleagues.

Starting with Darwin’s The Expression of the Emotions in Man and Animals (1872), the face came to be considered a central medium for emotional expression. A universality thesis was put forward, as Darwin argued that facial expressions of emotion were innate, universal, phylogenetically derived (in line with his evolutionary thesis), and consistent across cultures (Darwin, 1872).
Research on facial expression continued thereafter both in attempts to show that it was diagnostic of emotion, and in attempts to show that it was not (Cornelius, 1996). In the early 1970’s, Darwin’s ideas about the universality of facial expression came to be widely accepted, as strong confirmation was provided by the research of Paul Ekman, which showed that expressions of emotions were recognized as communicating the same feelings by people from different cultures in Europe, North and South America, Asia, and Africa (Bull, 2002).

Communicative Aspects of Facial Expressions

Focusing on emotion as a central aspect of non-verbal communication (Bull, 2002), there is also recent evidence suggesting that people from different cultures make sense of facial expressions of emotion differently. Yuki, Maddux, and Masuda (2007) present evidence that Westerners and Easterners attend to different aspects of the face when judging emotion, with Japanese attending more to the eyes, while Americans attend more to the mouth. Along the same lines, Masuda et al. (2008) present evidence that Japanese participants seem to take surrounding social context into account when judging the emotion of an individual more than Americans do, i.e. the same smiling face is judged differently if it is displayed in a context with four other smiling faces, or with four other frowning faces. Furthermore, these differences were shown by eye-tracking to be of an attentional nature (Masuda et al., 2008).

Cultural Differences

East-West differences have been extensively discussed in the literature, starting with Hofstede’s studies during the late 1960s and early 1970s, in which Westerners were shown to subscribe more to individualistic values, while Easterners subscribed more to collectivistic values. In future studies, differences between Westerners and East-Asians were also shown for patterns of thinking, with Westerners thinking more in analytical terms, while Easterners thought more in holistic terms (Nisbett, 2003) and also for attention, with Westerners attending more easily to focal objects, while Easterners attended more easily to contexts (Nisbett, Peng, Choi et al., 2003).

The Independent Self and the Interdependent Self

A slightly different line of research follows the seminal work of Markus and Kitayama (1991), which put forward the thesis of individuals versus relationships as fundamental units of consciousness, and that Westerners and Easterners differed along the same dimensions in self-construal. Independent and interdependent selves have been proposed as a main moderator in a series of cultural differences (Markus & Kitayama, 1991; Markus, Kitayama, & Heiman, 1996). Kühnen, Hannover, and Schubert later proposed a semantic-procedural model in which information about the self transfers to information processing and behavior, as the mechanism through which these differences operate (Kühnen, Hannover, & Schubert, 2001).
The present study sought to examine whether self-construal, a variable that is malleable via priming, can be the moderator of the context incorporation effect reported by Masuda et al. (2008), taking into account Markus and Kitayama (1991), and taking into account the model proposed by Kühnen et al. (2001). If Japanese participants showed a significant attentional propensity for incorporating social contextual information, i.e. other faces with either congruent or incongruent expressions, when inferring the emotion of a central target (Masuda et al., 2008), the present study sought to check whether participants primed with an interdependent self construal would do the same.

General Overview of the Study

Our hypothesis was that priming independence versus interdependence would replicate the effect found by Masuda and colleagues (2008), and that interdependence-primed participants would show a higher sensitivity to social context when judging the emotion of a central target, while independence-primed participants would show little or no sensitivity thereof. In order to test this hypothesis, we designed an emotion judgment task similar to the one used by Masuda and colleagues, to distinguish between the two different ways of inferring emotions, one ignoring and one including facial expressions from the social context (Masuda et al., 2008). We created a set of drawings depicting a central character in the context of a group of four other characters. We varied independently the facial expressions of the central character and those of the other characters in the group. Participants were asked to judge the emotions of the central character. On the one hand, we hypothesized that independence-primed participants would focus more on the target, ignoring the other characters, and that they would rate the central character’s emotion strictly on the basis of his own facial expression. On the other hand, we predicted that interdependence-primed participants would attend to the whole group, and incorporate the expressions of the other characters in their judgments of the target character's emotion. We did not expect that the judgments of the two groups would be categorically different, but that the expressions of the surrounding characters would influence the judgments of the interdependence-primed participants about the intensity of the central character's emotion.

Method

Participants

40 students (17 females and 13 males) from Jacobs University Bremen volunteered to participate in the experiment in return for a chocolate bar. Students’ country of origin included U.S.A, Canada, Mexico, Germany, Romania, Bulgaria, Slovakia, Poland, Ethiopia, Nicaragua, Nigeria, Ghana, Saudi Arabia, India, and Pakistan, a point to which we will return to in the discussion section.

Materials

Priming Task.For the priming of independent vs. interdependent self-construal, we used the task developed by Gardner, Gabriel, and Lee in 1999. In order to prime an independent self-construal, participants were asked to circle the pronouns in a text that contained only first person singular pronouns (e.g. “I”, “me”, “myself”). For priming interdependence, participants had to perform the same task, on a text containing this time only first person plural pronouns (e.g. “we”, “us”, “our”). The text was essentially the same, only pronouns differed.
Value Questionnaire. The questionnaire developed by Kuhn and McPfarlant in 1954, prompting agreement or disagreement with a series of items emphasizing either individualistic or collectivistic values was used to infer whether the priming was successful, as the questionnaire had been shown to render a pronounced endorsement for individualistic values after independence priming and for collectivistic values after interdependence priming (Gardner et al., 1999).
Stimuli for Emotion Judgment. In order to be able to have a clear and controlled variation in the facial expressions, we used computer-generated drawings as stimuli, with a centrally-situated character expressing a clear emotion, and a group of characters in the background expressing either the same or a different emotion. As it would have been very difficult in a tight-knit community like that of Jacobs University to successfully convey a cover-story in which the images were tested for future use in an educational television program, like Masuda and colleagues did, participants were only told that the purpose of the study was to test the "expressiveness of cartoons". We focused on two emotions thought to be universally recognized (Ekman & Friesen, 1975), and for which an effect was clearly shown by Masuda and colleagues, namely happiness and sadness (Masuda et al., 2008).

The facial expressions of the characters were created based on the descriptions in Ekman and Friesen (1975), using Adobe Photoshop CS 2 and Ultimate Flash Face v0.42b (See Fig.1 and Fig. 2 for examples). Each stimulus showed five characters: one centrally-placed character, which was the target, and four characters in the background. The central character showed either a happy, sad, or neutral facial expression, while the background characters showed one of three different levels of happiness, or sadness. Although our stimuli were not pretested and only built according to the descriptions of Ekman and Friesen (1975) the results of our study confirm that the emotions of the cartoon characters were indeed perceived as depicting the intended emotion. As opposed to Masuda and colleagues (2008), we varied the degrees of happiness and sadness in the peripheral characters and not in the central character, because we wanted to clearly differentiate the effects of the context. The target’s expressions were always moderately intense, or he could also take on a neutral expression, while the surrounding characters expressions varied in intensity. A total of 18 different stimuli were created, containing a central character with three different expressions (moderate happiness, moderate sadness, and neutral), and peripheral characters with one of six expressions (three degrees of happiness and three degrees of sadness, but never completely neutral).

Procedure

Participants were first asked to fill out a mood questionnaire (Watson, Clark, & Tellegen, 1988), after which they were administered the priming task (Gardner et al., 1999), the emotion judgment task, and the values questionnaire (Kuhn & McParlant, 1954). For the emotion judgment task, participants sat in front of a 43 cm monitor, with a display resolution of 1280 X 1024 pixels, at an approximate distance of 65 cm, where they viewed and rated the cartoon stimuli. The 18 different stimuli created were each repeated five times, but all 90 stimuli were presented in a single block. The order of presentation was randomized using Superlab version 4.0.5, in order to control for possible order effects. Each stimulus was presented for 3 seconds, after which participants were asked to rate the degree of happiness of the central character, followed by the degree of sadness of the central character, both on a 10-point scale ranging from 0 (not at all happy/not at all sad) to 9 (very happy/very sad). Ratings were recorded for both happiness and sadness, for each stimulus. Since the task was rather tedious, reaction times were also recorded in order to determine whether participants were answering too fast or too slow, thus not paying attention to the task and just randomly hitting the response key. Data points for which participants reacted faster than the mean minus standard deviation, or slower than the mean plus twice the standard deviation were considered erroneous and excluded from the dataset. Overall, this represented less than 7% of the entire dataset.

Results

Manipulation checks

Mood Questionnaire. We tested whether mood was a significant covariate each time when we ran the Repeated Measures ANOVA's to test our hypotheses (see Hypothesis Testing). This was never the case (p > 0.5 in each instance), suggesting that the results were not affected by the participants' mood in a significant manner.
Value Orientations. The value questionnaire administered after the emotion judgment task did not render the expected endorsement rate for the collectivistic value items, in accordance with the priming (p > 0.05). This could be due to having administered this questionnaire after the emotion judgment task and not immediately after the priming, and the effect having dissipated before we could confirm it. However, since our goal was to infer whether priming influenced the emotion judgment task, we chose to administer this questionnaire at the end, in order for the priming not to dissipate by the time participants got to the main task being tested. We thus continued with the analysis, bearing in mind that, for the future, one should treat priming more carefully, making sure that it is active for the entire duration of the emotion judgment task.

Hypothesis Testing

Our hypothesis was that in judging the target’s emotion the interdependence-primed participants would more likely than the independence-primed participants incorporate the background expressions into their judgment.
A 2 (priming: independence vs. interdependence) X 3 (target character’s emotion: happy, sad, or neutral) X 6 (background characters’ emotions: three degrees of happiness and three degrees of sadness) mixed design repeated measures ANOVA was employed, with priming as a between subjects factor and target and background as within subjects repeated factors. The target’s rated degree of happiness, and, respectively, sadness were the dependent variables.
There was a significant main effect of target character emotion for the happiness ratings, F (2, 153) = 612.534, p < .01, as well as for the sadness ratings, F (2, 153) = 625.077, p < .01, suggesting that the three targets' emotion was indeed of three different categories, as it was rated significantly different for both sadness and happiness measures, regardless of the background. This validates our stimulus material as far as the three targets' emotions are concerned.
For the interaction terms, we focused on whether the background information significantly affected the judgment of the target’s emotion, separately for each target.
Happy Target. We investigated the hypothesis that interdependence-primed participants’ judgments of happiness would differ according to the emotions of the background figures, whereas independence-primed participants’ judgments would not. A 2 (priming) X 6 (background) mixed design repeated measures ANOVA, with priming as a between subjects factor and background as a repeated within subjects factor, indicated only a main effect of background in the judgment of happiness, F (5, 169) = 27.460, p < .01, thus partially validating our stimulus material, but no interaction by priming, suggesting that both the interdependence-primed and the independence-primed participants equally incorporated the background characters’ emotion into their judgment of the target character’s happiness (see Fig. 4).

Sad Target. Similarly, the results did not yield support for the hypothesis that sadness judgments in the interdependence-primed group would vary according to the emotional expressions of the background characters’, whereas sadness judgments in the independence-primed group would not. A 2 (priming) X 6 (background) mixed design repeated measures ANOVA, with priming as a between subjects factor and background as a within subjects (repeated) factor, indicated a significant main effect of priming to the extent that the interdependence-primed participants gave lower sadness scores overall than the participants who were independence-primed, F (1, 179) = 4.439, p < .04, but no background interaction effect (see Fig. 5).

Neutral Target. The result of a 2 (priming) X 6 (background) mixed design repeated measures ANOVA's with priming as a between subjects factor and background as a within subjects (repeated) factor indicated a significant background effect for the happiness ratings, F (5, 158) = 9.707, p < .01, partially validating our stimulus material, as well as a significant priming X background interaction, F (5, 158) = 3.156, p < .01. The sadness ratings showed, in the same manner, a significant effect of background, F (5, 158) = 7.883, p < .01, and also a significant background X priming interaction, F (5, 158) = 2.883, p < .04. These preliminary results were in line with our hypothesis. However, a closer look at the means (see Fig. 5 and Fig. 6) indicated both assimilation (the target is judged in a way that resembles the emotion of the context) and contrast effects (the target is judged more towards the opposite pole from the emotion of the context) for both priming cases, without the interaction being systematic.

Overall, we interpreted the results as suggesting that self-construal, a popular variable in the literature so far, might not be the one driving the effect found by Masuda and colleagues (2008), as we had hypothesized.

Discussion

Our findings, even though a null result, are still important. Self-construal had been suggested to be the main moderator in many effects where cultural differences were found (Markus & Kitayama, 1991; Markus et al., 1996), but it appears that this might not be the only possibility, or the explanation in this particular case, as we failed to replicate the effect reported by Masuda and colleagues (2008) in an experimental design by priming independence versus interdependence.
The value questionnaire administered after the emotion judgment task did not render the expected endorsement rate for the collectivistic value items, in accordance with the priming. It could be that the priming never worked, however, given the robust results previously reported by Gardner et al. (1999) and Kühnen et al. (2001), we believe that it is more likely that the priming dissipated by the end of the experiment, when we administered said value questionnaire, or that the priming did not work in the same manner for all of our participants. At this point, we have to return to the participants’ very diverse cultural background and mention the possibility that priming may not have been able to override the participants’ chronic self-construal, therefore the conclusion that self-construal is not a moderator of the effects found by Masuda and colleagues has to be taken with a grain of salt. Also, despite randomization, most of the participants in our independence-priming group were females, and females have previously been shown to have a more interdependent self-construal than males, so they might not have been as easily primed with independence (Kühnen et al., 2001). It is very possible therefore, that chronic self-construal has confounded our data, as background information was shown to be incorporated in the judgments of both independent and interdependent groups. Future studies should definitely pay closer attention to the priming manipulation.
Nevertheless, the present results suggest that, if background information was incorporated in judgments, this 1) did not happen in the same manner for the happy, sad, and neutral target; the neutral target being more ambiguous, it was more prone to influence by the background and 2) the context rendered both assimilation and contrast effects, as opposed to the results reported by Masuda and colleagues, which indicated only assimilation effects.
Our hypothesis was not systematically supported: self construal priming failed to systematically replicate the differential effects in emotion perception/judgment as reported by Masuda et al. (2008). Assuming our priming did work, it apparently did not affect the participants' judgments about the happiness, or sadness, of any of the three targets in a systematic manner. The encouraging effect present for the neutral target could probably be built upon, in a study with enough statistical power.
However, it has to be mentioned that there are important differences between this study and the one reported by Masuda and colleagues (2008): 1) the stimuli used were more realistic and less symbolic, and 2) more variations of the contexts were included, which, as suggested by Masuda and colleagues, might render more contrast effects. One could argue that symbolic and more realistic stimuli cannot be expected to be processed in the same manner, and thus this is why our results are different. At the same time, we wanted our stimuli to be more realistic to be able to make inferences about emotion perception in real life. As to the unexpected contrast effects, it may be possible that, in our desire to see whether the same face can be judged differently under different contexts, and inclusion of more variations of the contexts, we turned the contexts into a reference point and made differences more salient, causing the confounding effect of e.g. central characters being judged as even more happy when the context was sad, alongside the assimilation effects.
Further studies with more tightly controlled conditions regarding the participants’ country of origin and gender might be able to more clearly tease-apart priming effects. Especially if also clearly differentiated contexts are used, instead of, arguably, the more vague contexts resulting from the variations in the degree of emotion displayed by the background characters, the assimilation effect found by Masuda and colleagues might then be replicated, rendering a different conclusion as regards the moderator role of self construal in perceiving emotion in its social contexts, across cultures.

Authors’ note

The current research was conducted as a requirement for the Lab Course in Experimental Psychology II, part of the B.A. curriculum at Jacobs University Bremen. Our thanks go to Dr. Bettina Olk for supervision, Dr. Ulrich Kühnen for support, Dr. Batja Mesquita for an inspiring talk at Stanford during the summer of 2007, and Preston Ray for help with stimulus design.

References

Bull, P. (2002). Communication under the microscope: The theory and practice of microanalysis. Hove, East Sussex: Routledge.
Cornelius, R. R. (1996). The science of emotion: Research and tradition in the psychology of emotion. Englewood Cliffs, NJ: Prentice-Hall.
Darwin, C. (1872). The expression of the emotions in man and animals. Retrieved April 29, 2006, from http://etext.virginia.edu/etcbin/toccernew2?id=DarExpr.sgm&images=images/modeng&data=/texts/english/modeng/parsed&tag=public&part=all
.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face. A guide to recognizing emotions from facial clues. Englewood Cliffs, NJ: Prentice-Hall
Gardner, W., Gabriel, S., & Lee, A. (1999). "I" value freedom but "we" value relationships: self-construal priming mirrors cultural differences in judgment. Psychological Science, 10(4), 321-326.
Kuhn, M. H., & McPartland, T. S. (1954). An empirical investigation of self-attitudes. American Sociological Review, 19, 68-76.
Kühnen, U., Hannover, B., & Schubert, B. (2001). The semantic-procedural interface model of the self: the role of self-knowledge for context-dependent versus context-independent modes of thinking. Journal of Personality and Social Psychology, 80, 397-409.
Markus, H. R., & Kitayama, S. (1991). Culture and the self: implications for cognition, emotion, and motivation. Psychological Review, 98, 224-253.
Markus, H. R., & Kitayama, S., & Heiman, R. J. (1997). Culture and "basic" psychological principles. In E. T. Higgins and A. W. Kruglanski (Eds.), Social psychology: Handbook of basic principles (Vol. 4, pp. 857-913). New York: Guilford.
Masuda, T., Ellsworth, P.C., Mesquita, B., Leu, J., Tanida, S., Van de Veerdonk, E. (2008). Placing the face in context: cultural differences in the perception of facial emotion. Journal of Personality and Social Psychology, 94, 365-381.
Nisbett, R. E. (2003). The geography of thought. How Asians and Westerners think differently...and why. New York, NY: The Free Press.
Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: Holistic vs analytic cognition. Psychological Review, 108, 291-310.
Watson, D., Clark, L.A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54, 1063-1070.
Yuki, M., Maddux, W.W., Masuda, T. (2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology, 43, 303-3