Inferring Emotion from Facial Expression in Social Contexts. A Role of Self-Construal?
Ana Maria Draghici, Raisa Petca and Hauke Hillebrandt1
1Jacobs University Bremen - Germany - adraghici@jacobs-alumni.de
Abstract
The study attempted to replicate the findings of Masuda and colleagues (2008), testing a modified hypothesis: when judging people’s emotions from facial expressions, interdependence-primed participants, in contrast to independence-primed participants, incorporate information from the social context, i.e. facial expressions of surrounding people. This was done in order to check if self construal could be the main variable influencing the cultural differences in emotion perception documented by Masuda and colleagues. Participants viewed cartoon images depicting a happy, sad, or neutral character in its facial expression, surrounded by other characters expressing graded congruent or incongruent facial expressions. The hypothesis was only (partially) confirmed for the emotional judgments of a neutral facial expression target. However, a closer look at the individual means indicated both assimilation and contrast effects, without a systematic manner in which the background characters' facial expressions would have been incorporated in the participants' judgments, for either of the priming groups. The results are discussed in terms of priming and priming success, and possible moderators other than self-construal for the effect found by Masuda and colleagues.
Starting with Darwin’s The Expression of the Emotions in Man and Animals (1872),
the face came to be considered a central medium for emotional expression. A universality
thesis was put forward, as Darwin argued that facial expressions of emotion were
innate, universal, phylogenetically derived (in line with his evolutionary thesis),
and consistent across cultures (Darwin, 1872).
Research on facial expression continued thereafter both in attempts to show that
it was diagnostic of emotion, and in attempts to show that it was not (Cornelius,
1996). In the early 1970’s, Darwin’s ideas about the universality of
facial expression came to be widely accepted, as strong confirmation was provided
by the research of Paul Ekman, which showed that expressions of emotions were recognized
as communicating the same feelings by people from different cultures in Europe,
North and South America, Asia, and Africa (Bull, 2002).
Communicative Aspects of Facial Expressions
Focusing on emotion as a central aspect of non-verbal communication (Bull, 2002), there is also recent evidence suggesting that people from different cultures make sense of facial expressions of emotion differently. Yuki, Maddux, and Masuda (2007) present evidence that Westerners and Easterners attend to different aspects of the face when judging emotion, with Japanese attending more to the eyes, while Americans attend more to the mouth. Along the same lines, Masuda et al. (2008) present evidence that Japanese participants seem to take surrounding social context into account when judging the emotion of an individual more than Americans do, i.e. the same smiling face is judged differently if it is displayed in a context with four other smiling faces, or with four other frowning faces. Furthermore, these differences were shown by eye-tracking to be of an attentional nature (Masuda et al., 2008).
Cultural Differences
East-West differences have been extensively discussed in the literature, starting with Hofstede’s studies during the late 1960s and early 1970s, in which Westerners were shown to subscribe more to individualistic values, while Easterners subscribed more to collectivistic values. In future studies, differences between Westerners and East-Asians were also shown for patterns of thinking, with Westerners thinking more in analytical terms, while Easterners thought more in holistic terms (Nisbett, 2003) and also for attention, with Westerners attending more easily to focal objects, while Easterners attended more easily to contexts (Nisbett, Peng, Choi et al., 2003).
The Independent Self and the Interdependent Self
A slightly different line of research follows the seminal work of Markus and Kitayama
(1991), which put forward the thesis of individuals versus relationships as fundamental
units of consciousness, and that Westerners and Easterners differed along the same
dimensions in self-construal. Independent and interdependent selves have been proposed
as a main moderator in a series of cultural differences (Markus & Kitayama,
1991; Markus, Kitayama, & Heiman, 1996). Kühnen, Hannover, and Schubert
later proposed a semantic-procedural model in which information about the self transfers
to information processing and behavior, as the mechanism through which these differences
operate (Kühnen, Hannover, & Schubert, 2001).
The present study sought to examine whether self-construal, a variable that is malleable
via priming, can be the moderator of the context incorporation effect reported by
Masuda et al. (2008), taking into account Markus and Kitayama (1991), and taking
into account the model proposed by Kühnen et al. (2001). If Japanese
participants showed a significant attentional propensity for incorporating social
contextual information, i.e. other faces with either congruent or incongruent expressions,
when inferring the emotion of a central target (Masuda et al., 2008), the present
study sought to check whether participants primed with an interdependent self construal
would do the same.
General Overview of the Study
Our hypothesis was that priming independence versus interdependence would replicate the effect found by Masuda and colleagues (2008), and that interdependence-primed participants would show a higher sensitivity to social context when judging the emotion of a central target, while independence-primed participants would show little or no sensitivity thereof. In order to test this hypothesis, we designed an emotion judgment task similar to the one used by Masuda and colleagues, to distinguish between the two different ways of inferring emotions, one ignoring and one including facial expressions from the social context (Masuda et al., 2008). We created a set of drawings depicting a central character in the context of a group of four other characters. We varied independently the facial expressions of the central character and those of the other characters in the group. Participants were asked to judge the emotions of the central character. On the one hand, we hypothesized that independence-primed participants would focus more on the target, ignoring the other characters, and that they would rate the central character’s emotion strictly on the basis of his own facial expression. On the other hand, we predicted that interdependence-primed participants would attend to the whole group, and incorporate the expressions of the other characters in their judgments of the target character's emotion. We did not expect that the judgments of the two groups would be categorically different, but that the expressions of the surrounding characters would influence the judgments of the interdependence-primed participants about the intensity of the central character's emotion.
Method
Participants
40 students (17 females and 13 males) from Jacobs University Bremen volunteered to participate in the experiment in return for a chocolate bar. Students’ country of origin included U.S.A, Canada, Mexico, Germany, Romania, Bulgaria, Slovakia, Poland, Ethiopia, Nicaragua, Nigeria, Ghana, Saudi Arabia, India, and Pakistan, a point to which we will return to in the discussion section.
Materials
Priming Task.For the priming of independent vs. interdependent
self-construal, we used the task developed by Gardner, Gabriel, and Lee in 1999.
In order to prime an independent self-construal, participants were asked to circle
the pronouns in a text that contained only first person singular pronouns (e.g.
“I”, “me”, “myself”). For priming interdependence,
participants had to perform the same task, on a text containing this time only first
person plural pronouns (e.g. “we”, “us”, “our”).
The text was essentially the same, only pronouns differed.
Value Questionnaire. The questionnaire developed by Kuhn and McPfarlant
in 1954, prompting agreement or disagreement with a series of items emphasizing
either individualistic or collectivistic values was used to infer whether the priming
was successful, as the questionnaire had been shown to render a pronounced endorsement
for individualistic values after independence priming and for collectivistic values
after interdependence priming (Gardner et al., 1999).
Stimuli for Emotion Judgment. In order to be able to have a clear
and controlled variation in the facial expressions, we used computer-generated drawings
as stimuli, with a centrally-situated character expressing a clear emotion, and
a group of characters in the background expressing either the same or a different
emotion. As it would have been very difficult in a tight-knit community like that
of Jacobs University to successfully convey a cover-story in which the images were
tested for future use in an educational television program, like Masuda and colleagues
did, participants were only told that the purpose of the study was to test the "expressiveness
of cartoons". We focused on two emotions thought to be universally recognized
(Ekman & Friesen, 1975), and for which an effect was clearly shown by Masuda
and colleagues, namely happiness and sadness (Masuda et al., 2008).



The facial expressions of the characters were created based on the descriptions in Ekman and Friesen (1975), using Adobe Photoshop CS 2 and Ultimate Flash Face v0.42b (See Fig.1 and Fig. 2 for examples). Each stimulus showed five characters: one centrally-placed character, which was the target, and four characters in the background. The central character showed either a happy, sad, or neutral facial expression, while the background characters showed one of three different levels of happiness, or sadness. Although our stimuli were not pretested and only built according to the descriptions of Ekman and Friesen (1975) the results of our study confirm that the emotions of the cartoon characters were indeed perceived as depicting the intended emotion. As opposed to Masuda and colleagues (2008), we varied the degrees of happiness and sadness in the peripheral characters and not in the central character, because we wanted to clearly differentiate the effects of the context. The target’s expressions were always moderately intense, or he could also take on a neutral expression, while the surrounding characters expressions varied in intensity. A total of 18 different stimuli were created, containing a central character with three different expressions (moderate happiness, moderate sadness, and neutral), and peripheral characters with one of six expressions (three degrees of happiness and three degrees of sadness, but never completely neutral).
Procedure
Participants were first asked to fill out a mood questionnaire (Watson, Clark, & Tellegen, 1988), after which they were administered the priming task (Gardner et al., 1999), the emotion judgment task, and the values questionnaire (Kuhn & McParlant, 1954). For the emotion judgment task, participants sat in front of a 43 cm monitor, with a display resolution of 1280 X 1024 pixels, at an approximate distance of 65 cm, where they viewed and rated the cartoon stimuli. The 18 different stimuli created were each repeated five times, but all 90 stimuli were presented in a single block. The order of presentation was randomized using Superlab version 4.0.5, in order to control for possible order effects. Each stimulus was presented for 3 seconds, after which participants were asked to rate the degree of happiness of the central character, followed by the degree of sadness of the central character, both on a 10-point scale ranging from 0 (not at all happy/not at all sad) to 9 (very happy/very sad). Ratings were recorded for both happiness and sadness, for each stimulus. Since the task was rather tedious, reaction times were also recorded in order to determine whether participants were answering too fast or too slow, thus not paying attention to the task and just randomly hitting the response key. Data points for which participants reacted faster than the mean minus standard deviation, or slower than the mean plus twice the standard deviation were considered erroneous and excluded from the dataset. Overall, this represented less than 7% of the entire dataset.
Results
Manipulation checks
Mood Questionnaire. We tested whether mood was a significant covariate
each time when we ran the Repeated Measures ANOVA's to test our hypotheses (see
Hypothesis Testing). This was never the case (p > 0.5 in each instance), suggesting
that the results were not affected by the participants' mood in a significant manner.
Value Orientations. The value questionnaire administered after
the emotion judgment task did not render the expected endorsement rate for the collectivistic
value items, in accordance with the priming (p > 0.05). This could be due to
having administered this questionnaire after the emotion judgment task and not immediately
after the priming, and the effect having dissipated before we could confirm it.
However, since our goal was to infer whether priming influenced the emotion judgment
task, we chose to administer this questionnaire at the end, in order for the priming
not to dissipate by the time participants got to the main task being tested.
We thus continued with the analysis, bearing in mind that, for the future, one should
treat priming more carefully, making sure that it is active for the entire duration
of the emotion judgment task.
Hypothesis Testing
Our hypothesis was that in judging the target’s emotion the interdependence-primed
participants would more likely than the independence-primed participants incorporate
the background expressions into their judgment.
A 2 (priming: independence vs. interdependence) X 3 (target character’s emotion:
happy, sad, or neutral) X 6 (background characters’ emotions: three degrees
of happiness and three degrees of sadness) mixed design repeated measures ANOVA
was employed, with priming as a between subjects factor and target and background
as within subjects repeated factors. The target’s rated degree of happiness,
and, respectively, sadness were the dependent variables.
There was a significant main effect of target character emotion for the happiness
ratings, F (2, 153) = 612.534, p < .01, as well as for the sadness ratings, F
(2, 153) = 625.077, p < .01, suggesting that the three targets' emotion was indeed
of three different categories, as it was rated significantly different for both
sadness and happiness measures, regardless of the background. This validates our
stimulus material as far as the three targets' emotions are concerned.
For the interaction terms, we focused on whether the background information significantly
affected the judgment of the target’s emotion, separately for each target.
Happy Target. We investigated the hypothesis that interdependence-primed
participants’ judgments of happiness would differ according to the emotions
of the background figures, whereas independence-primed participants’ judgments
would not. A 2 (priming) X 6 (background) mixed design repeated measures ANOVA,
with priming as a between subjects factor and background as a repeated within subjects
factor, indicated only a main effect of background in the judgment of happiness,
F (5, 169) = 27.460, p < .01, thus partially validating our stimulus material,
but no interaction by priming, suggesting that both the interdependence-primed and
the independence-primed participants equally incorporated the background characters’
emotion into their judgment of the target character’s happiness (see Fig.
4).

Sad Target. Similarly, the results did not yield support for the hypothesis that sadness judgments in the interdependence-primed group would vary according to the emotional expressions of the background characters’, whereas sadness judgments in the independence-primed group would not. A 2 (priming) X 6 (background) mixed design repeated measures ANOVA, with priming as a between subjects factor and background as a within subjects (repeated) factor, indicated a significant main effect of priming to the extent that the interdependence-primed participants gave lower sadness scores overall than the participants who were independence-primed, F (1, 179) = 4.439, p < .04, but no background interaction effect (see Fig. 5).


Neutral Target. The result of a 2 (priming) X 6 (background) mixed design repeated measures ANOVA's with priming as a between subjects factor and background as a within subjects (repeated) factor indicated a significant background effect for the happiness ratings, F (5, 158) = 9.707, p < .01, partially validating our stimulus material, as well as a significant priming X background interaction, F (5, 158) = 3.156, p < .01. The sadness ratings showed, in the same manner, a significant effect of background, F (5, 158) = 7.883, p < .01, and also a significant background X priming interaction, F (5, 158) = 2.883, p < .04. These preliminary results were in line with our hypothesis. However, a closer look at the means (see Fig. 5 and Fig. 6) indicated both assimilation (the target is judged in a way that resembles the emotion of the context) and contrast effects (the target is judged more towards the opposite pole from the emotion of the context) for both priming cases, without the interaction being systematic.

Overall, we interpreted the results as suggesting that self-construal, a popular variable in the literature so far, might not be the one driving the effect found by Masuda and colleagues (2008), as we had hypothesized.
Discussion
Our findings, even though a null result, are still important. Self-construal had
been suggested to be the main moderator in many effects where cultural differences
were found (Markus & Kitayama, 1991; Markus et al., 1996), but it appears that
this might not be the only possibility, or the explanation in this particular case,
as we failed to replicate the effect reported by Masuda and colleagues (2008) in
an experimental design by priming independence versus interdependence.
The value questionnaire administered after the emotion judgment task did not render
the expected endorsement rate for the collectivistic value items, in accordance
with the priming. It could be that the priming never worked, however, given the
robust results previously reported by Gardner et al. (1999) and Kühnen et al.
(2001), we believe that it is more likely that the priming dissipated by the end
of the experiment, when we administered said value questionnaire, or that the priming
did not work in the same manner for all of our participants. At this point, we have
to return to the participants’ very diverse cultural background and mention
the possibility that priming may not have been able to override the participants’
chronic self-construal, therefore the conclusion that self-construal is not a moderator
of the effects found by Masuda and colleagues has to be taken with a grain of salt.
Also, despite randomization, most of the participants in our independence-priming
group were females, and females have previously been shown to have a more interdependent
self-construal than males, so they might not have been as easily primed with independence
(Kühnen et al., 2001). It is very possible therefore, that chronic self-construal
has confounded our data, as background information was shown to be incorporated
in the judgments of both independent and interdependent groups. Future studies should
definitely pay closer attention to the priming manipulation.
Nevertheless, the present results suggest that, if background information was incorporated
in judgments, this 1) did not happen in the same manner for the happy, sad, and
neutral target; the neutral target being more ambiguous, it was more prone to influence
by the background and 2) the context rendered both assimilation and contrast effects,
as opposed to the results reported by Masuda and colleagues, which indicated only
assimilation effects.
Our hypothesis was not systematically supported: self construal priming failed to
systematically replicate the differential effects in emotion perception/judgment
as reported by Masuda et al. (2008). Assuming our priming did work, it apparently
did not affect the participants' judgments about the happiness, or sadness, of any
of the three targets in a systematic manner. The encouraging effect present for
the neutral target could probably be built upon, in a study with enough statistical
power.
However, it has to be mentioned that there are important differences between this
study and the one reported by Masuda and colleagues (2008): 1) the stimuli used
were more realistic and less symbolic, and 2) more variations of the contexts were
included, which, as suggested by Masuda and colleagues, might render more contrast
effects. One could argue that symbolic and more realistic stimuli cannot be expected
to be processed in the same manner, and thus this is why our results are different.
At the same time, we wanted our stimuli to be more realistic to be able to make
inferences about emotion perception in real life. As to the unexpected contrast
effects, it may be possible that, in our desire to see whether the same face can
be judged differently under different contexts, and inclusion of more variations
of the contexts, we turned the contexts into a reference point and made differences
more salient, causing the confounding effect of e.g. central characters being judged
as even more happy when the context was sad, alongside the assimilation effects.
Further studies with more tightly controlled conditions regarding the participants’
country of origin and gender might be able to more clearly tease-apart priming effects.
Especially if also clearly differentiated contexts are used, instead of, arguably,
the more vague contexts resulting from the variations in the degree of emotion displayed
by the background characters, the assimilation effect found by Masuda and colleagues
might then be replicated, rendering a different conclusion as regards the moderator
role of self construal in perceiving emotion in its social contexts, across cultures.
Authors’ note
The current research was conducted as a requirement for the Lab Course in Experimental Psychology II, part of the B.A. curriculum at Jacobs University Bremen. Our thanks go to Dr. Bettina Olk for supervision, Dr. Ulrich Kühnen for support, Dr. Batja Mesquita for an inspiring talk at Stanford during the summer of 2007, and Preston Ray for help with stimulus design.
References
Bull, P. (2002). Communication under the microscope: The theory and practice of
microanalysis. Hove, East Sussex: Routledge.
Cornelius, R. R. (1996). The science of emotion: Research and tradition in the psychology
of emotion. Englewood Cliffs, NJ: Prentice-Hall.
Darwin, C. (1872). The expression of the emotions in man and animals. Retrieved
April 29, 2006, from http://etext.virginia.edu/etcbin/toccernew2?id=DarExpr.sgm&images=images/modeng&data=/texts/english/modeng/parsed&tag=public&part=all
.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face. A guide to recognizing
emotions from facial clues. Englewood Cliffs, NJ: Prentice-Hall
Gardner, W., Gabriel, S., & Lee, A. (1999). "I" value freedom but
"we" value relationships: self-construal priming mirrors cultural differences
in judgment. Psychological Science, 10(4), 321-326.
Kuhn, M. H., & McPartland, T. S. (1954). An empirical investigation of self-attitudes.
American Sociological Review, 19, 68-76.
Kühnen, U., Hannover, B., & Schubert, B. (2001). The semantic-procedural
interface model of the self: the role of self-knowledge for context-dependent versus
context-independent modes of thinking. Journal of Personality and Social Psychology,
80, 397-409.
Markus, H. R., & Kitayama, S. (1991). Culture and the self: implications for
cognition, emotion, and motivation. Psychological Review, 98, 224-253.
Markus, H. R., & Kitayama, S., & Heiman, R. J. (1997). Culture and "basic"
psychological principles. In E. T. Higgins and A. W. Kruglanski (Eds.), Social psychology:
Handbook of basic principles (Vol. 4, pp. 857-913). New York: Guilford.
Masuda, T., Ellsworth, P.C., Mesquita, B., Leu, J., Tanida, S., Van de Veerdonk,
E. (2008). Placing the face in context: cultural differences in the perception of
facial emotion. Journal of Personality and Social Psychology, 94, 365-381.
Nisbett, R. E. (2003). The geography of thought. How Asians and Westerners think
differently...and why. New York, NY: The Free Press.
Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems
of thought: Holistic vs analytic cognition. Psychological Review, 108, 291-310.
Watson, D., Clark, L.A., & Tellegen, A. (1988). Development and validation of
brief measures of positive and negative affect: The PANAS scales. Journal of Personality
and Social Psychology, 54, 1063-1070.
Yuki, M., Maddux, W.W., Masuda, T. (2007). Are the windows to the soul the same
in the East and West? Cultural differences in using the eyes and mouth as cues to
recognize emotions in Japan and the United States. Journal of Experimental Social
Psychology, 43, 303-3