Faces
IsabellaPoggi1andCatherinePelachaud2
UniversityofRomeThree,DepartmentofLinguisticsViadelCastroPretorio,20,00185RomeItaly
poggi@uniroma3.it
UniversityofRome“LaSapienza”,DepartmentofComputerandSystemScience
ViaBuonarroti,12,00185RomeItaly
cath@peano.dis.uniroma1.it
1
2
Abstract.Thispapershowsthatemotionalinformationconveyedbyfa-cialexpressionisoftencontainednotonlyintheexpressionofemotionsperse,butalsoinothercommunicativesignals,namelytheperformativesofcommunicativeacts.Ananalysisisprovidedoftheperformativesofsuggesting,warning,ordering,imploring,approvingandpraising,bothonthesideoftheircognitivestructureandonthesideoftheirfacialexpression,anditisshownthatthemeaningandtheexpressionofemo-tionslikesadness,anger,worrying,uncertainty,happinessandsurprisearecontainedinthem.Wealsoshowthatacommoncoreofmeaningispresentinanemotion(surprise)aswellasinotherkindsofcommunica-tivesignals(emphasis,backchannelofdoubt,adversativesignals).Wethenargueonhowthecognitiveandexpressiveanalysesofthesecommu-nicativeactsmaybeappliedintheconstructionofexpressiveanimatedfaces.v
Keywords:Emotions,ConversationalAgent,Performative,Embodi-ment,CommunicativeAct
1Introduction
Researchhasshownthathumanstendtotreatcomputersassocialcharacters[6,24].Asaconsequencewemaythinkthatembodiedartificialagentswillhelphuman-computerinteraction.Arecentstudy[11]proposessomeguidelinesonwhenandwheretousesuchagents:educationandentertainmentarewellap-propriatedfields.Agentsmaybehelpfulandmayprovidesuggestionstousersthathavetomakedecisionsinuncertainsituations.Dialoguingwiththeagenttogetmoreinformation,suggestionandcriticismisoftenperceivedpositivelybytheuser.Severalresearches[15,25,27]havesuggestedthattheproductivityandperformanceofauserisenhancedbytheuseoftalkingfaces.Thesystemisperceivedasmoreengaging.Inparticular,showingemotionalexpressionin-creasesuserattention;theuserspendsmoretimeinteractingwithanagentwithasternfacethanonewithaneutralexpression[27].
EmotionalMeaning2
Intheconstructionofembodiedagentscapableofexpressiveandcommunica-tivebehaviors,animportantstepistoreproduceaffectiveandconversationalfacialexpressionsonsyntheticfaces[1,2,4,5,18,23].Thefacialexpressionofaffectivestatesisthefirstandmostobviousaspectoffacialcommunicationthathastobesimulatedinanimatedface;precedingworkshaveshownthatinhumansemotionslikefear,anger,happiness,sadness,surpriseareexpressedthroughspecificfacialmuscularactions[13]andthattheseperceivableexpres-sioncanbesimulatedinsyntheticanimatedfaces[3,16,19,26].
Inthiswork,however,wewanttoshowthatemotionalcontentsandtheirexpressionarenotpresentonlyinstrictlyemotionalfacialexpressions;someaf-fectivecontents,aswelltheircorrespondingfacialexpressions,arealsopresentintypicallycommunicativesignalsthatarenotatfirstsightemotionallyloaded,forinstanceintheexpressionoftheperformativeofaSpeechActor,moregenerally,ofany(verbalornonverbal)CommunicativeAct.Bypresentingacompositionalviewofthefacialexpressionofperformatives,weanalysesomeperformativesintotheircomponentsbothonthesideoftheircognitivestructureandoftheirexpressivemuscularactions,andweshowthatonbothsidessomeemotionalcomponentscanbefound.Thenweexplorehowatypicalfacialexpressionofanemotion(theraisingoftheeyebrows)isalsopresentinothercommunicativefunctions.Wetrytofindoutwhatisthecommoncoreofmeaningthatissharedbythefacialexpressionsofallthesefunctions.Weconcludebyshowinghowthecreationofsyntheticagentscanusetheseresultsandhowthecomputationoftheirbehaviorshouldbebasedonasemanticapproachratherthanonasurfaceapproach.
2EmotionsintheMeaningofPerformatives
Thesystemwepresenthereisbasedonagoalandbeliefmodelofvactionandcommunication[10,20].Inthismodelacommunicativeactcorrespondstotheminimalunitofcommunication.Acommunicativeactisanyverbalornonver-balactionthroughwhichaSenderhasthegoalthattheAddresseegetsomebeliefsabouttheSender’sgoalsandbeliefs.Tofullyanalyseacommunicativeactonehastoconsidertwoaspects:itssignalanditsmeaning.ThesignalofacommunicativeactisasetofmuscularactivitiesormorphologicalchangesthatcanbeperceivedbyanAddressee.Themeaningofacommunicativeactincludesaperformativeandapropositionalcontent,wherethepropositionalcon-tentisthesetofbeliefsmentionedandtheperformativeistheSender’sspecificsocial-communicativegoal,thatis,whattheSenderwantstheAddresseetodoinmentioningthatspecificpropositionalcontent.Forexample,inthesentence“Isuggestyoutakeyourumbrella”,‘suggest’istheperformativeand‘youtakeyourumbrella’isthepropositionalcontent.Inparticular,inpreviousworksonanimatedfaces[21,22],weproposedawaytoreproducethefacialexpressionsthatconveytheperformativeofacommunicativeact.Inourmodel,themeaningsideofaperformativeinacommunicativeactmaybeanalyzedasasetof“cog-nitiveunits”[7],logicalpropositionswhosepredicatesareprimitivepredicates
EmotionalMeaning3
likeGoal,Belief,andwhoseargumentsarepersons(Sender,Addressee),domain‘objects’,domain‘facts’,domain‘actions’,emotions.Thefacialexpressionisrepresenteddiscursivelyasperceivablestatesormovementsofspecificregionsoftheface(lips,eyebrows,cheeks),ormoreschematicallyintermsofEkmanandFriesen’sActionUnitsasdefinedinFACS(FacialActionCodingSystem)[14],thatcanbeimplementedin3Dfacialmodel[19].
SupposeaSenderSsuggestsanAddresseeAtodosomeactiona.Themeaningoftheperformativeofsuggesting(initsreadingasarequest,notasaninformation),thatisthesetofinformationthatSwantsAtobelievewhenprovidingasuggestion,mayberepresentedbythefollowingcognitiveunits[21]:1.ShasthegoalthatAdoaSrequestsAtodoa2.SbelievesaisusefultoagoalgofAaisintheinterestofA3.Sbelieves2.withlowdegreeofcertaintydegreeofcertainty
CognitiveUnitsofsuggestingOnthesideofthefacialsignal,theperformativeofsuggestingisexpressedbylookingattheaddressee,withtheheadabitleaningforwardandtheeyebrowsslightlyraised.Sometimes,theglobalexpressionofthefacecorrespondstothecompleteclusterofcognitiveunitsthatmakesupaperformative;butinsomecaseswecanfindevenaone-to-onecorrespondenceamongcognitiveunitsandfacialactions:forinstance,inthecaseofsuggestionheadleaningforwardmaybeaquitespecificwaytoexpressthattherequestedactionisnotintheinterestofSbutofA(CognitiveUnit2.);whileaslighteyebrowraisingmaytypicallyexpressperplexityoruncertainty(CognitiveUnit3.).Focusingonthemeaningofperformatives,ifweanalyzeafairnumberofthem,wecanseethatonthemeaninglevelseveralperformativescontaininformationaboutsomeaffectivestate.Letusseesomeexamples.Theperformativeofimploringmaybeanalyzedasfollows:
1.ShasthegoalthatAdoaSrequestsAtodoa2.SbelievesaisusefultoagoalgofSaisintheinterestofS3.AhaspoweroverSpowerrelationship4.ifAdoesnotdoa,thenSwillbesad.potentialaffectivestate
CognitiveUnitsofimploringTheperformativeofimplorationcontainstheaffectivestateofsadnessbe-cause,ifIaskyousomethingthatisveryimportanttomeandthatIcannotobtainwithoutyourhelp,IcananticipatethatIwillbesadifyoudonotfulfilmyrequest.Nowtaketheperformativeofaperemptoryorder:
1.ShasthegoalthatAdoaSrequestsAtodoa2.SbelievesaisusefultoagoalgofSaisintheinterestofS3.AhaspoweroverSpowerrelationship4.ifAdoesnotdoa,thenSwillbeangry.potentialaffectivestate
CognitiveUnitsofperemptoryorder
EmotionalMeaning4
Again,theperformativeofperemptoryordercontainstheaffectivestateofanger.IfIrequestyoutodosomethingwhileassumingthatIhavepoweroveryou,Icanshowpotentialanger,sinceifyoudonotfulfilmyrequestI’llbeangryatyou.Hereis,moreover,theperformativeofwarning:
1.ShasthegoalthatAbelievescSinformsAofc2.SbelievesthatnottobelievecmaycausesocialrelationshipsomegoalofAtobethwarted
3.ShasthegoalthatA’sgoalsarereached4.SisworryingforApotentialaffectivestate
CognitiveUnitsofwarningTheperformativeofwarningcontainstheaffectivestateofworrying:warningmeansthatIgiveyousomeinformationthatisimportantforyou,attheextentthatifyoudidnotknowitsomethingwrongcouldhappentoyou:thus,warningcontainsmybeingworriedforyou.
Letusnowtakethedifferencebetweenapprovingandpraising.1.SbelievesthatAhasdonea
2.SbelievesthatdoingaisagoodthingSevaluatesa3.Sishappyaffectivestate4.2.causes3.
5.ShasthegoaltheAbelieves4.SinformsAofanevaluation
CognitiveUnitsofapproving
1.SbelievesthatAhasdonea
2.SbelievesthatdoingaisagoodthingSevaluatesa3.Sishappyaffectivestate4.2.causes3.
5.SbelievesthatAdidaparticularlywellSevaluatesa6.Sissurpriseaffectivestate7.5.causes6.
8.SbelievesthatAisgoodSevaluatesA9.5.causes8.
10.ShasthegoalthatAbelieves8.SinformsAofanevaluation
CognitiveUnitsofpraisingInbothapprovingandpraising,SbelievesthatAhasdonesomeactionaandthatdoingaisagoodthing,(seetheCognitiveUnits1.and2.inbothperformatives),andinboththismakesShappy(CognitiveUnits3.and4.).However,praisingdiffersfromapprovinginthatSnotonlybelievesthatdoingaisgood(CognitiveUnit2.)butalsothatAdidaparticularlywell(CognitiveUnit5.of“praising”),betterthantheaverage,surprisingly,unexpectedlywell:therefore,Sisalsosurprised(CognitiveUnit6.-asweshallseelater,surprise
EmotionalMeaning5
occursanytimeaneweventdisconfirmssomeexpectation).Moreoverinpraising,differentfromapproving,Snotonlyhasagoodevaluationoftheactiona,butofthepersonA(CognitiveUnit8.)assomebodywhoisbetterthantheaverage(andthiscausallinkisexpressedbyCognitiveUnit9.).Inconclusion,whiletheperformativeofapprovingcontainsonlytheemotionofhappinessfortheother’saction,praisingcontainsnotonlyhappinessbutalsosurprise.Andtheseemotionscanbedetectedinthefaceofapprovingandpraising.
3EmotionintheExpressionofPerformatives
Thatsomeaffectivestatesarecontainedinsomeperformativesistruenotonlyonthelevelofthecognitivestructureofperformativesbutalsoonthelevelofhowtheyareconveyedthroughfacialexpression.IfweturntothesignalsideofperformativefaceswecanseethatsomeActionUnitsthattypicallyconveyaffectivestatestakepart,infact,inthefacialexpressionofperformatives,andsometimestheyevenshowaone-to-onecorrespondencewiththeiraffectivecog-nitiveunits.Forinstance,intheimploringfacetheinnerpartsoftheeyebrowsareraised(seeFigure1(a));whichisalsothetypicalexpressionthatcharac-terisessadness.Similarly,thefaceinaperemptoryorderperformsafrown(innerpartsoftheeyebrowsloweredandcloser),whichisalsoatypicalexpressionofanger(seeFigure1(b)).Awarningfacecontainsanexpressionofworrying,witheyebrowscloserandmakingwrinklesontheforehead(seeFigure2(a));asuggestingfacecontainsanexpressionofuncertainty,withslightlyraisedeye-brows(seeFigure2(b)),anapprovingfacecontainsaslightsmileofhappinesswithraisedcheeks(seeFigure3(a)),whileapraisingfacecontainstheeyebrowraisingandopenedeyestypicalofsurprise(seeFigure3(b)).
Fig.1.(a)Imploringeye(b)Orderingeye
4Raisingeyebrows:notonlysurprise
Sofarwehaveshownthatemotionalinformationisgenerallycontainedintheexpressionofaperformative,thatis,thespecificcommunicativegoalofasen-tenceorotherkindofcommunicativeact.Butemotionalcontentisalsopresentinothercommunicativematerial.Inordertoexplorethistopic,wedonotstart
EmotionalMeaning6
Fig.2.(a)Worryingeye(b)Suggestingeye
Fig.3.(a)Approvingeye(b)Praisingeye
anymorefromthemeaningsidebutfromthesignalsideoffacialexpression.Takeaspecificfacialsignal,theraisingofeyebrows,andseewhatdifferentmeaningsitmayconvey.Thisfacialsignaloccursindifferentsituationsandmaybearthefollowingdifferentmeanings:
1.Whenfacinganunforecastsituationitmeans“Iamsurprised”;
2.Whenlisteningtoaninterlocutorandwantingtoshowdoubtfulness,itmeansincredulity,anditcouldbeparaphrasedas:“it’sverydifficulttobelievethat!”
3.Whileutteringanadversativeadverborconjunction,like‘but’,‘however’,‘instead’,‘onthecontrary’,itbearsthesameadversativemeaning,thatis,itwarnstheinterlocutornottodrawtheinferencethatwouldbemostplausiblefromtheprecedingwords.Itequalswarning:“itisnotwhatyoumightthink,theoppositeistrueinfact”;
4.Inplainconversationorargumentation,itputstheemphasisonthewordbeinguttered,thusmeaningsomethinglike:“thisisthemostimportantthinginmysentence;thisiswhatIreallywantyoutounderstand”;Inapaperontheexpressivemovementsoftheeyebrows,Ekman[12]distin-guishesthesefourseeminglyverydifferentsituations:thefirstpossiblyholdingeveninabsenceofcommunicativeinteraction,thesecondinthecaseofthein-terlocutorinaconversationalsetting,thethirdandthefourthtypicalofthebehavioroftheSpeakerinconversation.Hearguesthattheupwardanddown-wardmovementsoftheeyebrowsareatworkbothinemotionalsignals-theexpressionofemotionsofsurprise,angerorfear-andconversationalsignals-interrogativeexpressions,emphasisandapprovingsignals.Hisaccountseemstoimplyastrongandinexplicablepolysemyinthesamesignal-forexample,thesameraisingoftheeyebrowsmaybeontheonesideanexpressionofsurprise,ontheothersideasignalofemphasis.
EmotionalMeaning7
OnthebasisofEkman’saccount,thesemeaningsdonotseemtoshareacommoncoreofmeaning.Webelievetheopposite.Thatis,inourview,itismoreeconomicandplausibletothinkthatthedifferentmeanings(includingtheemotionalmeanings)ofthesamesignal(surprise,here)haveapartoftheirmeaningincommon.
Letusseenowwhatmightbethecommoncoreofmeaningthattheraisedeyebrowsshareinthefourcasesabove:1.surprise,2.doubtandperplexity,3.adversativeword,4.emphasis.
Firstofall,letusseewhatissurpriseandwhatisitsfunctionfromacognitivepointofview.
Surpriseisanemotionthatoccurseverytimesomeexpectationisdiscon-firmed,becausesomeeventhasoccurredthatcannotbeinferredbypreviousbeliefs[8].
Butwhyisthedisconfirmofanexpectationsorelevantinhumanlifeastogiverisetoanemotion,andwhyisitsoimportanttocommunicateit?
Tobeawarethatsomethingisdifferentfromexpectedisimportant,fromanadaptivepointofview,particularlyforthehumananimal,whosesurvivalveryheavilydependsonitscognitivecapacity.Forhumans,themostimportantresourcestoreachtheirgoalsarenotasmuchtheirphysicalpowersastheirbeliefs.Thisiswhyhumansdonotonlyhavepragmaticorsocialgoals,likethegoalofperformingactionswellorthegoalofbeinglovedbyotherpeople;theyalsohaveepistemicgoals,thatis,thegoalofacquiringasmanybeliefsastheycanandofhavingone’sbeliefsreliable,integratedandlinkedwitheachotherincognitivenetworks[9].Amongepistemicgoalshumansalsohavethegoalofgeneratinginferences,ofdrawingexpectationsaboutevents:thisisessentialnotonlytowidentherangeoftheirbeliefs,butalsotolinkbeliefswitheachother,inordertofeelmoresureofthem;moreoverinferences,andparticularlyexpectationsaboutfutureevents,arenecessaryinplanningthefuturecourseofaction.Butsince,atthesametime,inferencesarebydefinitionnotcompletelyreliable,andmaybedisconfirmedbytheactualcourseofevents,humansalsohavetheepistemicgoalofbeingparticularlyalertedanytimesomeexpectationiscontradicted.Now,whensomegoalisparticularlyimportantfromanadaptivepointofview,itsbeingreachedorthwartedisoftensignalledandmonitoredbyanemotion;andsurpriseisjusttheemotionthatisfeltasanexpectationissuddenlydisconfirmed.Butagain,sinceanunexpectedbeliefjeopardisesthewholestructureofhumanknowledge,inthatitcannotbeinferredfrompreviousbeliefs,eachnewunexpectedeventalsotriggersthesearchforfurtherbeliefsthatallowhumanstoinferit.Thisiswhytheemotionofsurprisecausesthemtoraiseeyebrowsandtoopentheireyeswide:openingeyeswideisawaytowidenthevisualfield,andthentoseealargeramountofthings,tocatchalargeramountofbeliefs.Asurprisedpersonisbiologicallyprogrammedtobereadytocatchasmuchvisualinformationasshecan:thus,theraisingofeyebrowsandtheopeningofeyesbecomeanexpressivesignalofsurprise.Butwhyshouldthismeaningbesharedbytheothersituationsabove?Letustakethemonebyone,
EmotionalMeaning8
andseewhatdothesemeaningshaveincommonwithsurprise.Thisisquiteeasytoseewhencase1.(surprise)andcase2.arecompared.
Thesemanticstructureofsurprisemayberepresentedasfollows:
1.Scomestobelievep2.Sbelievesq
3.Sintendstoinferpfromq4.Scannotinferpfromq5.Sfeelssurprised6.4.causes5.
CognitiveUnitsofsurprise
SurpriseoccurswhenScomestobelievesomething(CognitiveUnit1.)thatScannotinfer(CognitiveUnit4.)byone’spreviousknowledgeq(CognitiveUnit2.),whichissomethingthatusuallySwants(CognitiveUnit3.).Thisimpossibilitytoinfernewinformationfromoldinformationcausestheemotionofsurprise(CognitiveUnit5.).
Letusnowseethesemanticstructureofcase2.,thebackchannelofdoubtorperplexity1:
1.S2.S3.S4.S
believesthatAhasthegoalthatSbelievespbelievesq
cannotinferpfromqfeelsdoubtfulaboutp
CognitiveUnitsofbackchannelofdoubt
Inthebackchannelofdoubt,SistheformerAddresseewhoisnowtheSenderofaback-channelmessage,somethinglike“Ican’tbelievewhatyouaresaying”.A(theformerSenderofamessage)hasthegoalthatSbelieveswhatAissaying(CognitiveUnit1.),butSalreadybelievesq(CognitiveUnit2.),fromwhichpcannotbeinferred(CognitiveUnit3.);thenSraisestheeyebrowsinordertoshowsheisdoubtfulaboutp(CognitiveUnit4.).
Therefore,showingdoubtorperplexity(case2.)asaback-channelsignaltosomeonewhoistellingussomethingincredibleisnotsodifferentfromshowingsurprise,becauseanincrediblebeliefisoneIcannotinferfrommypreviousknowledge,thensomethingthatdisconfirmsmyexpectations.Onecouldobjectinfactthatinsuchasituationtheraisingoftheeyebrowsisnotnecessarilyaccompaniedbytheopeningoftheeyes.Thiscouldbeaccountedforwiththefact
EmotionalMeaning9
thattheoriginalneedofhavingawiderfieldofviewinthecaseofsurprise,thatwaspresentinourancestors,isnotsocompellinghere;incaseslikethis,wheretheunexpectedeventisnothappeninginfrontofusbutreportedthroughotherpeople’sdiscourse,onlythesignal“raisingeyebrows”remainstocharacterisethemessage.
Letusnowcometocase3.:whatiscommonbetweenexpressingsurpriseandaccompanyinganadversativeword?Adversativewords(like‘but’,‘though’,‘yet’,‘nonetheless’...)signalthatanexpectedbelieforaplausibleinferenceisnottrueinfact.Takethesentence:“Sheisblond,butshehasblackeyes”.Here,fromthefirstpartofthesentence,“Sheisblond”,youcouldinferthat“shehasblueeyes”,sincethisiswhatusuallyoccurs;therefore,inordernottoletyoudrawthisinference,whichinthisspecificcaseisincorrect,Inotonlytellyouthesecondpartofthesentence,“shehasblackeyes”,whichprovidesthecorrectinformation;Ialsowarnyounottodrawtheplausiblebutwronginferencebysaying‘but’,and/orbyraisingmyeyebrows.Infact,‘but’,aswellasalladversativewordsandnonverbalsignals,hasjustthefunctionofstoppingplausiblebutincorrectinferences[9];itcouldthenbeparaphrasedas:“WhatIamgoingtosayinthenextpartofthesentence/discoursecannotbeinferred,andeven,thecontrarycouldbeinferred,fromthepreviouspartofthesentence/discourse”,
Themeaningofanadversativewordcouldthenberepresentedasfollows:
1.S2.S3.S4.S
hashashashas
thethethethe
goalthatAbelievesqgoalthatAbelievesp
goalthatAbelievesthatAcannotinferpfromqgoalthatAbelievesnotpfromq
CognitiveUnitsofadversative
Inraisingtheeyebrowswithanadversativefunction,SwantsAtobelieveboththefirstandthesecondpartofthesentence/discourse(CognitiveUnits1.and2.),butatthesametimeacknowledgesthatpcannotbeinferredfromq(CognitiveUnit3.),andeventhecontrary(notp)istrue(CognitiveUnit4.).
Thus,bothsurprise(case1.)andadversativewords(case3.)sharethesameelementofadisconfirmedexpectation:infact,aswellassurpriseoccurswhenanunexpectedbeliefisassumed,alsoanadversativesignalisprovidedwhenIwarnyouthatanunexpectedbeliefhastobeassumed,onewhichcontrastswithpreviousknowledge.
Finally,letusseewhatsemanticrelationshipcanbefoundbetweenraisingeyebrowsoutofsurprise(1.)andoutofemphasis(4.).AsIwanttoemphasiseawordoraclauseinmysentence,Ishowamorewakenedattentionmyselfinordertoaskmoreattentionfrommyinterlocutor,becausethepartofsentenceIamutteringisthecomment,mynewcontributionofinformation,andthenthepartIconsidermostimportantandmostworthofattention.Raisingtheeyebrowsinthiscasethenmeanssomethinglike:“paymoreattentionthanusual,becausethisisthenewest,leastobvious,mostrelevantinformationinwhatIamsaying”.Infact,thereisalinkbetweensurpriseandnewinformationbecauseanynewinformationispotentiallysomethingwecannotinfer.Thisis
EmotionalMeaning10
whytheelementofsomethingthatcannotbeinferredfrompreviousknowledgeis,atleastpotentially,presentalsointhecaseofemphasis.Emphasiscanberepresentedlikethis:
1.ShasthegoalthatAbelievesp
2.SbelievesthatAcannotinferpfromq3.ShasthegoalthatApaysattentiontop
CognitiveUnitsofemphasis
SwantsAtobelievep(CognitiveUnit1.),butSbelievesAcannotinferpbypreviousknowledge(CognitiveUnit2.),andaskAtopayattention(CognitiveUnit3.).
Tosumup,then,thesemanticelementofanewinformationthatcannotbeinferredfrompreviousknowledgeseemstobecommontoallfourcases:surprise,doubt,adversative,andemphasis.
5Theevolutionofmeanings
Theideathatacommoncoreofmeaningissharedbydifferentreadingsofthesamesignalimpliesahypothesisonthediachronic(historicandphilogenetic)evolutionofmeaningsofnonverbalsignals.Ourhypothesisisthatasignalisdisplayed,atitsfirstoccurrences,whilehavingaprimitivemeaning1..Supposeforinstancethattheeyebrowraisingfirstlymeansthatforthespeakersomeeventisdifferentfromexpected.Later,whenthatsignalisproducedindifferentcontexts,thankstoeachcurrentsituationsomespecificinferences(newbeliefs)maybedrawnfromthatsignal:forinstance,inasituation1.whatisdifferentfromexpectedisaneclipse,wheretheusuallyalwaysbrightsunisobscuredbysomething;inasituation2.,whatisdifferentfromexpectedisanincrediblethingthatmyinterlocutoristellingme.Now,supposetheseinferences,theseadditionalelementsofinformation,arealwaysthesameinthesameclassofcon-texts,attheextentthattheycometoberecurrentlyattachedtotheprimitivemeaning1.:inthiscasethenewinferencesbecomecrystallisedaroundtheprim-itivemeaning,thatis,theyarenotproducedonlyepisodically,buttheystarttosystematicallyformpartofanewmeaning,onemadeupbytheprimitivemeaningplusthecrystallisedinferences.Forexample,intheclassofcontexts“naturalphenomena”thesignalexhibitedmaystarttobeartwobeliefs(dis-confirmedexpectation+innaturalphenomena),whileintheclassofcontexts“verbalinteraction”thesamesignalmaybearagaintwobeliefs,butthefirstisthesameasintheformerclass,whilethesecondisadifferentone(disconfirmedexpectation+inwhatotherpeoplesay).Inthiscasetherecurrentinferenceshaveformedanewmeaning,wherethenewbeliefs,thatwerepreviouslyinferreda-systematicallyaccordingtothecontext,arenowpermanentlyattachedtotheprimitivemeaning,insuchawayastoformanewbunchofbeliefs-infact,anewmeaning-,thatisricherthantheprimitivemeaninginthatitbearsmorebeliefsthatitdidbefore,butstillincludestheprimitivemeaning.Thisis,in
EmotionalMeaning11
ourview,oneofthewaysanewmeaningevolvesfromaprimitivemeaning.Ac-cordingtothishypothesis,attheendofthisprocess,byanalyzingthedifferentmeaningsofthesamesignalweshouldfindthattheyallshareoneandthesamepartofmeaning-acommoncoreofmeaning-,buttheydifferfromoneanotherinthateachdifferentmeaninghasoneormorepartsthatdonotoverlapwiththeothermeanings.Thisstructureofpolysemicsignalsmayberepresentedinthisway:
MEANING1surprisedisconfirmed
expectationfromincredible
phenomenon
ofsentencefrompreviouspartdonotdrawmost
payattention
MEANING3adversativedisconfirmed
expectationnewinformation
6Meaning-basedversussurface-basedsystems
Inbuildingembodiedagentswithtalkingfaces,agentscapableofexpressiveandcommunicativebehavior,weconsideritimportantthattheagentexpresshiscommunicativeintentions.Supposeanagenthasthegoalofcommunicatingsomethingtosomeparticularinterlocutorinaparticularsituationandcontext:hehastodecidewhichwordstoutter,whichintonationtouse,andwhichfacialexpressiontodisplay.Iftheagentmovesonlyhislipstotalkbutusesnoothersignals(nointonationtomarkanaccentortheendofanutterance,nofacialexpression,nochangeinthegazedirection,nohandgesture,andsoon),theusermightsoonloosetheimpressionofdialoguingwithanembodiedagent.More-over,theusermighthaveahardtimeunderstandingwhattheagentissaying,sincenocommunicativenoraffectiveinformationotherthanplaintextwillbepresentinhisdiscourse.Suchaninteractionwilllackthenecessaryelementstoprovideanaturalinteraction.Sothecreationofanembodiedagentrequirestheagenttobeabletoexhibitnotonlymereexpressionofemotionbutalsootherexpressionswithsubtlecommunicativefunctions.Thetheorypresentedinthispaperbasedontheanalysisofvariouscommunicativeactsprovidesthenecessaryformalismfortheconstructionofsuchanagent.Itprovidesthefoundationsofawaytorepresentseveralcomplexelements(belief,goal,emotionalstate,powerrelationship,socialrelationshipandsoon)thatarepartofthementalstateoftheagent,aswellastheinferencerulesthattheagentusestodeducehisverbalandnonverbalbehaviors.Throughafine-grainrepresentationofmentalstates,we
EmotionalMeaning12
canmakeagentswithenoughknowledgeandflexibilityastoadapttodifferentcontextsandtoexhibitdifferentpersonalitiesandattitudes2.Afirstapproachofthisworkisbeingintegratedinadiscoursegenerationprogram[23]basedonMann,MatthiesenandThompson’s‘RhetoricalStructureTheory’(RST)[17].TheleavesoftheRSTtreecanbeeitherverbalornonverbalsignals.Agoal-mediaprioritisingmoduledetermineswhichmedia(face,gaze,speech)willbeused,whilethefunctionofthesynchronizermoduleistocomputetheoccurrenceanddurationofeachsignal(e.g.araisingeyebrowmaycoincidewithindividualwordswhileemotionaldisplaymayspanseveralclauses).Weareconsciousweareatthepremisesofsuchacreationbutwethinkthatanimatinganagentfromasystemthatisonlybasedonsurfacesignal(i.e.facialexpression)andnotonmeaning(i.e.communicativeact)willnotbeabletoembedthenaturalnessandrichnessofhumancommunication.
7
EmotionalExpressionastheCoreofCommunication:ConclusionandFurtherResearch
Inthispaperwehaveshownthatanemotionalcomponentofmeaning(sadness,happiness,worriness...)ispresentincommunicativesignalsotherfrommereex-pressionofemotion,namelytheexpressionoftheperformative;ontheotherhandwehaveshownthatacoreofmeaningwhichiscontainedintheexpres-sionofanemotion(surprise)isnotpresentonlyinthatemotionpersebutalsoinotherkindsofcommunicativesignals(emphasis,backchannelofdoubt,adversative).Theraisingoftheeyebrows,withitsmeaningofsurprise,thatisofadisconfirmedexpectation,maynotonlybefoundintheexpressionoftheemotionofsurprise,butitmayalsoaccompanyadversativeconjunctionsorem-phasisethecommentofasentence;andallofthesecommunicativesignalsimplysomethingnew,differentfromexpected,andthensurprising.Wearguethataconstructionofanimatedfacesbasedsimplyontherepresentationofsurfacesignalscannotcatchthesubtlenessandrichnessofthepossiblemeaningsthatunderlycommunicativefacialexpressions.Insteadacomponentialviewoffacialexpressioncouldbeparticularlyapttoproduceamodularandflexibleexpres-sivecapacityoffaces,makingthemabletoexhibitemotionalexpressioneitherbyitselforaspartofotherinteractionalcommunicativesignals.
References
1.G.BallandJ.Breese.Emotionandpersonalityinaconversationalagent.InS.PrevostJ.Cassell,J.SullivanandE.Churchill,editors,EmbodiedConversationalCharacters.MITpress,Cambridge,MA,2000.
2.J.Bates.Realismandbelievableagents.InLifelikeComputerCharacters’94,1994.
EmotionalMeaning13
3.J.BeskowandS.McGlashan.Olga-Aconversationalagentwithgestures.InProc.ofIJCAI’97-WorkshoponAnimatedInterfaceAgents-Makingthemintelligent,Nagoya,Japan,August1997.Morgan-KaufmannPublishers,SanFrancisco.4.J.Cassell,J.Bickmore,M.Billinghurst,L.Campbell,K.Chang,H.Vilhj´almsson,andH.Yan.Embodimentinconversationalinterfaces:Rea.InCHI’99,pages520–527,Pittsburgh,PA,1999.
5.J.Cassell,C.Pelachaud,N.I.Badler,M.Steedman,B.Achorn,T.Becket,B.Dou-ville,S.Prevost,andM.Stone.Animatedconversation:Rule-basedgenerationoffacialexpression,gestureandspokenintonationformultipleconversationalagents.InComputerGraphicsProceedings,AnnualConferenceSeries,pages413–420.ACMSIGGRAPH,1994.6.J.CassellandK.Th´orisson.Thepowerofanodandaglance:Envelopevs.emo-tionalfeedbackinanimatedconversationalagents.AppliedArtificialIntelligence,13(3),1999.
7.C.Castelfranchi,F.deRosis,R.Falcone,andS.Pizzutilo.Atestbedforinvesti-gatingpersonality-basedmultiagentcooperation.InEuropeanSummerSchoolofLogic,LanguageandInformation,Aix-en-Provence,France,1997.
8.C.CastelfranchiandI.Poggi.‘beh!’e‘beh...’.Analisisemanticadiunainteriezioneitaliana.InI.Poggi,Leinteriezioni.Studiodellinguaggioeanalisidellamente.Boringhieri,Torino,1981.
9.C.CastelfranchiandI.Poggi.Bugiefinsionisotterfugi.InPerunascienzadell’inganno.Carocci,Roma,1998.
10.R.ConteandC.Castelfranchi.CognitiveandSocialAction.UniversityCollege,
London,1995.
11.P.Doyle.Whenisacommunicativeagentagoodidea.InWorkshopon“Com-municativeAgents:TheUseofNaturalLanguageinEmbodiedSystems”,ThirdInternationalConferenceonAutonomousAgents,Seattle,May1999.
12.P.Ekman.Aboutbrows:Emotionalandconversationalsignals.InM.vonCranach,
K.Foppa,W.Lepenies,andD.Ploog,editors,Humanethology:Claimsandlimitsofanewdiscipline:contributionstotheColloquium,pages169–248.CambridgeUniversityPress,Cambridge,England;New-York,1979.
13.P.EkmanandW.Friesen.UnmaskingtheFace:Aguidetorecognizingemotions
fromfacialclues.Prentice-Hall,Inc.,1975.
14.P.EkmanandW.Friesen.FacialActionCodingSystem.ConsultingPsychologists
Press,Inc.,PaloAlto,CA,1978.
15.T.KodaandP.Maes.Agentswithfaces:Theeffectsofpersonificationofagents.
InHCI’96,August1996.
16.Y.Lee,D.Terzopoulos,andK.Waters.Realisticmodelingforfacialanimation.
InComputerGraphicsProceedings,AnnualConferenceSeries,pages55–62.ACMSIGGRAPH,1995.
17.W.C.Mann,C.M.I.M.Matthiessen,andS.Thompson.Rhetoricalstructuretheory
andtextanalysis.TechnicalReport89-242,ISIResearch,1989.
18.C.Pelachaud,N.I.Badler,andM.Steedman.Generatingfacialexpressionsfor
speech.CognitiveScience,20(1):1–46,January-March1996.
19.S.M.Platt.AStructuralModeloftheHumanFace.PhDthesis,Universityof
Pennsylvania,Dept.ofComputerandInformationScience,Philadelphia,PA,1985.20.I.PoggiandE.MagnoCaldognetto.Manicheparlano.GestiePsicologiadella
comunicazione.Padova:Unipress,1997.
21.I.PoggiandC.Pelachaud.Performativefaces.SpeechCommunication,26:5–21,
1998.
EmotionalMeaning14
22.I.PoggiandC.Pelachaud.Facialperformativeinaconversationalsystem.In
S.PrevostJ.Cassell,J.SullivanandE.Churchill,editors,EmbodiedConversationalCharacters.MITpress,Cambridge,MA,2000.
23.I.Poggi,C.Pelachaud,andF.deRosis.Eyecommunicationinaconversational
3dsyntheticagent.SpecialIssueonBehaviorPlanningforLife-LikeCharactersandAvatarsofAICommunications,2000.
24.B.ReevesandC.Nass.Themediaequation:Howpeopletreatcomputers,television
andnewmedialikerealpeopleandplaces.CSLIPublications,Stanford,CA,1996.25.A.TakeuchiandT.Naito.Situatedfacialdisplays:Towardssocialinteraction.
InProceedingsofACMCHI’95-ConferenceonHumanFactorsinComputingSystems,volume1,pages450–455,1995.26.K.R.Th´orisson.Layeredmodularactioncontrolforcommunicativehumanoids.
InComputerAnimation’97,Geneva,Switzerland,1997.IEEEComputerSocietyPress.
27.J.H.Walker,L.Sproull,andR.Subramani.Usingahumanfaceinaninterface.
InHumanFactorsinComputingSystems,pages85–91,April1994.
因篇幅问题不能全部显示，请点此查看更多更全内容