Repositório ISCTE-IUL

Purpose – To analyses the effect of emotions obtained by oral reproduction of advertising slogans established via Voice Emotion Response software on brand recall by gender; and to show the relevance for marketing communication of combining “Human-Computer-Interaction (HCI)” with“affective computing (AC)” as part of their mission. Design/methodology/approach – A qualitative data analysis did the review of the scientific literature retrieved from Web-of-Science Core Collection (WoSCC), using CiteSpace’scientometric technique; the quantitative data analysis did the analyse of brand recall over a sample of Taiwan’ participants by “optimal data analysis”. Findings – Advertising effectiveness has a positive association with emotions; brand recall varies with gender; and “HCI” connected with “AC” is an emerging area of research. Research limitations/implications – The selection of articles obtained depend the terms used in WoSCC, and this study used of only five emotions. Still the richness of the data gives some compensation. Practical implications – Marketers involved with brands need a body of knowledge on which to base their marketing communication intelligence gathering and strategic planning. Originality/value - Provides exploratory research findings related to the use of automatic tools capable of mining emotions by gender in real time, that could enhance the feedback of customers toward their brands.


Introduction
Emotions play an important role not only in successful and effective human-human communication, but also in human rational learning (Cambria, 2016).Affective Computing recognizes this inextricable link between emotions and cognition, and works to narrow the communication gap between the highly emotional human, and the emotionally-challenged computer, by developing computational systems that respond to the affective states of the user (Calvo & D'Mello, 2010).According to Rukavina, Sascha, Holger, et al. (2016), Affective Computing aims to detect users' mental states, revealing which feature customers enjoy and excluding those that receive negative feedback.It, therefore, shows great potential to enhance companies' capabilities to manage their customer relationships, by improving their marketing strategies, and constantly gathering and predicting the attitudes of the general public toward their products and brands.The basic principle behind most Affective Computing systems is that they automatically recognize and respond to users' affective states during their interactions with a computer, and thus provide data which can be used to enhance the quality of the interaction.Essentially, this is achieved by measuring multimodal signals, namely, speech, facial expressions and/or psychobiology, and from these measurements, designing a computer interface which is more usable and effective.Affective Computing focuses on extracting a set of emotion labels (Picard, 1997;Zeng et al, 2009;Calvo & D'Mello, 2010;Schuller, Batliner, Steidl, & Seppil, 2011;Gunes & Shuller, 2012), and polarity detection, usually a binary classification task with output such as positive versus negative, or like versus dislike (Pang & Lee, 2008;Liu, 2012;Cambria et al, 2013).
According to Fridja (2007), emotions have a short life in the field of consciousness, motivating behavior that requires immediate attention.Lang (2010) confirms the measurement of emotions as being critical in the advertising process, pointing to the fact that emotions are often conveyed by an advertising slogan, which arouses an appeal within consumers that positively predisposes them towards the message being communicated, and thereby helping to deliver the desired image of brand position, which could generate enourmous profit (Teixeira, Webel, & Piters, 2012).Several marketing researchers (e.g.Bagozzi, Gopinath & Nyer, 1999;Lee, Broderick & Chamberlain, 2007;Wang, Chien & Moutinho, 2015), state that Affective Computing supersedes self-report measures as a vehicle for evaluating emotions; and Cambria (2016) refers to the design of automatic Web mining tools for use in real time, as one of the most active research and development areas in Affective Computing.
Notable fallouts in marketing and financial market prediction have raised the interest of the scientific community and the business world in Affective Computing, which allows for the leverage of humancomputer interaction, information retrieval, and multimodal signal processing.
This study provides information about Affective Computing, by measuring and visualizaing the retrieved bibliographic references from the Web-of-Science Core Collection, between 1991 and 2016.It organizes these references into homogeneous clusters, identifying the most relevant sub-areas of research, and those references which are the most innovative, with more citations, with more connections between sub-areas, and which are responsible for recent advances in Affective Computing.

Marketing intelligence gathering, and strategic planning based on this body of knowledge on Affective
Computing, provides opportunities for companies to enhance their customer relationship capabilities, to capture general public emotions in respect of their products and services, and to react accordingly.
The remainder of this article consists of four sections.The next section deals with the methodology, explaining the data collection approach and the quality of CiteSpace results.The third section presents the results, identifying in clusters, the most efficient information on Affective Computing in metrical and graphical terms.Finally, concluding thoughts are offered in section four.

Methodology
Bibliographical records concerning Affective Computing, and published since 1991, were collected from the Web-of-Science of Thomson Reuters in the field of marketing, including the related areas of mangement and psychology.The resultant dataset contained 5,078 records relating to 782 core articles and 4,296 references that cite those articles at least once.This data set was exported to CiteSpace for the scientometric review.
CiteSpace, is a free Java application for visualizing and analyzing emerging trends and changes in scientific literature/ It allows for a multiple-perspective structural, and temporal approach, and semantic patterns of references which help in interpreting the nature of clusters (Chen, 1999(Chen, , 2013)).Structural metrics include centrality, modularity, and silhouette.High centrality identifies references that connect different clusters, and are responsible for the expansion of knowledge; high modularity shows that the references are distributed in non-overlapping clusters with clear boundaries; and high silhouette means great homogeneity of clusters, facilitating the uniformity of their labels.
Temporal metrics include citation burstness and novelty, burstness being used as a term to identify emergent references whose citation counts increase abruptly in a short period of time, and high novelty indicating references that represent creative ideas.
All the records were grouped into 40 non-overlapping and homogeneous clusters (#) based on their interconnectivity, applying the pathfinder pruning algorithm and a g-index with a scaling factor k =5.Seven major clusters were found in the references, and labeled #0 to #6.These account for 71.3% of the references (Figure 1).

Results from CiteSpace
The advantage of Citespace is its ability to provide the most useful information about Affective Computing in metrical and graphical terms, distributing the 5,078 references by clusters, identifying those with more links (centrality), that are cited, more creative (sigma), and more explosive (burstness).CiteSpace represents a fountain of opportunities for managers to capture general public emotion in respect of products and services, thereby enhancing the capabilities of companies to improve their customer relatiosnhips.

Characterization of the Major Clusters
The seven major clusters or areas of research, are labeled by these algorithtms (TF*IDF; LLR; MI), according to the index terms coming from those references that cite members of each cluster, presented in Table 1.The oldest cluster in Affective Computing is "emergence" (#2), with an average year of publications 1997, and the youngest cluster is "physiological signal" (#1), with an average year of publications 2009.On average, the mean year difference between publication and citation varies between a miminum of 3 years, corresponding to the major clusters #0 (multichannel physiology) and #1, and a maximum of 7 years, corresponding to the "speech", cluster #4.The largest cluster (#0) has 35 members and a silhouette value of 0.731, being labeled as "multichannel physiology" by both LLR and TFIDF, and as "modality" by MI.This cluster has 491 citations, being most quoted (0.17) in "Affect detection: an interdisciplinary review of models, methods, and their applications" (Calvo & D'Mello, 2010).
The second largest cluster (#1) has 27 members and a silhouette value of 0.802, being labeled as "response" by LLR, "physiological signal" by TFIDF, and "music" by MI.This cluster has 263 citations, being most quoted (0.11) in "Consistent but modest: a meta-analysis on unimodal and multimodal affect detection accuracies from 30 studies" (DMello & Kory, 2012).
The third largest cluster (#2) has 23 members and a silhouette value of 0.965, being labeled as "computer" by LLR, "emergence" by TFIDF, and "human-computer interaction" by MI.This cluster has 92 citations, being most quoted (0.3) in "On the role of embodiment in the emergence of cognition and emotion" (Pfeifer, 2011).
The fourth largest cluster (#3) has 21 members and a silhouette value of 0.857, being labeled as "naturalistic interaction" by both LLR and TFIDF, and as "order crossing" by MI.This cluster has 180 citations, being most quoted (0.38) in "Affect detection: an interdisciplinary review of models, methods, and their applications" (Calvo & D'Mello, 2010).
The fifth largest cluster (#4) has 21 members and a silhouette value of 0.724, being labeled as "strategies" by LLR,"speech" by TFIDF, and "audio-visual spontaneous emotion recognition" by MI.This cluster has 229 citations, being most quoted (0.24) in "Cross-corpus acoustic emotion recognition: variances and strategies" (Schuller & all, 2010).
The sixth largest cluster (#5) has 18 members and a silhouette value of 0.731, being labeled as "delivery" by TFIDF, "content delivery" by LLR and "modality" by MI.This cluster has 157 citations, being most quoted (0.17) in "Affect detection: an interdisciplinary review of models, methods, and their applications" (Calvo & D'Mello, 2010).
The seventh largest cluster (#6) has 12 members and a silhouette value of 0.95, being labeled as "agent" by TFIDF and LLR and "multi-score learing" by MI.This cluster has 78 citations, being most quoted (0.42) in "Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features" (D'Mello & Graesser, 2010)

More inovative, central and cited references by cluster
Intellectual collaboration between references is fundamental to the overall understanding of a knowledge domain (Hu & Racherla, 2008).References in the literature with more connections between different clusters or sub-areas of research (centrality), more citations, and more innovative (sigma) are revolutionary scientific publications.
"Affective computing" (Picard, 1997) and "Toward machine emotional intelligence: Analysis of affective physiological state" (Picard, Vyzas & Healey, 2001) are the two most creative references ever in this area, according to their high sigma values; and, together with "Toward an affect-sensitive multimodal humancomputer interaction" (Pantic & Rothkrantz, 2003), are central references for the expansion of knowledge due to their connection between different clusters, summarized below.
The book "Affective computing" (Picard, 1997), from cluster "emergence" (#2), states that the future "ubiquitous computing" environments will need to have human-centered designs instead of computercentered designs, a fundamental component of human-human communication.Computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground.This reference prompted a wave of interest among computer scientists and engineers looking for ways to improve human-computer interfaces by co-ordinating emotion and cognition with task constraints and demands.Picard described three types of affective computing applications: first, systems that detect the emotions of the user, second, systems that express what a human would perceive as an emotion (e.g., an avatar, robot, and animated conversational agent), and third, systems that actually "feel" an emotion.
The paper "Toward machine emotional intelligence: Analysis of affective physiological state" (Picard, Vyzas & Healy (2001), from cluster "delivery" (#5), proposed that machine intelligence needed to include emotional intelligence and demonstrated results suggesting the potential for developing a machine's ability to recognize human affective states given physiological signals.
The paper "Toward an affect-sensitive multimodal human-computer interaction" (Pantic & Rothkrantz, 2003), from cluster "naturalistic interaction" (#3), reviewed the efforts toward the single modal analysis of artificial affective expressions and discussed how to integrate into computers a number of components of human behavior in the context-constrained analysis of multimodal behavioral signals.
The more innovative, central and cited references are shown in Table 2, by title, authors, year, source and cluster.Applications" (Calvo & D'Mello , 2010), are the references with more citations, both from the major cluster "multichannel physiology" (#0), summarized below.
The paper "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions" (Zeng, Riisman, & Huang, 2009), developed algorithms to detect subtleties and changes in the user's affective behavior, in order to initiate interactions based on this implicit information rather than on explicit messages usually involved in the tradition interface devices, such as the keyboard and mouse.These algorithms are intended to process naturally-occurring human affective behavior, with a view to multimodal fusion for human affect analysis, including audiovisual, linguistic, paralinguistic and multi-cue visual fusion based on facial expressions, head movements, and body gestures.
The paper "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications" (Calvo & D'Mello,2010), stresses the need to include within Affective Computing practice, emotion theories that have emerged, and that rely on cross-disciplinary collaboration and active sharing of knowledge.It reviews models that emphasize emotions as expressions, embodiments, outcomes of cognitive appraisals, social constructs, and products of neural circuitry, aiming to be a useful tool for new researchers by providing taxonomy of resources for further exploration and discussing different theoretical viewpoints and applications.

Trends in Affective Computing
A timeline view shows references through time and clusters (Figure 1).CiteSpace identifies all references by the first name of the author(s).The size of the nodes corresponds to the number of citations of the reference, i.e., a large node indicates many citations.Citations with an unexpected increase over a short period of time (burst strength) are marked as red rings around a node.
The majority of references were published after 2000, and the more active clusters are #0 and #1, being essential to the literature of affective computing, the main papers about which have already been discussed.Burstness provides a temporal prespective, indicating where the frequency of a reference increases abruptly in relation to its peers during a short period of time (Lee, Chen, & Tsai, 2016).A citation burst has two attributes: the intensity (strength), and the length of time the status lasts.Healey, 2001), are respectively the second and third references with strong intensity, the last also showing the highest duration of six years, over 2003-2009, as well as "Toward an affect-sensitive multimodal humancomputer interaction" (Pantic, & Rothkrantz, 2003), over the period 2005-2011.
"Emotion recognition in human-computer interaction" (Cowie, Douglas-Cowie, Tsapatsoulis, & et al., 2001), has also been one of the most comprehensive and widely cited reference in reviewing the efforts to reach a single modal analysis of artificial affective expressions, and in providing a comprehensive summary of qualitative acoustic correlations for prototypical emotions.The writers of that paper discuss the recognition of seven different human negative and neutral emotions, (bored, disengaged, frustrated, helpless, over-strained, angry, impatient) by technical systems, focusing on problems of data gathering and modelling, in an attempt to create a "Companion Technology" for Human Computer Interaction that allows the computer to react to human emotional signals.
"Affective computing: challenges", (Picard,2003), from cluster #3, shows a statistically significant flutuation over the period 2006-2008.It raises and responds to several criticisms of affective computing, emphasizing the need for a balance, and articulated state-of-the art research challenges, especially with respect to affect in human-computer interaction.This paper suggested that designers of future computing can continue with the development of computers that ignore emotions, or they can take the risk of making machines that recognize emotions, communicate them, and perhaps even "have" them, at least in the ways in which emotions aid in intelligent interaction and decision making.
With the exception of Zeng, Roisman and Huang, (2009) and Calvo and D'Mello, (2010), previously noted, below is a summary of the references that have and abrupt increase of citations by 2016.
"LIBSVM: A Library for Support Vector Machines" (Chang & Lin, 2011) presents details of how to implement a library for support vector machines (SVMs), and the package LIBSVM.It discusses in detail, issues concerning how to solve SVMs' optimization problems, theoretical convergence, multi-class classification, probability estimates, and parameter selection.
"DEAP: A Database for Emotion Analysis Using Physiological Signals" (Koelstra, 2012), considered vital for multimedia information retrieval, characterizes multimedia content with relevant, reliable, and discriminating tags.This reference presents a multimodal dataset for the analysis of spontaneous emotions, where implicit tagging of videos using affective information helps recommendation and retrieval systems to improve their performance.The dataset was made publicly available and other researchers were encouraged to use this data for testing their own affective state estimation methods.
"Emotion recognition based on physiological changes in music listening" (Kim & Elizabeth, 2008), investigates the potential of physiological signals for emotion recognition as opposed to audiovisual emotion channels such as facial expression or speech.This paper develops a novel scheme of emotion-specific multilevel dichotomous classification and shows an improvement in its performance compared with direct multiclass classification.Computer systems are now attempting to interact more naturally with the users as human beings.Graphical user interfaces are becoming more flexible and intelligent to adapt to different human interests.The adaptive systems applications learn the user interactions and then make user-friendly platforms.The application of the recognized affect in creating a diversified and context-specific response would be more domain specific, forming another level of work in Affective Computing, one with importance because of its wide application in different areas.
The accuracy of human emotion recognition has been improved by utilizing advanced analysis methods and techniques including image processing, voice recognition, natural language processing, and electroencephalography devices, which are time and resource-consuming, and which have certain preequisites such as the availability of a webcam, microphone, and highly technical equipment.This contrasts with the common situation whereby computers, and portable digital devices such as handhelds, tablets and mobile phones, process and analyze keyboard keystroke dynamics, mouse movements, and touch screen interactions (Bakhtiyari & Husain, 2014).Since the mouse is available in all computers, and users rely very much on this, it is possible to capture the human physiological signals for determining the human affective state via the mouse in a non-intrusive manner, without the user being consciously being aware of a physiological data-capturing device.Hence, a solution is found through the use of low-cost equipment,-, without imposing any burdens on the user (Fu, Leong, Hong, Grace, & et al., 2014).The importance of touch in communicating emotions and intensifying interpersonal communication has been analyzed in Affective Computing to detect and display emotions (Eid & Osman, 2016).
The development of systems capable of mining emotions and sentiments over the Web in real time to track public viewpoints on a large scale, represents one of the most active research and development areas, being important not only for commercial purposes but also for monitoring hostile communications or model cyberissue diffusion (Cambria, 2016).The Web is evolving to become an area in which consumers are defining future products and services, as it has made users more enthusiastic about sharing their emotions and opinions through several online collaboration media, like social networks, online communities, blogs, and wikis, in all fields related to everyday life, such as commerce and tourism.With the increasing number of webcams installed in end-user devices such as smart phones, touchpads, and netbooks, an greater volume of affective information is being posted to social online services in an audio or audiovisual format rather than on a purely textual basis.Public opinion is destined to gain increased prominence, and so is affective computing with its capacity for recognizing and classifying emotions.
Table 4 provides examples of some recent research studies that have investigated references included in the two major clusters (#0 and #1) just discussed.Affective computing is mainly interpreted in terms of emotions and sentiments, with an emphasis on the classification and recognition of emotions via human computer interaction, and the provision of implicit information about the changes in the affective states.This is depicted in Figure 2, which displays the keywords assigned to each reference in the dataset.

Concluding Thoughts
This article has traced the advancement of affective computing through the analysis of expert references in the literature.It has done this by using computational techniques to discern patterns and trends at various levels of abstraction: cited, central, innovative, and burstness references; sources of publications; and keywords.The major clusters are #0, "multichannel physiology", and the newest cluster #1 "physiological signal", which continue to be the most cited, and to demonstrate the largest burst citations.The most creative and central references are those of Picard (1997), andPicard, Vyzaz andHeley (2001); and those with recent burstness are seen to come from Kim and Elizabeth (2008), Calvo and D'Mello (2010), Chang and Lin (2011), Koelstra (2012);and Zeng, Riisman and Huang (2009), the latter also being the most cited reference ever in this area.

Figure 1 :
Figure 1: Seven major clusters in AC, labeled by TF*IDF

Figure 1 :
Figure 1: Timeline view by author's first name and cluster

Figure 2 :
Figure 2: Network with the relevant keywords

Table 1 :
Major clusters, average year of publications and citations

Table 2 :
The more innovative, central and cited references by clusters in Affective Computing "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions"(Zeng, Riisman,   & Huang, 2009), and "Affect Detection: An Interdisciplinary Review Models, Methods, and Their

Table 3
lists the references with the strongest citation bursts across the entire dataset, according to the clusters to which they belong.It can be seen that most of the references started to burst in year 2000 and "Toward machine emotional intelligence: Analysis of affective physiological state"(Picard, Vyzas, &

Table 3 :
Trends on Affective Computing by periods of time

Table 4 :
More relevant citers of the most active clusters