Speech-Driven Animation with Meaningful Behaviors

Simple item page
dc.contributor.authorSadoughi, Najmeh
dc.contributor.authorBusso, Carlos
dc.contributor.utdAuthorSadoughi, Najmeh
dc.contributor.utdAuthorBusso, Carlos
dc.date.accessioned2020-02-28T17:26:06Z
dc.date.available2020-02-28T17:26:06Z
dc.date.issued2019-04-05
dc.descriptionDue to copyright restrictions and/or publisher's policy full text access from Treasures at UT Dallas is limited to current UTD affiliates (use the provided Link to Article).
dc.descriptionSupplementary material is available on publisher's website. Use the doi.org link below.
dc.description.abstractConversational agents (CAs) play an important role in human computer interaction (HCI). Creating believable movements for CAs is challenging, since the movements have to be meaningful and natural, reflecting the coupling between gestures and speech. Studies in the past have mainly relied on rule-based or data-driven approaches. Rule-based methods focus on creating meaningful behaviors conveying the underlying message, but the gestures cannot be easily synchronized with speech. Data-driven approaches, especially speech-driven models, can capture the relationship between speech and gestures. However, they create behaviors disregarding the meaning of the message. This study proposes to bridge the gap between these two approaches overcoming their limitations. The approach builds a dynamic Bayesian network (DBN), where a discrete variable is added to constrain the behaviors on the underlying constraint. The study implements and evaluates the approach with two constraints: discourse functions and prototypical behaviors. By constraining on the discourse functions (e.g., questions), the model learns the characteristic behaviors associated with a given discourse class learning the rules from the data. By constraining on prototypical behaviors (e.g., head nods), the approach can be embedded in a rule-based system as a behavior realizer creating trajectories that are timely synchronized with speech. The study proposes a DBN structure and a training approach that (1) models the cause-effect relationship between the constraint and the gestures, and (2) captures the differences in the behaviors across constraints by enforcing sparse transitions between shared and exclusive states per constraint. Objective and subjective evaluations demonstrate the benefits of the proposed approach over an unconstrained baseline model. ©2019 Elsevier B.V.
dc.description.departmentErik Jonsson School of Engineering and Computer Science
dc.description.sponsorshipNational Science Foundation grants IIS:1718944
dc.identifier.bibliographicCitationSadoughi, N., and C. Busso. 2019. "Speech-driven animation with meaningful behaviors." Speech Communication 110: 90-100, doi: 10.1016/j.specom.2019.04.005
dc.identifier.issn0167-6393
dc.identifier.urihttp://doi.org/10.1016/j.specom.2019.04.005
dc.identifier.urihttps://hdl.handle.net/10735.1/7312
dc.identifier.volume110
dc.language.isoen
dc.publisherElsevier B.V.
dc.rights©2019 Elsevier B.V. All Rights Reserved.
dc.source.journalSpeech Communication
dc.subjectComputer animation
dc.subjectHuman-computer interaction
dc.subjectConversation
dc.subjectVariables (Mathematics)
dc.subjectSpeech
dc.titleSpeech-Driven Animation with Meaningful Behaviors
dc.type.genrearticle

Files

Original bundle

Now showing 1 - 1 of 1
Thumbnail Image
Name:
JECS-6770-260953.62-LINK.pdf
Size:
164.48 KB
Format:
Adobe Portable Document Format
Description:
Link to Article
Download

Collections

Busso-Recabarren, Carlos A.