Expressive Speech-Driven Lip Movements with Multitask Learning

Simple item page
dc.contributor.authorSadoughi, Najmeh
dc.contributor.authorBusso, Carlos A.
dc.contributor.utdAuthorSadoughi, Najmeh
dc.contributor.utdAuthorBusso, Carlos A.
dc.date.accessioned2019-08-05T19:50:27Z
dc.date.available2019-08-05T19:50:27Z
dc.date.created2018-05-15
dc.descriptionFull text access from Treasures at UT Dallas is restricted to current UTD affiliates (use the provided Link to Article).
dc.description.abstractThe orofacial area conveys a range of information, including speech articulation and emotions. These two factors add constraints to the facial movements, creating non-trivial integrations and interplays. To generate more expressive and naturalistic movements for conversational agents (CAs) the relationship between these factors should be carefully modeled. Data-driven models are more appropriate for this task than rule-based systems. This paper provides two deep learning speech-driven structures to integrate speech articulation and emotional cues. The proposed approaches rely on multitask learning (MTL) strategies, where related secondary tasks are jointly solved when synthesizing orofacial movements. In particular, we evaluate emotion recognition and viseme recognition as secondary tasks. The approach creates shared representations that generate behaviors that not only are closer to the original orofacial movements, but also are perceived more natural than the results from single task learning.
dc.description.departmentErik Jonsson School of Engineering and Computer Science
dc.description.sponsorshipUS National Science Foundation grant IIS-1718944.
dc.identifier.bibliographicCitationSadoughi, N., and C. Busso. 2018. "Expressive speech-driven lip movements with multitask learning." Proceedings - International Conference on Automatic Face and Gesture Recognition, 13th: 409-415, doi:10.1109/FG.2018.00066
dc.identifier.issn9781538623350 (ISBN)
dc.identifier.urihttps://hdl.handle.net/10735.1/6771
dc.language.isoen
dc.publisherInstitute of Electrical and Electronics Engineers Inc.
dc.relation.isPartOfProceedings - International Conference on Automatic Face and Gesture Recognition, 13th
dc.relation.urihttp://dx.doi.org/10.1109/FG.2018.00066
dc.rights©2018 IEEE
dc.subjectLearning
dc.subjectGesture
dc.subjectSpeech
dc.subjectConversation
dc.subjectEmotion recognition
dc.titleExpressive Speech-Driven Lip Movements with Multitask Learning
dc.type.genrearticle

Files

Original bundle

Now showing 1 - 1 of 1
Thumbnail Image
Name:
JECS-6770-279919.12-Link.pdf
Size:
164.67 KB
Format:
Adobe Portable Document Format
Description:
Link to Article
Download

Collections

Busso-Recabarren, Carlos A.