A framework for integrating gesture generation models into interactive conversational agents


Embodied conversational agents (ECAs) benefit from non-verbal behavior for natural and efficient interaction with users. Gesticulation - hand and arm movements accompanying speech - is an essential part of non-verbal behavior. Gesture generation models have been developed for several decades: starting with rule-based and ending with mainly data-driven methods. To date, recent end to- end gesture generation methods have not been evaluated in a real-time interaction with users. We present a proof-of-concept framework, which is intended to facilitate evaluation of modern gesture generation models in interaction. We demonstrate an extensible open-source framework that contains three components: 1) a 3D interactive agent; 2) a chatbot backend; 3) a gesticulating system. Each component can be replaced, making the proposed framework applicable for investigating the effect of different gesturing models in real-time interactions with different communication modalities, chatbot backends, or different agent appearances. The code and video are available at the project page https://nagyrajmund.github.io/project/gesturebot.

Divisions: College of Engineering & Physical Sciences > Aston Institute of Urban Technology and the Environment (ASTUTE)
College of Engineering & Physical Sciences
Additional Information: Copyright © 2021 by International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). Permission to make digital or hard copies of portions of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
Event Title: 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021
Event Type: Other
Event Dates: 2021-05-03 - 2021-05-07
Uncontrolled Keywords: Conversational embodied agents,Non-verbal behavior synthesis,Artificial Intelligence,Software,Control and Systems Engineering
ISBN: 978-1-4503-8307-3, 9781713832621
Last Modified: 18 Jun 2024 17:05
Date Deposited: 17 Sep 2021 11:00
Full Text Link: https://arxiv.o ... /abs/2102.12302
http://www.ifaa ... /pdfs/p1779.pdf
Related URLs: http://www.scop ... tnerID=8YFLogxK (Scopus URL)
http://www.ifaa ... ings/aamas2021/ (Publisher URL)
https://dl.acm. ... 3463952.3464235 (Publisher URL)
PURE Output Type: Conference contribution
Published Date: 2021-05-03
Authors: Nagy, Rajmund
Kucherenko, Taras
Moell, Birger
Pereira, André
Kjellström, Hedvig
Bernardet, Ulysses (ORCID Profile 0000-0003-4659-3035)



Version: Draft Version

| Preview

Export / Share Citation


Additional statistics for this record