Emotion Embodied Avatars: Novel Remote Communication with Motion Unit AI

Human-rich communication relies on non-verbal cues heavily. Among them, we are conducting research focusing on physical movements that are particularly important for enriching communication and are academically challenging. In particular, we built an AI that links human emotions and intentions with the “Motion Unit,” which is an analysis unit for body movements, and we are conducting research to generate emotionally rich motions of avatars through a system of international arts-science and industry-academia collaboration.

This research is conducted as an adopted project “Innovative Remote Technology Development by Utilizing Artificial Intelligence (scheduled for 2021-2024)” by of the New Energy and Industrial Technology Development Organization (NEDO).

Background and Aim

Nonverbal information plays an essential role in our daily life.
The difficulty in communicating nonverbal information online has led to a variety of problems.
Among nonverbal signals, this research focuses on human body movements, which contain a lot of vital latent information.

What we do

Define an analysis unit of body movement (Motion Unit) in terms of social psychology and machine learning approach.
Develop a high-precision motion capture system that is robust against noise and data loss.
Create a Motion Unit AI that estimates people’s emotions and intentions.
Propose a method for automatically generating emotional movements of avatars using Motion Unit AI.
Apply to real-time character animation.

Expected Outcomes

Motion Unit, an analysis unit of body movement
Motion Unit AI, estimating people’s emotions and intentions based on Motion Unit
Emotional movements of characters generated automatically
A remote communication environment that enriches all forms of online communication, from daily conversations and workplace meetings to live performances
High-precision motion capture system resilient to data loss due to noise, occlusion, etc.

Research Members

PI: Yoshifumi Kitamura (Research Institute of Electrical Communication, Tohoku University)

Taku Komura (Hong Kong University / University of Edinburgh / Tohoku University)
Ken Fujiwara (National Chung Cheng University / Tohoku University)
Chia-Huei Tseng (Research Institute of Electrical Communication, Tohoku University)
Cheng Miao (Research Institute of Electrical Communication, Tohoku University)

Participating Companies