[ English | Japanese ]

VocaWatcher: Natural Singing Motion Generator for a Humanoid Robot

Shuuji Kajita / Tomoyasu Nakano / Masataka Goto / Yosuke Matsusaka / Shin'ichiro Nakaoka / Kazuhito Yokoi
National Institute of Advanced Industrial Science and Technology (AIST)

Links:
• VocaListener: A Singing-to-Singing Synthesis System Based on Iterative Parameter Estimation
• VocaListener2: A Singing Synthesis System Able to Mimic a User's Singing in Terms of Voice Timbre Changes As Well As Pitch and Dynamics
• Analysis and Automatic Detection of Breath Sounds in Unaccompanied Singing Voice

Abstract:

This research presents VocaWatcher, a novel robot motion generator that enables a humanoid robot to sing a song with realistic facial expressions as well as natural synthesized singing voices. Such a robot singer is important not only for showing an attractive humanoid robot application in an entertainment scene, but also for promoting the state-of-theart integration of robot engineering, music processing, and image processing. To overcome difficulties generating natural facial expressions precisely synchronized with singing voices, VocaWatcher imitates a human singer by analyzing a video clip of the human singing recorded by a single video camera. It can control mouth, eye, and neck motions by imitating the corresponding human movements estimated without any markers in the video, and also synthesize singing voices by imitating the pitch and dynamics of the human singing in the same video.

Fig 1. Overview of VocaWatcher.

Demonstrations:

Acknowledgments:

This research utilized the RWC Music Database "RWC-MDB-P-2001" (Popular Music). In our current implementation, VocaWatcher uses commercial singing synthesis software based on Yamaha's Vocaloid2 technology.

We thank to members of the Humanoid Robotics Group of AIST, especially, Kanako Miura for her help and advice at an early stage experiment, and Kenta Yonekura for his work of choreographing HRP-4C's arm motions. We also thank to Yoshio Mastumoto for his helpful advice. Finally, we gratefully acknowledge the support of Hirohisa Hirukawa and Satoshi Sekiguchi, directors of Intelligent Systems Research Institute and Information Technology Research Institute of AIST, respectively.

References:

1. Masataka Goto, Tomoyasu Nakano, Shuuji Kajita, Yosuke Matsusaka, Shin'ichiro Nakaoka, and Kazuhito Yokoi:
   "VocaListener and VocaWatcher: Imitating a Human Singer by Using Signal Processing",
   In Proceedings of the 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2012),
   pp.5393-5396, March 2012.
   [PDF]
2. Shuuji Kajita, Tomoyasu Nakano, Masataka Goto, Yosuke Matsusaka, Shin'ichiro Nakaoka, and Kazuhito Yokoi,
   "VocaWatcher: Natural Singing Motion Generator for a Humanoid Robot,"
   In Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2011),
   pp.2000-2007, September 2011.
   [PDF][DOI (IEEE Xplore)]

The copyright of each publication is retained by the corresponding academic society (copyright holder). Copyright and Consent Form (IEEE).