Outline

The purpose of our project is to provide a method and tools for the online transmission of spatial audio information with a low computational cost. The online transmission of spatial audio is illustrated in Fig. 1.

Transmission of spatial audio information
Figure 1 Transmission of spatial audio information

Spatial audio information is captured, transmitted, and presented to the listener at the remote site. The reproduced sound gives the listener a sensation of being listening to the sound in the space where the sound is actually recorded.

[Conventional methods]

Dummy head recording

A dummy head that simulates a head and torso of a human has been used for the binaural recording. Figure 2 shows pictures of a dummy head (Head And Torso Simulator). The dummy head has microphones in its left and right ear canals.

Photographs of a dummy head
Figure 2 Head And Torso Simulator (HATS)

When the sounds recorded by the microphones in the dummy head are presented to the listener through headphones, a sensation of being listening to these sounds in the real world environment can be given to the listener (Fig.3).

Because of its size and weight, however, it is not very easy to use the conventional dummy head.

Microphone array

Spatial audio information can be captured by a microphone array with tens or hundreds of microphones and reproduced by a loudspeaker array as illustrated in Fig. 4. In this case, headphones are not necessary and a wide listening area can be provided. It reqiures, however, high bit rate and high computational cost for transmitting and processing the data.

[Miniature head simulator system]

A miniature head simulator system is a new technology for capturing and processing spatial sound. It consists of a tetrahedral microphone system and a signal processor [1]. Figure 5 shows the microphone systems that consist of 4 omnidirectional microphones. These microphones are at the vertices of a tetrahedron. Since each side of the tetrahedron is as short as a few cm, the microphone system is far smaller than the conventional dummy head (Figure 6).

Photograph of a miniature head simulator
Figure 5 Microphone systems of a miniature head simulator
Each side of a tetrahedron is 30 mm (left) and 15 mm (right).


Figure 6 Comparison with the conventional dummy head


Panning on demand

A unique feature of this technology is that it allows for interactive manipulation of panning.

In the dummy head recording, since the front and back are physically determined by the shape of the dummy head, directions of sound images are also determined by the relative position of the sound sources and the dummy head.

In the miniature head simulator, the front can be arbitrary determined and therefore panning of the sound can be controlled by the listener during the reproduction.


Demonstration of the miniature head simulator

Here, we provide samples of sounds recorded by the miniature head simulator.

In these demonstrations, stereo headphones are necessary.

The demo can be opened by clicking the thumbnail below.

Once the image is loaded and the page says "Click to start" or "Tap to start," you can start reproducing the sound by clicking (tapping) on the image.

If the instruction saying "Tap on the image" appears, tap on the screen and the browser starts loading data. Then the page will say "Tap to start" and you can start reproducing panoramic sound by tapping the image.

During the reproduction, panning of sound changes according to the motion of the mouse pointer.

If you are using the mobile device, pan and tilt can be controlled by the rotation of the device instead of the motion of the mouse.


Forest.sopa

Camptown.sopa

Battle.sopa

Cycle.sopa

Sample demonstration (Click on the thumbnail above to open the demonstration)

[Recommended browsers]

In this demonstration, spatial audio information is streamed via a SOPA format that is a custom data format proposed by us.

More demonstrations

An array of more than one miniature head simulator has been proposed -> Panoramic sound morphing.

Thanks to Three.js (http://threejs.org/) for JavaScript 3D library.

You can make it

If you are interested in creating panoramic sounds on your own, see Create and share panoramic sound. To make your Web site full of immersive panoramic sounds, consider using the JavaScript library "SopaJS."

Detect and locate ultrasound

The miniature head simulator system can be used even for inaudible ultrasound [2].

By making the microphone distance very short, the upper limit of the frequency range expands into the ultrasonic frequency. For more information, see Panoramic ultrasound.