The Joshua Bell VR Experience Out Today on PS VR

Today, we are releasing an exciting new musical experience for PlayStation VR. In partnership with award-winning violinist and Sony artist Joshua Bell, we’ve brought to life a live studio performance of Brahms’ Hungarian Dance No. 1 in full 360-degree VR.

With adaptive audio and positional tracking, the Joshua Bell VR Experience will transport you into the session and allow you to immerse yourself completely in the performance from every direction. The Experience is available now at PlayStation Store as a free download for PlayStation VR.

Joshua Bell is a celebrated, Grammy award-winning violinist with a career spanning more than 30 years as a soloist, chamber musician, recording artist, and conductor. Generally regarded as one of the finest classical musicians of this generation, he just completed a critically-acclaimed residency at the Kennedy Center for the Performance Arts and National Symphony Orchestra in Washington D.C. He is also a technology enthusiast.

When he tried an early PlayStation VR prototype a few years back, he was overwhelmed by its potential. He wondered how it could be used for classical music. At the same time, our group here in the UK happened to be researching new techniques for filming live events, particularly with music, and it seemed like the perfect opportunity to work together and collaborate with some of our other Sony companies.

We filmed at Air Studios’ Lyndhurst hall in London. Not only is it one of the best rooms in the world for recording orchestral instruments, but it also provides a stunning background for the performance. Lyndhurst Hall has been used to record the soundtracks to many famous movies and it is often used for our PlayStation games, too.

Using a number of Sony FDR-X1000V cameras, we filmed a high quality 360 degree video of Joshua’s brilliant performance with his accompanist Sam Haywood. Now with most 360 videos, you can only watch from a fixed point in space, but PlayStation VR provides positional tracking, allowing the player to not just look around, but move around within the virtual world. Even the smallest amount of movement helps to create the sense of presence, convincing your brain that you are actually there. After some further development using techniques normally only found in Hollywood’s special effects, we were able to render out the entire scene in 3D. We maintained the correct shape, size and scale of the scene, including all the subtle surface details, so you could enjoy the performance as if you were there.

We also added positional audio tracking. The PS VR’s binaural audio processing always provides amazing directional surround sound, but the audio in this experience now responds to proximity; move close to Joshua to hear all the detail of his violin; move back, and the natural reverb of the hall takes over.

Providing free movement around a live action VR scene is cutting edge and normally involves cumbersome camera and playback equipment. The new approach we used for the Joshua Bell VR Experience allowed us to capture the informality of a real recording session without the constraints of a special studio. It also played to the strengths of the PS4s processing capabilities. To learn more about the process, check out this behind the scenes video we captured while in development.

We’re all really happy with the results and felt that the full VR experience brought to life elements of the performance that could not be conveyed using the music on its own.

Creating this experience was an enormous technical challenge and our next steps are to make everything easier to do so that we can capture many more performances. All types of music, theatre or even sport, could benefit from this technology.