MIT, Microsoft and Adobe researchers recently took a video of a room from 15 feet away through a pane of soundproof glass. Even though they obviously couldn’t hear, they were able to tell exactly what was said inside the room. Their secret? A vibrating bag of potato chips caught on film.
“When sound hits an object, it causes the object to vibrate,” lead paper author Abe Davis said in a release. “The motion of this vibration creates a very subtle visual signal that’s usually invisible to the naked eye. People didn’t realize that this information was there.”
The team created an algorithm that can translate those tiny vibrations back into audio, allowing them to extract intelligible speech from objects. The researchers successfully pulled sound from videos of tinfoil, water in a glass and a plant.
Their work could be useful to determine the composition of different types of objects. As Davis put it, “different objects are going to respond to sound in different ways,” potentially allowing people to use a camera to analyze what a material is made of. The technology could also be used in law enforcement and forensics to recover or record audio.
The best way to capture sound with a video is to use a high-speed camera that capture frames faster than audio signals move. In this case, the researchers used cameras that captured footage at 2,000 to 6,000 frames per second. But they were also able to determine the gender, number and voice characteristics of speakers from cameras as slow as 60 frames per second–within reach of mobile phones like the iPhone 5s and basic point and shoot cameras.
The researchers will present their results at the Siggraph conference this week.
“We’re scientists, and sometimes we watch these movies, like James Bond, and we think, ‘This is Hollywood theatrics. It’s not possible to do that. This is ridiculous.’ And suddenly, there you have it. This is totally out of some Hollywood thriller,” Alexei Efros, associate professor of electrical engineering and computer science at the University of California-Berkeley, said in the release. “I’m sure there will be applications that nobody will expect. I think the hallmark of good science is when you do something just because it’s cool and then somebody turns around and uses it for something you never imagined.”