Audio & Video Fingerpriting - Web 2.0 Ad-Hoc Communities with Dynamic Ad Insertion Omni-Media

So I was once again I was pondering the future of online video and podcasting and its extensions for the future. I came upon the “Best Paper”, (Which Google won), for its Interactive TV work presented at the Euro Interactive Television Conference held in Greece recently. The concept is capturing ambient audio from Television and using it for Mass Personalization.

OK you say… tell me more….

The paper describes mass personalization; a framework for combining mass media with a highly personalized Web-based experience. They introduce four applications for mass personalization: personalized content layers, ad hoc social communities, real-time popularity ratings and virtual media library services.

They use the ambient audio originating from the television to make the four applications available with no more effort than simple television channel surfing. Their audio identification system does not use dedicated interactive TV hardware and does not compromise the user’s privacy, they claim. Feasibility tests of the proposed applications were conducted with good results.

But how does this play into online video and podcasting?

If ambient audio can be used in loud uncontrolled environments like living rooms, why can’t it be used on computers for those who are viewing videos. As long as there is a way to catalogue or upload your shows into a mass-media audio database server, I see no reason.

Now of course, we haven’t brought up the subject of Big Brother listening in. This is one of the major social hurdles of this technology, how do you convince the common public that there conversations are not being recorded?

Of course Google answers this in their paper; they suggest that “the viewer’s acoustic privacy is maintained by the irreversibility of the mapping from audio to summary statistics”. The Audio fingerprinting software that they use in their research is a version of Ke, Y., Hoiem, D., & Sukthankar (2005) “Computer Vision for Music Identification; In proceedings of computer vision and pattern recognition”

Here is their description of the audio fingerprinting technique;

“Audio is converted into a spectrogram. The spectrogram frames are processed by 32 contrast filters and thresholded to produce a 32-bit descriptor. Contrast filters subtract neighboring rectangular spectrogram regions and can be calculated using the integral-image technique.”

Once they fingerprint the technology they suggest organizing ad-hoc social communities based on individuals who are watching the programs at the exact same moment in time, then serving targeted dynamic advertising based on preferences and geographic location. This concept is Sweet!

So once again, what if these tools can be applied to any video or any audio for that matter, which is data-based and fingerprinted. Individual producers and creators could then offer Google platforms for targeted niche distribution, i.e. bringing the “long tail” mentality to large scale monetization through the currently mighty Google.

Well I am not sure if this will actually happen, or if Google’s control over those publishers who choose to use the services will inhibit the popularity. But if Google can stay away from the Big Brother scare, accurately open an audio fingerprint database to public producers, and effectively integrate it into the Ad programs, it might become another viable source of targeted niche monetization, with practical user benefits for Web 2.0

Here is the link to the paper:

Google Audio Fingerprinting Paper

Tags : Audio Fingerprinting; Interactive Media Mass Personalization, interactive advertising; dynamic advertising, interactive podcasting, Audio fingerprint