Automatic content recognition (ACR) is a technology used to identify content played on a media device or presented within a media file. Devices with ACR can allow for the collection of content consumption information automatically at the screen or speaker level itself, without any user-based input or search efforts. This information may be collected for purposes such as personalized advertising, content recommendations, or sale to companies that aggregate customer data.

How it works

To start the process, a short media clip (audio, video, or both) is selected from within a media file or captured as displayed on a device such as a

smart TV A smart TV, also known as a connected TV (CTV or, rarely, CoTV), is a traditional television set with integrated Internet and interactive Web 2.0 features that allow users to stream music and videos, browse the internet, and view photos. Smart T ...

. Using techniques such as fingerprinting and watermarking, the selected content is compared by the ACR software with a database of known recorded works. If the fingerprint of the media clip finds a match, the ACR software returns the corresponding

metadata Metadata (or metainformation) is "data that provides information about other data", but not the content of the data itself, such as the text of a message or the image itself. There are many distinct types of metadata, including: * Descriptive ...

regarding the media as well as other associated or recommended content back to the client application for display to the user, or for collection by the device manufacturer or a company that collects user data.

Fingerprints and watermarking

Two leading methodologies for audio-based ACR are

acoustic fingerprint An acoustic fingerprint is a condensed digital summary, a digital fingerprint, deterministically generated from an audio signal, that can be used to identify an audio sample or quickly locate similar items in a music database. Practical u ...

ing and watermarking. Similarly, video fingerprinting is used to facilitate ACR for visual media. Acoustic fingerprinting generates unique fingerprints from the audio content itself. Fingerprinting techniques are agnostic to content format,

codec A codec is a computer hardware or software component that encodes or decodes a data stream or signal. ''Codec'' is a portmanteau of coder/decoder. In electronic communications, an endec is a device that acts as both an encoder and a decoder o ...

bit rate In telecommunications and computing, bit rate (bitrate or as a variable ''R'') is the number of bits that are conveyed or processed per unit of time. The bit rate is expressed in the unit bit per second (symbol: bit/s), often in conjunction ...

and compression techniques. This makes employment of acoustic fingerprinting possible across various networks and channels and is widely used for interactive TV, second screen application, and content monitoring sectors. Popular apps like Shazam,

YouTube YouTube is an American social media and online video sharing platform owned by Google. YouTube was founded on February 14, 2005, by Steve Chen, Chad Hurley, and Jawed Karim who were three former employees of PayPal. Headquartered in ...

Facebook Facebook is a social media and social networking service owned by the American technology conglomerate Meta Platforms, Meta. Created in 2004 by Mark Zuckerberg with four other Harvard College students and roommates, Eduardo Saverin, Andre ...

, TheTake,

WeChat WeChat or Weixin in Chinese ( zh, c=微信, p=Wēixìn , l=micro-message) is an instant messaging, social media, and mobile payment mobile app, app developed by Tencent. First released in 2011, it became the world's largest standalone mobile a ...

and Weibo reportedly use audio fingerprinting methodology to recognize content played from a TV to trigger additional features like votes, lotteries, topics or purchases. In contrast to fingerprinting,

digital watermarking A digital watermark is a kind of marker covertly embedded in a noise-tolerant signal such as audio, video or image data.H.T. Sencar, M. Ramkumar and A.N. Akansu: ''Data Hiding Fundamentals and Applications: Content Security in Digital Multimedia'' ...

require the inclusion of digital "tags" embedded within the digital content stream prior to distribution. For example, a broadcast encoder might insert a

watermark A watermark is an identifying image or pattern in paper that appears as various shades of lightness/darkness when viewed by transmitted light (or when viewed by reflected light, atop a dark background), caused by thickness or density variations i ...

every few seconds that could be used to identify the broadcast channel, program ID, and time stamp. This watermark is normally inaudible or invisible to the users, but is detectable by display devices like phones or tablets which can read the watermarks to identify the content it is playing. Watermarking technology is also utilized in the media protection field to help identify where illegal copies originate.

History

In 2011, ACR technology was applied to TV content by the Shazam service, which captured the attention of the television industry. Shazam was previously a music recognition service which recognized music from sound recordings. By utilizing its own fingerprint technology to identify live channels and videos, Shazam extended their business to television programming. Also in 2011, Samba TV (at the time known as Flingo) introduced its patented video ACR technology, which uses video fingerprinting to identify on-screen content and power cross-screen interactive TV apps on Smart TVs. In 2012, satellite communications provider

DIRECTV DirecTV, LLC is an American Multichannel television in the United States, multichannel video programming distributor based in El Segundo, California. Originally launched on June 17, 1994, its primary service is a digital Satellite television, s ...

partnered with TV loyalty vendor Viggle to provide an interactive viewing experience on the second screen. In 2013, LG partnered with Cognitive Networks (later purchased by Vizio and renamed Inscape), an ACR vendor, to provide ACR driven interaction. In 2015, ACR technology spread to even more applications and smart TVs. Social applications and TV manufacturers like

Twitter Twitter, officially known as X since 2023, is an American microblogging and social networking service. It is one of the world's largest social media platforms and one of the most-visited websites. Users can share short text messages, image ...

Google Google LLC (, ) is an American multinational corporation and technology company focusing on online advertising, search engine technology, cloud computing, computer software, quantum computing, e-commerce, consumer electronics, and artificial ...

, Weibo, LG,

Samsung Samsung Group (; stylised as SΛMSUNG) is a South Korean Multinational corporation, multinational manufacturing Conglomerate (company), conglomerate headquartered in the Samsung Town office complex in Seoul. The group consists of numerous a ...

, and Vizio TV have used ACR technology either developed by themselves or integrated by third-party ACR providers. In 2016, additional applications and mobile OS embedded with automatic content recognition services were available including Peach, Omusic and Mi OS.

Applications

* Advertising and customer data collection: Data collected on the media consumption habits of customers can be very valuable to device manufacturers, advertisers, and data aggregation companies. ACR technology helps these companies survey the interests of customers and collect data so that they can be more precisely targeted with

personalized marketing Personalized marketing, also known as one-to-one marketing or individual marketing, is a marketing strategy by which companies use data analysis and digital technology to show adverts to individuals based on their perceived characteristics and ...

and advertising campaigns. It was reported in Nov 2021 that smart television manufacturer Vizio profited more from the sale of their customers' data than from the televisions they sold. *Audience measurement: Real-time

audience measurement Audience measurement calculates how many people are in an audience, usually in relation to radio listenership and television viewership, but also in relation to newspaper and magazine readership and, increasingly, web traffic. The term is someti ...

metrics are now achievable by applying ACR technology into smart TVs, set top boxes and mobile devices such as smart phones and tablets. This measurement data is essential to quantify audience consumption to set advertising pricing policies. *Content identification: ACR technology helps audiences retrieve information about the content they watched or listened to. The identified video and music content can be linked to internet content providers for on-demand viewing, third parties for additional background information, or complementary media. *Content enhancement: Because devices can be "aware" of content being watched or listened to, second screen devices can feed users complementary content beyond what is presented on the primary viewing screen. ACR technology can not only identify the content, but also it can identify the precise location within the content and present additional information to users. ACR can also enable a variety of interactive features such as polls, coupons, lottery or purchase of goods based on timestamp.

Privacy concerns

Organizations ranging from consumer rights advocates

Electronic Frontier Foundation The Electronic Frontier Foundation (EFF) is an American international non-profit digital rights group based in San Francisco, California. It was founded in 1990 to promote Internet civil liberties. It provides funds for legal defense in court, ...

to tech web sites such as

PCMag ''PC Magazine'' (shortened as ''PCMag'') is an American computer magazine published by Ziff Davis. A print edition was published from 1982 to January 2009. Publication of online editions started in late 1994 and continues . Overview ''PC Magaz ...

have expressed serious objections to the collection of user viewing consumption habits by their devices on privacy grounds.

References