Watching the Watchers

A privacy analysis of automatic content recognition on U.S. smart TVs

82% of U.S. households have a smart TV that can take snapshots of whatever is on screen and report them back. We measured where that data goes, and whether turning the feature off actually stops it.

See what we found Read the paper
8
smart-TV operating systems, about 95% of the U.S. market
12
commercial TVs tested
10
viewing scenarios per TV
2
privacy settings: ACR opted in and opted out
5,943
automated experiments
How ACR works

Your TV recognizes what you watch

Automatic content recognition (ACR) is built into most smart TVs. It identifies what's playing, whether it comes from an app, an antenna, a game console or a cable box.

  1. Capture

    The TV takes short audio or video snapshots of whatever is on screen.

  2. Send

    Those snapshots, or fingerprints made from them, are sent over the internet to the ACR provider.

  3. Match

    The provider compares them with a library of known content to work out exactly what you're watching.

  4. Monetize

    Viewing habits can be used for targeted ads and shared with or sold to other companies.

Research questions

What we set out to learn

Little is publicly known about how ACR behaves in practice. Deployments are opaque, and smart TVs are hard to test automatically. We asked three questions.

Method

An automated living room

We built a testbed that plays the same content on every TV without anyone at the remote, records all of its network traffic, and then works out which traffic belongs to ACR.

  1. Collect the data

    Run thousands of controlled viewing experiments on 12 TVs and capture every packet they send and receive.

  2. Identify ACR endpoints

    Use the controlled content to pick out the servers that respond to what's on screen, then compare their traffic across conditions.

Diagram of the testbed: a control machine feeds content to each TV through an antenna modulator, an HDMI converter and screen mirroring, while all traffic is captured
Add the testbed diagram (Figure 2) at images/testbed.png
The testbed. [Edit caption.]

The testbed

Twelve smart TVs from ten manufacturers cover the eight operating systems that run about 95% of U.S. smart TVs, and more than 80% of the TV brands with over 3% of U.S. households. The operating system is what we compare, so we chose TVs to cover every major OS first, adding a second TV where one OS is sold by several brands.

All traffic to and from each TV was captured with the Mon(IoT)r software and the SPHERE research infrastructure. In total we ran 5,943 controlled experiments.

Getting content onto the screen

A control machine sends the same content to every TV through three routes, so each viewing scenario is identical across devices.

Antenna

HDMI-to-RF modulators broadcast our own custom channel to each TV's antenna input.

HDMI

IP-to-HDMI converters turn video streamed from the control machine into an HDMI signal, like a cable box or console.

Screen mirroring

Google Pixel phones and a Windows laptop running Pigeoncast and Miracast cast to each TV using its own mirroring protocol.

Ten viewing scenarios

The scenarios cover the common ways people use a smart TV. Idle, with the TV on the home screen and nobody using it, is the control for spotting baseline traffic.

ScenarioSub-scenarioWhat plays
Idle–The home screen, with no interaction
AntennaChannelOur custom broadcast channel with moving picture and sound
Black screen + audioA constant black frame with sound, over the antenna signal
Dynamic + no audioMoving picture with no sound, over the antenna signal
First-party streaming–The free streaming service built into the TV's operating system
Third-party streamingNetflixA subscription-only service
Tubi TVA free, ad-supported service
YouTubeA hybrid of free and paid content
HDMI–Moving picture with sound from an external device
Screen mirroring–A phone or laptop mirrored over the TV's own protocol, playing video with sound

Separating sound from picture

Some ACR systems recognize content by its audio, others by its video. By switching each one off in turn on our custom antenna channel, we can tell which a TV relies on. The antenna is a natural ACR target, because what you watch over the air can't be tracked by any app.

Antenna scenarioMoving pictureSoundWhat it tells us
ChannelYesYesBaseline: ACR can use either signal
Black screen + audioNoYesTraffic here points to audio fingerprinting
Dynamic + no audioYesNoTraffic here points to video fingerprinting

Running it without a remote

We wrote control scripts using each OS's Python remote-control library for six of the operating systems. Xumo OS and VIDAA OS had no reliable library, so we drove them through their Android remote apps using ADB.

The hard part was that TV menus change, so each OS needed its own tested sequence of commands. An IP camera on every TV recorded each run, and we reviewed the frames to confirm it worked. Runs that failed because of internet outages or unexpected TV behavior were repeated.

How we answer each question

  • Under what viewing conditions does ACR send traffic?We identify each TV's ACR endpoints, then compare the traffic they receive in each of the ten scenarios.
  • Are ACR opt-outs offered, and do they work?Every scenario runs 30 times with ACR on and 30 times after opting out in the TV's settings, in a random order so one test can't bleed into the next. We then compare ACR traffic between the two states.
  • What viewing information is exposed?Where ACR traffic is unencrypted, we read it directly. Where it's encrypted, we infer what it carries from its timing, volume, destinations and protocol details. We also check whether content-delivery services leak viewing data outside ACR.

The TVs we tested

Twelve TVs from ten manufacturers, covering operating systems with about 95% of the U.S. smart-TV market. Where one operating system is sold under several brands, we tested more than one TV.

Operating systemU.S. market shareTVs tested
Tizen OSSamsung
Roku OSRoku, TCL
webOSLG
SmartCast OSVizio
Google TV OSSony (2), Hisense, TCL
Fire TV OSAmazon Fire
Others, including VIDAA and XumoSupersonic, Westinghouse
Total12 TVs

Results reflect these devices and firmware versions at the time of testing and may not carry over to other models or regions.

Finding: when ACR runs

Every TV watched the antenna. None watched Netflix.

We counted a scenario as tracked when the TV sent regular traffic to its ACR endpoint in more than 80% of the 30 opted-in runs. The results were clear-cut: scenarios were tracked in nearly every run or almost never.

  • Antenna broadcasts were tracked on every operating system we could analyze. Over-the-air TV can't be tracked by any app, which makes it a natural target for ACR.
  • No TV tracked Netflix or YouTube. For Netflix, this matches publicly reported agreements.
  • Tubi, a free ad-supported service, was tracked on Tizen OS (in 29 of 30 runs), Google TV on Sony, and VIDAA OS.
  • Most operating systems also tracked their own built-in streaming service. Fire TV OS and Google TV on Sony were the exceptions.
  • HDMI was tracked on five of the seven operating systems, but not on Roku OS or SmartCast OS. Screen mirroring was tracked only on VIDAA OS.
  • Vizio's SmartCast OS sent ACR-pattern traffic even when the TV sat idle on its home screen, in every run.

Runs with content-correlated ACR traffic, ACR turned on

0%100% of runs
Select a square to see the scenario and what we observed.

Each square shows the share of 30 runs in which the ACR endpoint received content-correlated traffic; 80% or more counts as tracked. Only operating systems where we identified and validated an ACR endpoint are shown, and Roku OS has a row for each of its two TVs.

Finding: opting out

Turning ACR off doesn't always turn it off

We ran every scenario 30 more times after opting out in the TV's settings. Small amounts of background traffic (under 5% of the opted-in volume) were set aside; anything more was flagged as traffic consistent with ACR.

4
operating systems where opting out fully worked: webOS, Google TV on Sony, VIDAA OS and Fire TV OS
30 of 30
opted-out runs of Samsung's built-in streaming service still showed traffic consistent with ACR
100%
of opted-out runs on Vizio still sent traffic to its Amazon Kinesis ACR endpoint, with no decrease
  • Samsung (Tizen OS): after opting out, traffic consistent with ACR continued in 41 of 60 antenna runs, 18 of 30 HDMI runs and 6 of 30 Tubi runs. It also appeared in a few Idle, Netflix and YouTube runs, even though those aren't tracked when ACR is on.
  • Roku OS: the opt-out doesn't apply to Roku's own streaming service. TCL tells users this in its settings menu; Roku TV doesn't. For antenna viewing the opt-out worked on Roku TV, while on TCL traffic continued in 9 of 90 runs.
  • Vizio (SmartCast OS): the opt-out setting had no observable effect on traffic to the identified ACR endpoint.

Runs with traffic consistent with ACR, after opting out

0%100% of runs Opt-out doesn't apply
Select a square to see the scenario and what we observed.

Most ACR traffic is encrypted, so these results come from how the traffic behaves, not from reading it. We can't confirm what the data contains, or whether it is processed or kept on the companies' servers. VIDAA OS is the exception: its ACR traffic is unencrypted, so we read it directly.

Finding: what's exposed

Viewing data left in the open

Even where we can't read ACR traffic, its behavior shows how a TV recognizes content. And on two platforms, what you watch travels unencrypted.

Listening or looking?

By removing the picture or the sound from our antenna channel, we can tell which signal each TV's ACR relies on.

Two charts of upload traffic to the ACR endpoint. On LG webOS, traffic drops sharply when video plays without sound. On Sony Google TV, traffic drops when a black screen plays with sound.
Upload traffic to the ACR endpoint for each antenna sub-scenario. Lines further left mean less traffic. LG's webOS goes quiet without sound (audio-based ACR); Sony's Google TV goes quiet on a black screen (video-based ACR).
Operating systemRecognizes content byWhat we saw
Tizen OS (Samsung)VideoACR traffic disappeared on a black screen; removing sound changed nothing.
Google TV OS (Sony, Samba TV ACR)VideoTraffic fell by 52.9% on a black screen with sound.
webOS (LG, Alphonso ACR)AudioTraffic fell by 97.3% when video played without sound.
VIDAA OS (Supersonic)AudioThe unencrypted messages contain an audio fingerprint.
Fire TV OS, Roku OS, SmartCast OSCan't tellTraffic looked the same with and without picture or sound.

VIDAA OS sends ACR data unencrypted

VIDAA OS sends ACR data to servers run by Unruly over MQTT without encryption, so anyone on the network path between the TV and those servers can read it. Each message carries a Client ID, a permanent identifier for the TV that can't be reset, along with the time zone, the app name, the input (antenna, HDMI), the antenna channel and an audio fingerprint.

The app name alone can be revealing. We saw ACR on apps such as VIDAA Kids, which raises concerns under the U.S. children's privacy law (COPPA), and Christianity and Money. VIDAA's own HOME OS app sent show and film titles in place of the app name.

The servers reply with a genre for what they recognized, including categories such as Religious, Health and Medicine, and Political. When we played The Story of Moses over HDMI, it came back labeled Religious.

What an eavesdropper can read
Client ID
Permanent device ID
Time zone
Your region
App name
VIDAA Kids
Input source
HDMI
Tuner channel
For antenna viewing
Audio fingerprint
What's playing
Genre returned
Religious
Illustration of the fields we observed, not a real captured message.
57,000,000+
ACR detection results from other people's TVs, readable by anyone through the ACR server used by VIDAA OS, which required no login
1,000,000+
unique TVs, each tagged with its device ID
82
content categories, including Politics, Health and Medicine, and Religious
~3 hours
the ethically limited sample we collected

Partly fixed. VIDAA confirmed the problem after we reported it. The server now requires a login, but messages are still sent unencrypted.

We could also send a previously seen message back to the server, and our own test TV received it. That means an attacker could push fake or harmful messages to TVs. For ethical reasons, we didn't try.

A viewing history in plain sight, even with ACR off

Separately from ACR, Vizio's free streaming service WatchFree+ puts the name of each show or channel into unencrypted web address lookups (DNS queries and SNI) to Amagi, the company that delivers the streams.

Anyone on the network path, such as an internet provider or whoever runs the network, can rebuild an ordered list of what was watched without any ACR data. None of the operating systems we audited mention Amagi in their privacy policy or settings, and there is no way to turn this off.

A lookup as it travels the network
soul-5mincrafteng-vizio.amagi.tv
Program title
hyphenated but readable
Platform
the TV maker
An example of the address format from the paper.
What it means

The off switch needs to mean off

  • ACR watched antenna broadcasts on every TV we could analyze, and most TVs also watched their own built-in streaming services.
  • Opting out fully worked on only four operating systems. On Samsung and Vizio, traffic consistent with ACR kept flowing, and Roku's opt-out doesn't cover its own streaming service.
  • VIDAA OS sent viewing data, a permanent device ID and sensitive categories without encryption, and its server exposed more than 57 million detection results.
  • Even users who fully opt out aren't safe: Vizio's WatchFree+ reveals program titles to anyone watching the network.

An opt-out should do what it says, and what you watch should never travel unencrypted.

If you own a smart TV

  • Disconnect the TV from the internet and watch through an external streaming device that doesn't run ACR.
  • Or turn ACR off on systems where opting out worked in our tests: webOS (LG), Fire TV OS, Google TV on Sony, and VIDAA OS.
  • On other systems, use our blocklist. In our lab, blocking these endpoints didn't break the TV or affect viewing quality in the scenarios we tested.

Neither step stops the program-title leak in Vizio's WatchFree+.

For TV makers

  • List every input that ACR watches, such as antenna, HDMI, apps and screen mirroring, in the privacy policy.
  • Authenticate and encrypt all ACR traffic.
  • Close the Amagi leak: use encrypted DNS, and stop putting what people watch into web addresses.

For regulators

  • Audit ACR disclosures, and test whether opt-outs actually work.
  • Our automated method can collect that evidence at scale.
  • Request what researchers can't see, such as the data stored on ACR servers.