How the tragic death of Do Not Track ruined the web for everyone
A decade ago–long before the current controversies over what big companies are doing with our data–a lot of people were already irate about ad networks that followed their activity across sites in order ever more precisely to target marketing messages. A feature called Do Not Track arose as a simple, comprehensible way for browser users to take back their privacy. To opt out of being tracked, you’d check a box in your browser’s settings.
Notably, this didn’t opt out of advertising–just the technology used to target ads. With Do Not Track checked, no web server or embedded code would associate your behavior at a given site with actions elsewhere on the web. It was a great idea.
And now it’s dead.
Oh, for all practical purposes, DNT died years ago. But Apple’s removal of the Do Not Track preference from Safari for Macs and iOS in an update in early February officially signaled the end of what might have been a workable understanding between consumers and the advertisers that rely on ad-tech networks to target them.
Apple’s move follows the dissolution of a World Wide Web Consortium (W3C) project, the Tracking Protection Working Group, which shut down after eight years on January 17. The release notes for Safari 12.1 called Do Not Track an “expired” standard, which is sadly accurate.
In October 2018, on its public mailing list, the W3C group discussed how to describe Do Not Track’s failure in a preface to its final piece of work. After some back and forth, the group agreed on the language that appears:
…there has not been sufficient deployment of these extensions (as defined) to justify further advancement, nor have there been indications of planned support among user agents, third parties, and the ecosystem at large.
It’s an artful self-own by the group’s participants, which included representatives from ad industry trade groups, large advertisers, and ad delivery platforms, as well as ones from privacy groups, governments, and browser makers. After a flurry of work from 2011 to 2013, the group hadn’t met face to face since 2013, according to its notes.
The working group’s existence used to imply that the ad industry was actively moving towards a consensus on self-regulation when it came to online privacy. But DNT turned out to be a useful fig leaf, not a solution. “The best way to sabotage a process is by wholeheartedly participating in it,” says Alan Toner, a privacy and data protection special adviser at the Electronic Frontier Foundation (EFF), who represents his organization at the W3C.
In the ultimate irony, Apple told me via a spokesperson that it removed Do Not Track after the W3C group shuttered because, if enabled, it could help ad networks “fingerprint” a browser, a technique used by tracking systems to defeat ad blockers by identifying unique characteristics in a user’s browser configuration.
It could have all been so different.
One-click privacy
Do Not Track bubbled up from the seeming success of the Federal Trade Commission’s National Do Not Call registry, which went into effect in 2003. It allowed consumers to register their phone numbers as being unwelcoming to commercial solicitations. Companies making calls to people other than customers have to purge these numbers from calling databases. (Do Not Call was ultimately a failure, because it only prevented scrupulous parties from calling, not those who blithely ignored the law or were engaged in outright scams.)
Initially, Do Not Track was going to be a similar kind of central registry. But in 2009, privacy advocates Chris Soghoian and Sid Stamm implemented the idea as a simple Firefox plug-in. The plug-in would add a Do Not Track header to the metadata a browser sends to a server on initiating a connection. If a user had enabled Do Not Track, the value of the header would be “1”; otherwise, “0.” It was that simple. It didn’t matter from a technical perspective that no server knew how to interpret that header at the time and therefore ignored it; the policy details could be worked out later.
This straightforward idea caught fire, and within a couple of years, all the major browser had added an option to express a preference. Stamm, now an associate professor at the Rose-Hulman Institute of Technology, says the header was “a way to shout, ‘Hey, I don’t like this!’” He developed the plug-in with Soghoian because “people were really unaware how much data was collected about them.”
Stamm and Soghoian, who is now a privacy and cybersecurity adviser to U.S. Senator Ron Wyden (D-OR), were part of a group of privacy advocates and security engineers who advocated for DNT. By 2011, the FTC appeared poised to recommend that DNT evolve from a nascent browser feature into a regulatory requirement. The W3C opened a working group to study how to turn DNT into a fully recognized standard that would define how it could be implemented.
Arvind Narayanan, now an associate professor at Princeton and part of that early DNT-formulating group, said via an email statement that the prospect of federal legislation brought ad players to the table. But when that legislation didn’t materialize, “the prolonged negotiations in fact proved useful to the industry to create the illusion of a voluntary self-regulatory process, seemingly preempting the need for regulation.”
The moment passed. Those involved in the ad industry, whether social networks or ad-tech firms, had little interest in pursuing DNT if they could avoid it. Publishers didn’t demand the technology as a way to protect visitors to their sites; advertisers didn’t act as though it affected them directly.
One of the wrenches in the works was the issue of whether Do Not Track was really a binary deviation. As a two-position switch, it was either off or on. But if a user hadn’t considered the matter–or didn’t even know DNT existed–a third state existed: not yet decided. If a user hadn’t chosen to turn on DNT, browsers either left it turned off–or didn’t send DNT info one way or the other to websites.
Microsoft broke the model. In 2012, the company opted to preset Internet Explorer’s Do Not Track to the “on” state without requiring a user to pick or confirm that choice. Though the move defaulted to the most privacy-friendly option, it also put a crimp in Google’s ad hegemony, which Microsoft would not have seen as a bad thing.
Companies that were part of the ad economy already had reason to be wary of DNT; a DNT that stopped users from being tracked without them explicitly opting in looked like an existential threat. “Do Not Track started with one leg cut off the moment Microsoft used it as a marketing tool, by turning it on by default,” says Dan Jaye, a veteran online-ad veteran, most recently the founder of aqfer.
Sam Tingleff, the chief technology officer of the Interactive Advertising Bureau’s Tech Lab, provides another reason why the ad-business players took issue with DNT: From their perspective, it was too simple. A user could only turn it on or off for the browser as a whole, without per-site whitelist or blacklist options, something that the W3C group was working to elaborate on.
The lack of legislative or regulatory action, Microsoft’s DNT misstep (which the company reversed–too late–in 2015), and the W3C’s stalled movement forward left the DNT checkbox in place but without any power. Narayanan says that it was clear to him Do Not Track had failed by 2013. The corpse only stopped kicking recently. Do Not Track died before consumers had a chance to gain a taste for being tracked.
The ad-blocking arms race
In the absence of consumers’ ability to express a preference and without U.S. regulation governing tracking, what did the ad industry expect would happen? It’s not clear.
Even as DNT made its way into browsers, tracking and advertising bloat became ridiculous. Some mainstream content sites such as those affiliated with newspapers might have 70 to 100 individual pieces of remotely loaded JavaScript or tracking images, inflating a relatively simple page of text into multiple megabytes while also bringing browsers to a crawl as they executed all that code.
Consumers may not read the minutes of W3C task group meetings or study the details of underlying web page code, but they do notice things like purchasing a leaf blower or reading recommendations about it, and then having leaf blower ads follow them around the Internet for weeks. They can also tell if a website takes forever to load. Without knowing the particulars, people are generally aware they’re tracked without explicit consent and that it’s ruining their Internet experience.
That has led inexorably to the rise of ad blockers. Ad blockers prevent or deter some of this tracking code from loading, reducing cruft and speeding browsers while–not incidentally–reducing the ability to track a user uniquely across sites.
In its 2017 report, covering the previous year, PageFair found that 11% of internet users worldwide employ an ad blocker, and that usage grew 30% year over year. (One of the most popular, AdBlock Plus, is controversial, as it allows large advertisers to pay to bypass its filter with ads that meet certain criteria.)
While the ad business suffers revenue loss from ad blockers, some people deeply embedded in the industry don’t blame the user. Aqfer’s Jaye says, “Most of the adoption of this is about poor user experience,” and says the ad industry has created this problem for itself. “We think that most users choosing ad blockers are doing so to improve the user experience–a goal which we fully support,” echoes IAB Tech Lab’s Tingleff.
With blockers on the rise, some ad networks took took to examining all of a browser’s characteristics and the results of executing JavaScript silently on a user’s machine to create a composite fingerprint that could with varying degrees of confidence identify a browser–and therefore a user–uniquely. You can get a sense at this at the research project Am I Unique?, which looks at your browser and tells you how vulnerable it is to this sort of tracking.
Jaye says he was forced to install AdBlock Plus in 2018–not to disable ads or tracking, but to prevent crashes in his Chrome browser. He says he determined that a large number of websites he visited relied on an obscure JavaScript function that reported the state of his Lenovo laptop’s gyroscope–information that would be helpful for fingerprinting, but not for site functionality. Some bug in how Lenovo allowed browser access to this data in his model of laptop caused the crashes.
It’s in this environment that Apple identified Do Not Track as a potential fingerprinting problem. Only a subset of Safari users enabled Do Not Track, which meant that those users can be differentiated from the vast majority who did not. As Apple said via a statement, “Fingerprinting attempts to identify a device uniquely based on its combination of detectable settings. Because Do Not Track was an opt-in setting that was visible to websites, it could potentially be used as a variable for fingerprinting.”
As Narayanan noted in a series of tweets, this information increases a tracking network’s ability to ID a browser uniquely only by a small amount. But it was enough that Apple disabled DNT.
Apple has increasingly built anti-tracking methods into successive releases of iOS and MacOS under the rubric Intelligent Tracking Protection (ITP), countering the attempts by ad networks to follow users’ web travels. Mozilla has followed with its Firefox browser and Enhanced Tracking Protection. Google’s Chrome–well, Google has pursued a different direction that aligns with its ad-driven business model.
Everybody loses
The losers in this arms’ race are publishers and their potential audience, as the more people turn to the last resort of ad blocking, the less revenue flows through to sites that rely upon it. As sites shed staff or shut down, readers and viewers suffer from a lack of variety–and a drop in investigatory journalism. Some publishers block readers who have ad blockers installed, showing messages that range from polite to accusatory. (Some ad blockers use anti-anti-ad-blocking tech to fool anti-ad blockers, and so on.)
Targeted ads represent just a few percentage points of all ad spending. Jaye says they’re far more potent than nontargeted ones, delivering perhaps a tenfold difference in revenue. He makes the case that the failure to provide users with tracking control decreases the diversity of ad-supported content on the Internet, because of the revenue hegemony of a few sites–Google, Facebook, and Amazon, primarily–which are far less dependent on third-party ad networks and cross-site interactions to deliver targeted ads.
Yet it’s understandable that a growing number of web users are employing technical means to prevent tracking that appears to have little benefit to the site they visit. “You should be able to do targeting without doing tracking,” says the EFF’s Toner. The industry has poured so much effort into building targeted, tracking-based systems that it hasn’t spent much time examining other approaches, he adds.
Jaye promotes the notion of delivering choice by pushing data ownership back into users’ hands, and relying on advanced mathematical techniques that use cryptographic proofs to show an ad was delivered to the right demographic target without needing to tag and track the person in question across sites. More generally, he says that it would be healthier for marketing to involve fewer technological middlemen, asking, “How do we move the dialog back to the consumer and advertiser directly?”
Stamm, the co-creator of the original DNT plugin for Firefox, says that the current situation is a shame, because the failure of the ad industry to engage over Do Not Track has meant a battle that didn’t have to be fought. “It could have been so much better if people had just had conversations with consumers and listened with consumers about what they want,” he said.
Tingleff of the IAB Tech Lab expresses some hope: “A collaborative approach–browsers and mobile operating systems working with the media industry–to improving user experiences would go a long way to ensuring continued free access to the content and services that make the Internet open and useful.”
The failure of Do Not Track has had a result. It’s led to the imposition of more severe regulation than would be expected, due to the failure of industry players to find a middle ground. The General Data Protection Regulations (GDPR) that went into effect in the European Union in May 2018 wasn’t designed in response to DNT’s failure, but it was built on the premise that too many sites track users without consent and disclosure.
With significant penalties, the GDPR’s disclosure and opt-in provisions put teeth into protection. France’s data-protection authority levied a 50 million euro ($57 million) fine against Google for “lack of transparency, inadequate information, and lack of valid consent regarding ads personalization” in January. Toner says we’re “only at the beginning,” as the GDPR provides many avenues for citizens and advocacy groups to file complaints, many of which are underway against Amazon, Apple, LinkedIn, Facebook, and Google, he notes.
In the wake of the GDPR, California passed a less comprehensive privacy bill affecting one of the world’s largest economies, but the law has some force behind its regulatory bite, too. It takes effect in 2020. At the federal level, U.S. legislators and regulators have considered laws that would preempt California’s privacy and disclosure rules, some weakening them and some veering closer to the GDPR.
Senator Wyden circulated a draft bill in November 2018, the Consumer Data Protection Act, which would require something close to the original Do Not Track registry. It would ban companies from denying services to those who opt out of tracking, and require them to offer fee-based versions to serve them that couldn’t charge more than the lost revenue.
Government will have its say, but so will the market. As ads have delivered less of a return on investment, many publishers have shifted their revenue strategy by erecting paywalls and inducing consumers to pay for access to quality content. That’s been effective for some.
Do Not Track tried to offer a simple way to cut through a thorny situation and avoid a battle, by letting users clearly express their intent so advertisers could honor it. When the noble idea fell apart in the real world, ad networks ceded the field to regulators (to declare what kind of tracking was legal) and ad-blocking technology (which crushes everything it sees). Faced with an increasingly skeptical audience, advertisers and publishers alike have many motivations to find a different path forward that honors privacy.
(39)
