EFF joined a group of 18 civil society organizations to send a letter encouraging New York Governor Kathy Hochul to
Senate Bill 9934A, the New York Stealth Crawler Prohibition Act
. The letter states:
While framed as a measure to protect local journalism, this legislation harms free expression and establishes a dangerous precedent by effectively deanonymizing and criminalizing automated access to the open web. By requiring all web crawlers to disclose their identity and explicit purpose, and by granting media outlets unchecked authority to obtain judicial subpoenas to unmask unidentified automated web traffic without any showing of misconduct or actual injury, this bill threatens digital privacy, compromises the foundational architecture of the internet, and will ultimately stifle the very independent journalism it seeks to protect.
As we’ve
previously explained
, so-called “stealth crawlers” are simply automated tools to access and collect public web data—without disclosing the user’s identity. Private crawlers like these facilitate all kinds of
important work
that benefits the public, including investigative reporting, academic research, cybersecurity protection, and EFF’s own
Privacy Badger
. As we illustrate in the letter:
Anonymous crawling fuels important investigative journalism. For example,
The Markup
, a non-profit news site, used anonymous crawlers to
investigate
potentially anti-competitive practices by tech companies, such as Amazon’s tendency to prioritize Amazon brands and Amazon-exclusive products over competitors with higher ratings. The crawlers identified themselves as ordinary Firefox browsers to web servers, which
allowed The Markup
to understand how Amazon search results pages would appear to ordinary users. Similarly,
ProPublica
used an automated tool designed to simulate an ordinary Amazon customer to reveal that the site steered shoppers to more expensive products over cheaper alternatives.
Anonymous web scraping is
… [more]
