UK Information Commissioner Issues Warnings to AI Search Engines Over robots.txt Violations
The UK ICO has issued formal enforcement notices to several autonomous AI search crawlers that bypassed webmaster robots.txt directives to harvest proprietary publisher content.
The United Kingdom's Information Commissioner's Office (ICO) has warned generative search providers that systematically ignoring robots.txt exclusion directives constitutes unlawful personal data extraction under UK GDPR.
The investigation was sparked by reports of AI crawlers rotating user-agent strings and using residential proxies to bypass standard rate limits on news publishers and independent blogs.
Webmasters can generate strict crawler access rules using the Robots.txt Generator on Luminus.
Subscribe for Updates
Get official press announcements and version releases sent directly to your email.