Meta web crawlers cause technical issues for small businesses

Small business owners are reporting significant technical disruptions to their websites linked to Meta Platforms' web crawlers. The bots, identified as Meta-ExternalAgent and Meta-WebIndexer, have been associated with high CPU usage and site crashes, impacting the digital operations of various online businesses.

In one documented instance, a website experienced over 100 hits per minute from a Meta bot, totaling approximately 550,000 hits over several days. This volume pushed the site's CPU usage to 100%, resulting in a crash. These technical challenges coincide with broader scrutiny regarding Meta's proposed updates to its commercial terms of service, which critics argue could grant the company expansive authority to scrape content.

Meta has maintained that website managers can control crawler interactions through robots.txt files. A spokesperson stated that the company provides guidance on managing these tools and respects the preferences set by site owners. The company also noted that it uses crawlers for various purposes, including indexing content and advertising products.

The implementation of the updated commercial terms, originally scheduled for Oct. 30, 2026, has been delayed to allow for industry consultation. Meta spokesperson Andy Stone confirmed that the company is committed to the consultation process and that the rollout could be pushed beyond the revised date of Nov. 20, 2026. Until then, existing terms remain in effect.

Following industry feedback, Meta adjusted its proposed terms on Oct. 17, 2026, by removing the word "train" from the U.S. English version and narrowing the stated purpose of its crawling activities to advertising and business products. Various organizations, including the Norwegian Media Businesses' Association and the U.K. Competition and Markets Authority, have raised concerns regarding the company's data practices.