Zyte study shows the biggest transformation of web access architecture since the dawn of smartphones
Only 18.5% of the world’s most-visited landing pages are operating without any technical barrier to automated access, a new study by web data extraction leader Zyte has revealed. This means that less than one in five of all of the most-visited sites are fully open and easy for automated tools to access without being blocked or challenged.
The State of Web Access 2026 report – which covers nearly 25,000 websites across 230 countries & dependencies – highlights numerous key findings, including:
- 81.5% of websites deploy at least one technical barrier, such as Web Application Firewalls (WAFs), dynamic rate-limiting, or bot management systems, to govern incoming automated traffic.
- Only 18.5% operate without detectable barriers, suggesting a shift in the ‘open web’.
- 50% of websites use two or more barriers, creating complex technical hurdles at both the network and application transport layer.
- Despite barriers, 88% of landing pages still sit at simple or easy accessibility – the two lowest tiers.
View the full report HERE
While CAPTCHAs are the most visible barrier to human users, invisible back-end security dominates automated access control, led by WAFs on over nine in ten sites, and dynamic rate-limiting deployed on over half.
Apparel & Fashion, Adult, Furniture, and Gambling are the hardest gated industries, while Airline, Mass Media & Construction sectors remain significantly more open, maintaining low-barrier access.
Over half of all websites are now stacking two or more defensive technologies on top of each other, attempting to stop AI crawlers and data collection at the protocol level long before they reach the actual page.
Three in four websites are also using robots.txt files to disallow access by the new wave of AI training bots. Newspapers, media, and publishing sites have the most sophisticated robots.txt files, featuring a curated list of allowed and disallowed agents. Retail and online services follow behind due to high perceived value of product data.
Industries are facing a strategic decision: allow open access to maximise visibility in AI search, or erect technical barriers to protect their content.
“The data confirms what our customers have been telling us for years – the web keeps getting more systematically defended, not easier to access,” said Shane Evans, CEO of Zyte. “But this research also shows that harder doesn’t mean inaccessible. The sites that matter most to our customers are well within reach with the right platform. State of Web Access 2026 gives data teams the clearest picture yet of exactly what they’re up against and what it takes to operate reliably at scale. For example: despite barriers, 88% of landing pages are nevertheless Simple or Easy to access through Zyte API, which can access the vast majority of pages using only level-1 or level-2 capabilities from its five-tier set of tooling, ensuring the price of access is kept low.”
“What strikes me most in this data is the precision of the response to AI crawlers,” said Iain Lennon, Chief Product Officer at Zyte. “Sites aren’t just tightening access broadly – they’re making deliberate distinctions between which automated visitors they’ll tolerate and which they won’t. That level of intentionality is new, and it’s accelerating. For anyone building systems that depend on web data – whether for AI training, price intelligence, or market research – understanding these access patterns isn’t optional anymore. It has to be built into the architecture from day one.”
The State of Web Access 2026 draws on Zyte’s unique position as the infrastructure layer for large-scale web data extraction, combining proprietary access data with purpose-built detection methodology across the full barrier stack.
Methodology
Zyte’s State of Web Access Report analysed technical configurations, response headers, transport-layer security (TLS) behaviours, and access-control mechanisms across nearly 25,000 high-traffic web properties globally. The dataset spans 110 distinct industry sectors across 230 countries and territories, providing an authoritative baseline of automated web access mechanics
About Zyte
Zyte is the world’s leading web data extraction platform, helping businesses and developers access, structure, and use web data at scale. Zyte API provides automated unblocking, browser rendering, and proxy infrastructure across millions of domains, serving customers in e-commerce intelligence, AI development, market research, and financial services. Founded in 2010 and headquartered in Dublin, Ireland, Zyte operates globally with offices across North America, Europe, and Asia.