Cloudflare changes how AI crawlers scrape the internet

Website owners can choose if they want AI crawlers to access their content, and decide how AI companies can use it. AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow. Cloudflare's new default setting is the first step towards a more sustainable future for both content creators and AI innovators.

For decades, the internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information.

That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the internet is at risk.

“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself,” said Matthew Prince, co-founder and CEO of Cloudflare. “Original content is what makes the internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”

“Cloudflare’s innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Roger Lynch, CEO of Condé Nast. “This is a critical step towards creating a fair value exchange on the internet that protects creators, supports quality journalism, and holds AI companies accountable.”

“We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements,” said Neil Vogel, CEO of Dotdash Meredith. “We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web.”

"As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorised scraping and the use of our original content without fair compensation is critically important,” said Renn Turiano, chief consumer and product officer of Gannett Media. “As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP.”

AI transforming Vietnam’s businesses for the better While AI is a key in the process of innovation and accelerating economic development, it needs support from many sides to strengthen the workforce and related startups.

Accuracy more vital than ever in dawn of AI In the past year, Vietnam Investment Review has propelled itself forward by continuing its mission to deliver timely and informative content across a range of medium. In that time, in this author’s eyes, there have been some improvements in terms of scope, usefulness of content, and widening the platform for more experts and organisations to provide much-needed analysis.

How startups can navigate AI revolution Artificial intelligence is the most transformational technology of our time, with the potential to add value to every facet of our lives. Turning this potential into reality means carefully navigating the evolution of AI to find the balance between harnessing its economic benefits, while ensuring the right policies are in place to mitigate potential risks.