Noise-to-Motion/#005/Post
Cloudflare will score crawlers on whether they obey your robots.txt
Cloudflare is starting to rate its Verified bots on whether they honor robots.txt and HTTP 429s, so write down which crawlers you meant to let in before the scores go public.
01 ⬩ What shipped
Cloudflare will grade the bots it already verified
Cloudflare said on Oct 16 that it now tracks whether its Verified bots keep to the rules a site publishes, and that it will turn what it sees into a trust band it plans to publish on Cloudflare Radar. The first two checks: does the bot follow your robots.txt, and does it slow down when your site answers with an HTTP 429. My read: the crawlers that feed answer engines just got a report card, and your robots.txt is the exam.
Chris Tsoukaladelis's post on the Cloudflare Blog is clear about timing. Cloudflare is using the system internally first, the public scores come later, and every Cloudflare customer will eventually choose the trust threshold for how Verified bot traffic is handled. Glenn Gabe flagged it the same day.
02 ⬩ What it changes
Your robots.txt is a decision you already made
The last time I opened a company's robots.txt, it still blocked a staging folder that hadn't existed in years and said nothing at all about AI crawlers. Nobody had chosen that. It was just what launch left behind.
The crawlers you let in decide which answer engines can describe you to a buyer, and a score only helps if you've written down which ones you meant to let in. Until now nobody checked whether a crawler followed the file. Soon someone will, in public. Cloudflare is also blunt that the file is a floor, not a fence: in its words, "a robots.txt sets the minimum rules for what bots and agents should not do."
That's the same mistake as on by default passing for approved. A crawler your file never mentions isn't one you approved. It's one you never thought about.
03 ⬩ What to do
Write down who you meant to let in
Open your robots.txt and list every AI crawler it names, with allow or block beside each. Then list the ones it doesn't name and decide on purpose. If your site runs on Cloudflare, check that Bot Management handles those crawlers the way your file says. Cloudflare's Bot Preference Sync exists to keep the two in line.
When the Radar scores go public, check each crawler you allow against its trust band. The file says what you want. The score will say whether you got it.
- 1Open yourdomain.com/robots.txt and list every AI crawler it names.
- 2Mark each one allow or block, and decide on the ones it doesn't name.
- 3On Cloudflare, match Bot Management to the file. Bot Preference Sync keeps them aligned.
- 4Check each allowed crawler on Radar once the trust bands are public.
04 ⬩ Sources
Where this came from
- Continuous accountability for Verified bots, Cloudflare Blog,
- Glenn Gabe (@glenngabe) on X,
Get the next one the day it lands.Follow on LinkedIn