# Cloudflare will score crawlers on whether they obey your robots.txt

*Published 2026-10-19* | Author: brad-webb

Cloudflare is starting to rate its Verified bots on whether they honor robots.txt and HTTP 429s, so write down which crawlers you meant to let in before the scores go public.

## Cloudflare will grade the bots it already verified

Cloudflare said on Oct 16 that it now tracks whether its Verified bots keep to the rules a site publishes, and that it will turn what it sees into a trust band it plans to publish on Cloudflare Radar. The first two checks: does the bot follow your robots.txt, and does it slow down when your site answers with an HTTP 429. My read: the crawlers that feed answer engines just got a report card, and your robots.txt is the exam.

[Chris Tsoukaladelis's post on the Cloudflare Blog](https://blog.cloudflare.com/verified-bots-accountability/) is clear about timing. Cloudflare is using the system internally first, the public scores come later, and every Cloudflare customer will eventually choose the trust threshold for how Verified bot traffic is handled. Glenn Gabe flagged it the same day.

> Cloudflare will start to score bots on trust and risk (based on observed behavior of those bots on sites running Cloudflare)
>
> Glenn Gabe, @glenngabe, [post on X](https://x.com/glenngabe/status/2111168608478838815)

## Your robots.txt is a decision you already made

The last time I opened a company's robots.txt, it still blocked a staging folder that hadn't existed in years and said nothing at all about AI crawlers. Nobody had chosen that. It was just what launch left behind.

The crawlers you let in decide which answer engines can describe you to a buyer, and a score only helps if you've written down which ones you meant to let in. Until now nobody checked whether a crawler followed the file. Soon someone will, in public. Cloudflare is also blunt that the file is a floor, not a fence: in its words, "a robots.txt sets the minimum rules for what bots and agents should not do."

That's the same mistake as [on by default passing for approved](https://www.civic.com/field-notes/on-by-default-is-not-approved-by-default). A crawler your file never mentions isn't one you approved. It's one you never thought about.

## Write down who you meant to let in

Open your robots.txt and list every AI crawler it names, with allow or block beside each. Then list the ones it doesn't name and decide on purpose. If your site runs on Cloudflare, check that Bot Management handles those crawlers the way your file says. Cloudflare's Bot Preference Sync exists to keep the two in line.

When the Radar scores go public, check each crawler you allow against its trust band. The file says what you want. The score will say whether you got it.

*A crawler check in four steps*

This week, open your robots.txt, list every AI crawler it allows or blocks, and decide on purpose about the ones it doesn't mention.

## Where this came from

- [Continuous accountability for Verified bots](https://blog.cloudflare.com/verified-bots-accountability/), Cloudflare Blog, Oct 16, 2026
- [Glenn Gabe (@glenngabe) on X](https://x.com/glenngabe/status/2111168608478838815), Oct 16, 2026

Source: https://www.civic.com/noise-to-motion/cloudflare-verified-bots-trust-score