We’ve been seeing two completely different approaches from our clients when it comes to AI. While many are trying to figure out how to get referenced more by AI platforms and show up in those AI-generated answers on Google and in conversations on ChatGPT, Perplexity, and others, we also have numerous clients who are actively trying to prevent AI from scraping their sites and training their platforms on the content they’ve created.

For that second group, there’s finally some good news. The battle between content creators and AI companies just got a new weapon. Cloudflare rolled out something called Content Signals Policy, and in my opinion, it’s about time someone tried to give publishers more say in how their work gets used.

So, here’s the deal: Most websites use a robots.txt file, which tell search engines what they can and can’t crawl. Well, Cloudflare just supercharged that concept. They’ve added three new commands that let you be way more specific about what you’re okay with.

What These New Signals Actually Do

Think of it like setting boundaries at a party. You can now tell different types of bots exactly what they’re allowed to do with your content:

• Search signals provide permission for traditional search indexing (the approach we’ve always been comfortable with).
• AI-input signals** control whether your content can be used to generate those AI Overview answers that pop up in Google.
• AI-train signals decide if your content becomes training data for future AI models.

The syntax looks pretty straightforward. You’d add something like this to your robots.txt:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Translation: “Sure, index my stuff for regular search results, but don’t you dare use it to train your next AI model.”

The Elephant in the Room…

Now comes the big question that everyone’s thinking but maybe not saying out loud: Will Google actually respect these signals? Cloudflare’s CEO, Matthew Prince, gave Google a heads-up about this whole thing, but Google hasn’t promised to comply.

And let’s be real here. Robots.txt has always been more like a polite suggestion than a legal requirement. Companies can ignore these signals if they want to, and there’s not much you can do about it, legally speaking.

Why This Matters More Than You Think

Publishers have been stuck in this impossible situation for months now. Either you keep your content open and watch AI systems potentially cannibalize your traffic (we saw this trend starting in late 2023 on some of our larger news clients’ sites), or you lock everything down and lose search visibility entirely. These aren’t exactly what you’d call great options.

This new system at least gives you a middle ground. You can say “yes” to traditional search (which still drives traffic) while saying “no” to AI training or those AI Overviews that answer questions without sending people to your site.

The Numbers Game

Cloudflare dropped an interesting stat: they predict bot traffic could overtake human traffic by 2029. That’s not just search engine crawlers anymore. We’re talking about AI training bots, research scrapers, and who knows what else, all competing for your content.

When you put it that way, having more granular control over who gets to do what with your content starts to feel pretty essential.

So what happens next?

Cloudflare has made its Content Signals Policy open source (CC0 license), hoping that other companies will adopt it as well. The more platforms that support these signals, the more pressure there’ll be on the big players to respect them.

But here’s the thing: signals alone probably won’t cut it. Smart publishers are going to want to combine this with actual bot management and firewall rules. Think of the content signals as your “please don’t” sign, and the technical barriers as your actual fence.

The Reality Check

Will Google suddenly start honoring these new directives? Probably not immediately. Will other AI companies follow suit? Hard to say. However, at least now publishers have a standardized way to express their preferences, rather than just hoping for the best.

The whole situation still feels a bit like putting up a “No Soliciting” sign and hoping door-to-door salespeople actually read it. Some will respect it, others won’t. But having the sign is still better than having no way to communicate your wishes at all.

For content creators who’ve been feeling powerless in this whole AI revolution, Cloudflare’s move represents something important: recognition that publishers deserve a voice in how their work gets used. Whether that voice gets heard is another story entirely.

The ball’s in Google’s court now. We’ll see if they’re willing to give publishers the control they’re asking for, or if this becomes another case of big tech companies doing whatever benefits them most.