Cloudflare's new AI traffic options for customers

> Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. https://developers.google.com/crawling/docs/crawlers-fetcher...

Exclusion from grounding does mean that your site won't get sourced in the AI overview, but I'm not sure what the click through rates are like on those.

Saying this as a European who is pro EU.

Why should they?

How is that not conflict of interest??

Huh?

Search and AI are hand-in-hand.

They both rely on embeddings. (Unless you still do keyword-only search, but that's not as good.)

This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over this

This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.

"Block on pages with ads" is probably about preventing the AI crawlers from clicking on the ads which maybe considered cheating by the ad company.

If you want to prevent "bot attacks", maybe the "Block" option will do the trick.

But of course, to do all that you need to put some trust on Cloudflare, because they're the one identifying the bots from normal users.

For me, as someone who's hosting a Gitea instance behind Cloudflare, I have a Configuration Rule set that says: if the client is trying to access a URL that is beyond certain length limit, then trigger "Browser Integrity Check" and "I’m Under Attack", a.k.a stricter security checks.

The match expression of the rule looked something like this:

    (
      len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
      not lower(http.request.uri.path) contains ".git/" and
      not lower(http.request.uri.path) contains "api/"
    )

This rule alone basically blocked all abusive bot traffic to almost zero for my site (https://i.imgur.com/LaOjjvV.png, see the traffic drop around 10 clock and Cloudflare mitigation kicks in).

But of course, you need to figure out your own rules based on the characteristic of the website. Also, you can be more creative: for example, my actual rule is more complex than that, it also checks to see if a cookie is not set, and only triggers when all condition are met:

    (
      len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
      not lower(http.request.uri.path) contains ".git/" and
      not lower(http.request.uri.path) contains "api/" and
      not http.cookie wildcard "*!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!*"
    )

    (
      len(http.request.uri) <= !!!!!SET LENGTH LIMIT!!!!! and
      not http.cookie contains "!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!"
    )

(Note: `!!!!!COOKIE NAME!!!!!` and `!!!!!COOKIE VALUE!!!!!` are the variables you need to customize)

When you put the two rules together, it forces clients to access "shallow" (short URL) pages first as an user would normally do, before they can access "deeper" (long URL) content hosted on the site without triggering more strict security checks. If that makes sense.

Also, don't forget the cookie basically also dug a hole in the security setting. So it's really a balance between avoid annoying the user and protect your site. You need to be smart and be flexible about it, otherwise your users will just leave.

I've never understood why that isn't more common honestly. It would be cool if I could actually transfer POW from one domain to another as well. Probably wouldn't stop scrapers, but it would likely stop low margin fraud type things.

Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones.

Certainly, I imagine Anubis would be interested in adopting it if it’s effective!

However another option is to support both. Have a challenge page that requires the visitor to select one of several options. Cloudflare could be one of those.

You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge.

> If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.

Cloudflare blocks legitimate users itself sometimes [1].

[0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward.

[1]: https://news.ycombinator.com/item?id=43329320

Where do we draw the line between a personal-bot and a custom browser?

That being said, the issue with it right now is that Google uses the same crawler for both AI training and search, so this step puts a small pressure on google nonetheless to hopefully split them.