Brave Sells Its Search Index to AI, and Your Website Is Probably in It

Posted by linhns 1 day ago

Counter15Comment4OpenOriginal

Comments

Comment by pogue 23 hours ago

I use Brave browser & Brave Search and I find them both excellent. But, it clearly sounds like their profits have been coming from scraping the web and selling that to other AI companies, rather than advertising on their search engine (I almost never see ads on there despite have adblock off on that domain) or their browser products (VPN, their stupid Brave Rewards crypto which everyone should disable immediately, etc).

But, bare minimum they should have a useragent that people can opt out of through robots.txt. That's just basic etiquette. Beyond that, I'll wait for their response.

Comment by iagiagiag 14 hours ago

If they respected the robots.txt, the quality of the search engine would be much lower unfortunately. Many of the most visited websites refuse to be indexed these days. Brave search gives good result because Brave doesn't care.

Comment by pogue 14 hours ago

I don't know how true that is, to be honest. I see far more people using Cloudflare turnstile and other much more restrictive tools to try and prevent users of Brave browser, VPNs, and other anti-fingerprinting/adblocking tech from accessing their website and that no doubt includes headless browsers used for scrapping.

I understand Brave wanting to have the best results for their search, but that doesn't seem to be their primary motivation if the lawsuit is to be believed. It seems to be they're doing the whole "chad third party scraper meme"[1] and then making big bucks selling those results to the other AI companies like Perplexity that have already gotten in trouble doing that same thing in the past.

I wonder in the future if they're be a whole underground/darkweb or p2p/distributed scraping economy to get scraping data from people/countries that have no restriction on where they get their data and have no qualms about getting sued about it, like Anna's Archive. Or some kind of local client people can run on their home computer that works as a peer to peer service, slowly scraping websites from their residential IPs and getting paid a few pennies to submit the results to a huge service that collects it then resells it to the big AI companies.

[1] https://knowyourmeme.com/photos/2615826-twitters-view-rate-l...

Comment by nazmoonmunny 16 hours ago

[dead]

Comment by cite_please 23 hours ago

Free SEO? It could happen to you.