What happened to TheNumbers.com

The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem.

> And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.

Those who object to the scraping fall into several camps, but the biggest complaint I am hearing is that it increases both maintenance costs and time. In other words: it sucks when people are using your work in a manner that you find offensive, but it goes beyond that by doing actual harm.

The whole internet is about to experience USA-level greed because people are too dumb not to use AI. Just like Google killed small sites because people were too lazy to look at page 2.

I suspect the next AGPL (if not normal GPL) will explicitly prohibit training of closed models against the source material. Ideally it will prohibit training of open weight models too. Either share the entire process, or go piss up a rope.

I feel the same way too. But guess what, all the code you did not publish gets into the training corpus anyway (when you gave Codex or whatever full read access to your filesystem).

I dont get this.

You provided something for free to help people, but dont want to do that anymore because it might go into training data and help many many more people?

So far LLMs have been loan funded donations of loss leading services. They might never actually make their first dollar of profit.

Meanwhile ISPs the world over have been monetizing access to your content.

I feel like what you mean is that you want control over attribution.

AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads.

As for monetizing the training data, (a) it cost hundreds of millions of dollars to generate the weights, so why begrudge the companies that made the investment and did the research necessary to make it happen?; and (b) rest assured, whatever your doo-dad does, an open-weight model like GLM 5.2 can generate it for free using your own hardware.

So you don't have to pay anyone in that case. Well, except nVidia, I guess. Point granted there.

You can get a beefy one for about $40/mo on their server auction site.

Just... don’t miss payments. Ever. Or they’ll delete your server within a week or so.

No, they decided that would be a great way to convince you of the narrative and persuade you to pay for "security" services that further the incumbent browser monopoly.

They're not "AI scrapers", they're DDoS'ers manufacturing consent.

Where they more exhaustive or more frequent?

The "good bots" which identify themselves and follow the rules are easy enough to block, so not a problem. It is the "bad bots" which pretend to be real users and hide behind residential proxies, and so are almost impossible to block at the moment, which are the problem.

Given that there are big companies openly (i.e. on the clearweb, not even darkweb) selling access to these residential proxy botnets of compromised smart TVs[0] and mobile phones and other devices, can we not simply get a database of the these IPs and block access from them? I would venture that almost every single one of those residential users are unaware that they have devices in their homes which have been compromised and are being abused in this way, so if they were to start seeing messages from more and more sites along the lines of "Access to this site has been blocked because unusual traffic has been detected from your computer network. Please check all devices on your network and remove any malware which may be routing this traffic." then maybe we could start addressing the problem at the source.

[0] https://news.ycombinator.com/item?id=49000864

Which brings us to the old fashioned flood control mechanisms. That is, the toolkit you propose already exists and has been used for decades in various iterations to protect against various forms of attack (slashdotting, DDOS attacks, overzealous search engines, and now AI scrapers).

Have you looked into those before? Companies like Cloudflare have been at the forefront of this field for a long time now.

I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to its cooperative threading model. But today sites have a remarkable slowness to them, running many hundreds or thousands of database queries due to ORMs and N+1 problems, so that response times can be 500 ms or more and even 1000 simultaneous users stresses servers.

What went wrong is that nobody solved stuff like Russian doll caching in a general way separate from the programming language and database. We should have had ways to make dependency graphs using Etag headers as keys with real cache invalidation of dependent data. I went down that rabbit hole 10 years ago using touch events in Laravel with callbacks to handle cache invalidation when class model data was saved to the database. Also a query cache using Redis which I think might have been handled better at the database level anyway. After that experience, I can honestly say that cache invalidation is so difficult to get right that it's effectively an open problem. Meaning that programmings should use a package instead of rolling it by hand, and it should be a major concern from the start (along with sharding by user id or using something like Firebase).

Don't get me started on how the web should have been a P2P content-addressable memory anyway. Nearly everything should be available from a nearby edge peer, similarly to BitTorrent. But nobody bothered to solve how to make that work with HTTPS/SSL. I suspect that has to do with early flaws in the browser security model where the whole page has to be behind HTTPS or warnings appear. So it was never clear what was personally identifiable information (PII) or merely public data being served over HTTPS. To really solve that, we probably need real trust networks and maybe even zero-knowledge proofs.

Since these problems are so challenging to fix, and big companies can't be bothered to do it since they pulled the ladder up behind them, we're probably stuck with banal "are you human" challenge screens for the foreseeable future.

It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!

I don't think there would be any issues per se if the scrapers just paid licensing fees to get the good / complete data. But the issue was that they started to try and find exploits to get to data earlier.

[0] https://news.ycombinator.com/item?id=49000864

I presume it would also cut you off even more from referral traffic.

So most subdirectory URLs get two requests from Claude bot, the first one needless because that wasn't the URL in the tag.

Disclosure: I am a shareholder and would love for them to solve the AI bot problem and the ad problem like this.

I eventually just shut down my mediawiki instance. I couldn't find a way to keep it online and still run on an affordable VPS.

So GPTBot is now blocked.

Webscraping is among the more benign things that gambling-on-anything can drive. And even that has a negative impact, as seen here.

I have no idea why those sites are legal.

From the article.

Not the same site, but an example of the same issue.

(Agree with you more generally.)

Deplorable behavior indeed

The intelligence community came to the conclusion (after research and experiments) that things like "prediction markets" were a bad idea in 1996.

Far worse now than then, with the net and AI of 2026 and the insane number of people now online that want to make an easy buck.

Don't get me wrong; I am guessing some of us on here would do well on those places. But they should not exist.

Or at least somewhere between that and full on protection racket.

LLMs come on the scene, hammer the site until it goes offline or racks up bills that threaten bankruptcy, and the solution then is to pay for an LLM to fix it and monitor it.

That's a real nice site ya got there, it would be a shame if massive datacenters going up around the world were to start hammering it from tens of thousands of IP addresses....