It took a year to ship WebAssembly in Anubis

I'd say "wide variance in the level of kindness" is a rather British way to describe some of the opinions that have been voiced.

I forgot to do so on my phone. I noticed Firefox had slowed to a friggeb crawl when opening pages. Disabling web workers fixed that.

My wish is actually for the days of NPAPI where Webassembly would be just another plugin (like Flash) and browsers clung to mainly HTML, JS, and CSS. Then developers wouldn't be quite so keen to rely on it and the web would be much less bloated. Yes, those days are long gone... As a tradeoff, I'd settle for a simple toolbar icon that lit up when Webassembly is running, and/or a high level UI (just like Location, Camera, and Microphone) so I can enable it for just those sites that need it. I tried selling Mozilla on the idea once but they apparently weren't interested.

Baselines aren't entirely universal. I remember the days when the Mozilla (or early Firefox) browser was the only viable web client for Linux, but some sites still opted to run behind ActiveX. Today I don't think you'll find a Haiku or Amiga browser with Webassembly support, nor Firefox for Linux on IBM Power motherboards. It's easy to forget that just because most any 'nix or BSD OS has a plethora of choices, that others aren't still dealing with the "ActiveX" situation and feeling like everybody's out to get them.

This amounts to giving Google effectively complete control over the web. I'm not sure that's a good idea. IMO it's important that Firefox and Safari remain popular enough that webdevs are forced to support them.

I'm somewhat sympathetic to "If you want to 'customize' your web experience, you shouldn't be surprised when things don't work right." in terms of turning off commonly-used features, but the request isn't to make it work right. OP just wants a helpful error message.

When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.

The situation you are in is far less dire sounding when you consider that you have these options available to you.

Yes and that should be fixed before you subject real users to resource-wasting scripts.

That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.

And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.

I only hear bad things about bots lately, are there any benevolent bots?

I've done a tiny bit of scraping myself (usually to archive old websites before they die), and I've gotten banned a few times even though I wasn't hammering anything.

Looks like it's getting harder and harder to do scraping for legitimate purposes. Although I suppose legitimate is subjective :) Everything becomes valuable when it's old enough...

The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.

So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.

After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves

Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing

last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)

This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.

Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.

There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.

And then regular users: who now can't really browse your site.

And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down.

The solution basically makes the favored customer well funded AI scrapers, not users.

- https://techaro.lol/robots.txt

- https://anubis.techaro.lol/robots.txt

and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.

But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.

Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.

This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.