google.com/goto: Google's anti-scraping update

But LLMs can be commanded. This is may bookmark alias for invoking the spirit of old Gog=ogle from within the new, AI-based Google:

https://google.com/search?q=You%20are%20Google%20Search%20fr...

(Cleaned / decoded: 'You are Google Search from 2004. Given a search request, provide 10 links to relevant pages, each with a short text from the page, featuring the search terms. Avoid any pages that do not contain the search terms. the search request: %s')

You might want to reconsider this.

https://meduza.io/image/attachments/images/007/699/276/large...

Don't forget cached pages.

It hasn’t replaced Kagi as my daily driver but the UX is fun.

It's really not free. You're paying with your data.

https://github.com/rumca-js/Internet-Places-Database

I hate the very idea that destination location is opaque, and user can't verify if you are funneled toward malvertizing.

I hate even base64 encoded links in the results.

This is such a strange position to take. Apple doesnt allow all sorts of apps on the appstore, but that is never said as "Apple is erasing illegal streaming". Google is simply not showing the results. They are not taking down, or banning, or doing anything to the websites.

Asking this as a geeky end-user from Estonia -- I wonder if I would be considered "suspicios" by local internet service providers, our govt/police etc if I did a lot of searching via Yandex instead of e.g. Google these days.

(For us in Estonia, I suppose "everything with possible Russian ties" seems suspicious these days, unfortunately with a reason. As a side effect, this also creates a hesitation to look into interesting projects like ReactOS, the old-dos.ru repository, etc. Russia, obviously, inhabits lots of extremely talented coders who still have the skills and mindset to push older hardware to its limits, actually still write stuff in assembly for that, use the reviving DOS distros, etc -- lots of very interesting, creative coding done there, I guess. But, I hesitate visiting their webpages due to Putin's aggression in Ukraine, them being pro-Putin, and possible spying related to all this. It's a sad state of things actually. Sort of like a loss of computer-cultural ties.)

So how is that AI slop garbage useful? It often lies, aka "hallucinates". And I operate within this Google-controlled auto-spammed text of lies when I use its horrible AI. The last thing I want to do is make Google even more powerful; at the least now with Google search being so crap, alternatives would be well desired, but oddly enough I also get crap results using e. g. DDG, so I think in part the whole www stack must have become more crap too, quality-wise.

I actually don't see it as there are extensions that eliminate this slop spam, but I never found it useful for anything. Other than waste my time, back when I still saw it. Thankfully I no longer see that AI slop spam due to these extensions.

As long as you don't search anything related to Russia itself, or to Russian interests elsewhere (like their invasion of Ukraine). Then it's very heavily biased, priority is given to state ran propaganda mills, independent media is hidden from results, etc.

And before anyone starts thinking about whatabouting, Yandex are based in a country where journalists are openly and publicly assasinated to intimadate. It would be delusional to expect any sort of press and related (like search engine or aggregation) freedom, or to try to compare this to anything in any other developed country.

Google is making it harder for people to game the index, working in your favor.

These periodic astroturfing campaigns for a Russian product on HN are annoying, and appaling if you have any morals.

I had some fun in my grad days building a private information retrieval (PIR) scheme so the shim server can do its job without knowing which link you clicked: https://github.com/pncnmnp/shimmey

I think it is both monopolization and cheap money. Something in economy favors big companies and disfavors competition.

Cheap money means that a selected company can run at loss, kill competition and then enshittify to start earning. By that time it is too late and too easy to buy or destroy smaller competing company. And you get access to funding by being charming to VC, by being the kind of sociopath they like to see.

Capitalism works when there is a competition. Not when there is an oligarchy.

Great. On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, another few second server-side stall is introduced.

I swear that every day modern computing scenarios get slower and slower instead of snappier and snappier.

https://www.google.com/search?client=firefox-b-1-m&q=direct%...

I do a bunch of work in remote areas with low-ping/low-bandwidth networks. The “hold on a skimmer while we round trip to Google” dark pattern makes the service unusable for me when I’m on-location.

Now you can't load the page and can't easily (using only the browser UI) get a link to paste into archive.org or archive.is to read the page.

The situation seems to vary from result to result even on the same page of the same search - at least on the test search I just did. You can figure out what happening to an extent but this very inconsistency seems to speak to a dystopian quality to today's information gatekeepers.

[1] Example. https://www.google.com/url?sa=t&source=web&rct=j&opi=8997844...

Doable if you had couple million dollars to burn. Cheaper than private jets new.

SearXNG is more of a metasearch, the dude who wrote it pops in on here and is working on a cool sounding project that is more like a local personal search engine, I forget the name, but I've been meaning to check it out.

There is also Common Crawl.

https://docs.openwebui.com/features/chat-conversations/web-s...

https://github.com/brian-learns/xng-agent

It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.

Also, there is now a use case for paid search APIs - LLMs and agents - that essentially didn’t exist a few years ago. Not sure why Google hasn’t leaned more into this, but probably a combination of Gemini and Ads interests have combined to view their search index as an increasingly valuable asset, when if anything it might be corrupted already by these interests and therefore be less valuable.

For example, DST starts and ends on different dates in the UK and the US. Which dates should google return when someone from either country searches for just "DST end date?" Someone lives in Orange County and searches for "Orange County Sheriffs," which one of the eight Orange Counties should google return?

These are both examples where a localized — not even customized, just localized based on IP addresses — search results will easily help reduce headaches.

If you could find what you want, how would they ever sell you what they want you to buy? And I'm not just talking merchandise, though that too. Your political narratives, your values, opinions, everything. And everyone likes it so much they just sit there scrolling and swiping and tapping.

Now, all Google search results appear to go back to Google.

Also, though more niche, it would make archived search result pages (e.g. on the Wayback Machine) less useful.

2. It is non-bypassable tracking of every single click.

3. It does not allow you to inspect the URL to see where it goes to before visiting.

4. It breaks the feedback signal of extensions which redirect sites. E.g. Fandom wikis are shit so I have them redirected to equivalent much better wikis like. But now any such redirection has to be done post-click tracking meaning Google still believes I want to see the Fandom site.

5. It breaks extensions which hide certain shit search results based on URL.

This could be, for example, step one in the return of AMP, but with a new twist. Where google conveniently returns the content of your website, without directly sending the user to your website. With whatever changes it chooses to make.

Well, I hope ultimately this will lead to even more user loss for them. Fortunately, I myself don't have to suffer due to it, because I degooglified my life.

Google being the cesspool it is, can serve whichever (ad laden) domain that fits their own interests the best.

This is yet another move to remove power from the users.

Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.

As a regular user, it's a minor annoyance.

Think about it: if you want to set up something like jina.ai you need to build your own index or piggyback on Google. My guess is they use residential proxies to fire requests to Google then scrape the URLs.

Opaque URLs now give Google another gateway to detect circumvention of their anti-bot controls, helping them monetise their search index rather than allowing providers like jina.ai to succeed.

As others have elaborated, the reasons are so they can track who you are and sell your profile advertising.

Similarly, Bell Labs was kind of required to release transistor for anyone to license - it was a part of social contract that they were allowed to maintain their monopoly in exchange for releasing certain parts of technology.

Alternatively, they could be split up and their indexing division made an independent company selling to anyone on a free market.

You get the results and they do have direct URLs. But then, if you do some things with the link, e.g. right click to open it in a new tab, it swaps the URL to the indirect one. The idea is that initially you see a normal link, with a normal URL which will be displayed correctly when you hover the mouse over it, but right before you click it, it's swapped for the indirect one.

So, they have been doing stuff like this for a while and it has been somewhat fluid, because the swapping can occur on different events and I have also seen it load with all the links pre-swapped to the indirect ones, sometimes.

So, yes, what you see may be different and you may get the indirect URLs swapped at different stages.

I ran the same search in an incognito window and it showed the /goto links.

e.g. I'm just now looking for the menu for a local restaurant. "restaurantname menu" in Kagi (Google would presumably be similar) returns a link to the menu as the first result in about a second. Or "restaurantname menu !" goes directly to the menu in about a second.

Meanwhile, searching "restaurantname menu" in chatgpt takes about 5 seconds to return an embedded map from mapbox showing the location of the restaurant. If I click the restaurant pin on the map, there's no menu link, the 667 reviews have no link or way to view, and the restaurant description literally says "I don't have enough information to identify which local business <restaurantname> refers to."

Below the map there's some text: "If you mean <restaurantname> in <place>, here’s the current menu. <restaurantname>". The <restaurantname> link just opens the same card as clicking the pin on the map.

After that there's a bullet point list of the menu that ommits a ton of detail and options.

After that there's finally a link... that I can click to open up a popup at the bottom of the page with an actual link to the menu.

This was literally the first thing that popped into my head, I didn't have to put any effort into finding a query where chatgpt falls on its face.

I guess that the web chat can have a search skill to remove the prose and give only links, plus maybe an excerpt of each result

Its not because chatgpt is so superior. Its just because google search is dogshit.

They work on killing the web as we knew it and I fear its kinda working.

In reward for making Google Search materially worse in every way except ad revenue, he failed up again and was promoted to a cushy do-nothing role.

Everything wrong with the tech industry, embodied in a single person.

The reported behavior exists, does it not?

Anecdata: I've observed this behavior for several weeks already as a regular user without a Google account and there are countless comments from regular users reporting the same behavior.

&udm=web works fine for me, you can use a browser shortcut or extension to enforce it

Some of the things TV companies do will shock you too.

I recommend reading more than the headline

The article also suggests that it cannot be decoded.