There's a new "Google Jail" for independent wikis
Posted by pizzaiolo 11 hours ago
Comments
Comment by JayNeely 8 minutes ago
Only after months of making sure our Steam news posts were consistently linking to pages within our wiki did we eventually see Google suddenly open the floodgates and start connecting searchers with the info they were searching for.
I've built out highly-ranked content sites before, and none of what I'd learned from getting those started seemed to work with this wiki, for no apparent reason. Google Search Console showed plenty of pages on the wiki were indexed; it just wasn't displaying them. Seeing this post I fully believe the premise that there's simply an arbitrary Google Jail for independent wikis. That sucks for search quality.
Comment by raincole 50 minutes ago
The experience was terrible. It wasn't always update-to-date with the latest patch, and on mobile there were so many ads rendering the whole thing unreadable.
The catch is that there was a better wiki, called Liquipedia. But Google never showed it first and I kept getting tricked to click that shitty fandom site.
(It was years ago. But I just tested it a bit and for most Dota-related keywords, fandom is still ranked higher than liquipedia. I don't know if the quality of fandom site has improved though.)
Comment by jmuguy 9 minutes ago
Using an extension to block fandom.com results is helpful because for other games I'm not really thinking about it when searching for something and find myself on that tumor of a website.
Comment by saghm 37 minutes ago
Comment by warmedcookie 30 minutes ago
Comment by SyneRyder 2 hours ago
XML Parsing Error: no root element found
Location: https://www.poe2wiki.net/sitemap/sitemap-poe_wiki-poe2_wiki--NS_0-0.xml
Line Number 18040, Column 1
(EDIT: Of course, after I post this, it's now working again...)On to the Hytale Wiki example. I don't know if what I'm about to say applies to Google, but I'm approaching it from the perspective of my tiny dumb indieweb indexer for my personal search engine. It is much easier for me to index from a sitemap rather than try to crawl a website, so I basically look exclusively at sitemaps.
Looking at Hytale Wiki, my process:
* Site has a robots.txt file - good!
* Robots.txt mentions a sitemap - excellent!
* The sitemap is stored at /images/sitemaps/index.xml ... oh. I would usually not index anything from a /images/ folder, because I want to index pages only, not images. This would likely trip my exclusion filters. Let's ignore that and continue.
* The sitemap is a list of an additional 26 nested Gzipped sitemaps. Oh.
This is the point where my tiny dumb indexer would stop. Gzipped sitemaps are part of the standard, but they're relatively rare on the web for small sites. They typically only get used if a sitemap file exceeds the 50,000 pages-per-sitemap limit. In this case, 26 * 50,000 makes the HyTale Wiki look like a 1.3 Million page site. Do I really want to index 1.3 Million pages, an estimated 13GB of indexed text data, about a video game I'll probably never play?
My search index is storage constrained, and my indexer is very time constrained. The time I spend indexing your site is time not spent indexing another, possibly higher quality website. So at this point, I'd just grab the front page and disappear... like Google apparently does too.
Of course, the HyTale site isn't 1.3 Million pages, it's only 4,277 articles. That would all fit in the root sitemap file, and that might be the better approach for getting indexed.
Comment by SyneRyder 2 hours ago
Then I realized - the sitemap files are being overwritten in real time. Every edit on the Wiki is causing the sitemap file to be edited in real time. That's why the sitemaps sometimes stop right in the middle of a filename when I access it - the sitemap file is in the process of being rewritten.
That's a behavior unique to a Wiki, and might explain the entire phenomenon.
Comment by masklinn 2 hours ago
Comment by HexPhantom 1 hour ago
Comment by dianliang233 1 hour ago
Comment by iamacyborg 1 hour ago
The recent issues relate to site errors due to aggressive crawling of uncached, server-intensive pages (diffs, etc) hidden behind residential proxies. We were being hit sufficiently hard that the server had stability issues and we were penalised by Google.
Comment by HexPhantom 1 hour ago
Comment by KingMob 2 hours ago
I would expect Google to handle broken or missing sitemaps, honestly.
Comment by viraptor 28 minutes ago
Comment by shuwix 1 hour ago
Comment by unglaublich 4 hours ago
Damn, another push to centralizing and walling off user content in 'fandoms', 'reddits' and other closed communities?
Have the internet cards been handed out now, and are we in the end-game?
Will old domains resell for premiums like low-background-radiation steel?
Comment by walrus01 2 hours ago
This has been a thing for a long time in domain name resale for speculation or link farming or similar, domains that are aged with "backlinks" sell for more money.
Comment by captainbland 3 hours ago
Comment by HexPhantom 1 hour ago
Comment by anal_reactor 3 hours ago
Comment by TFNA 1 hour ago
Comment by captainbland 1 hour ago
Comment by KingMob 2 hours ago
Comment by jeanmichelselli 3 hours ago
Comment by HexPhantom 1 hour ago
Comment by Ygg2 3 hours ago
Comment by Perz1val 3 hours ago
Comment by bityard 2 hours ago
Comment by remedan 6 hours ago
Comment by iamacyborg 5 hours ago
/edit
It appears this is no longer the case and the extension does things a lot better than it used to!
Comment by maroider 5 hours ago
Comment by arjie 2 hours ago
It turned out that default Mediawiki stuff doesn’t do a lot of standard SEO stuff. I just did all of it. From the jsonld tags, to the meta description, to making sure the sitemaps were correct. As soon as I did, my site started showing up on Google.
I cannot speak to these wikis’ problems but the first port of call for me would be checking that everything “standard” for this is now done. It’s just a reality of the web these days. I do get random traffic from Google now. Much less than I used to under the old regime in the 2000s but it feels more like a secular change than the flat zero listings I previously did.
I wish I had kept track of everything so that one day I could say what I did but I just followed literally everything, including removing the index.php thing that Mediawiki uses by default. So I cannot even say which actions worked and which didn’t. I can’t even recall which changes to base I did do. Link rel canonical. A robots.txt that disables access to pages that accidentally duplicate content (e.g. permalink pages). Hard to tell. But the difference was stark 0 to hundreds of pages. Literally zero dude. And it happened weeks after the changes and very suddenly.
0: It’s not what he does as a job but he might remember https://www.jrhizor.dev/
Comment by skhameneh 7 hours ago
I've been looking for explicit alternatives to Fandom for some content and it's annoyingly sparse.
Just the other day I saw some meme about how if we saw as many ads taking over sites like we do now, that was a sure sign of having your machine getting infected with a virus.
Generally I've found a good portion of websites that block users with ad blockers tend to be the worst offenders with aggressive ads...
Comment by Suzuran 50 minutes ago
Comment by nkrisc 3 hours ago
Ad blockers are malware protection.
Comment by latexr 2 hours ago
https://www.pcmag.com/news/fbi-recommends-installing-an-ad-b...
Comment by dspillett 3 hours ago
It is a common cycle: as they get more aggressive with trying to make the ad based business model work, more and more people are pushed to installing blockers. At this point there is a choice: find a less crappy business model (not an easy task, many have failed) or double down and get more iffy with the adverts and who you partner with to serve them. Eventually it gets to a point where such a high proportion or the viewers are using blockers and the next step in ads/stalking is a step too far even for them, and then the blocker blocking attempts start.
Comment by nicce 5 hours ago
That used to be the reality around decade ago for those who don’t know. Some applications changed the default front page of the browser, hijacked other websites to inject popups and so on
Comment by AlAjem 4 hours ago
Two decades now actually. By 2016 we were already seeing the mass transition to browsing on phones instead of PCs, Windows 10 was already out and for reasons I am not quite sure of the old homepage and toolbar hijackers were on the decline and adblockers were on the rise although I am sure there were still people using malware ridden Internet Explorers for years after.
Comment by nicce 4 hours ago
Comment by kotaKat 5 hours ago
I'm noticing that Ad-Shield is becoming a continual fight these days with pageloads that complete then they fly in with a "ha ha here's a fake error report turn off your adblocker aye?".
Every time their live support chat is open (why would the anti-adblock firm even offer one), I send them Goatse.
Comment by dspillett 3 hours ago
Unfortunately there is little chance that a human is anywhere near the other end of that chat these days, so your evil is somewhat wasted.
> (why would the anti-adblock firm even offer one)
They want to appear to care. And appear to be a company of real people, who you might like if you met in real life, who will be out of pocket if you don't look at their ads.
Comment by IanCal 2 hours ago
It’s like screaming at a cashier because the supermarket head office made annoying changes to the company.
Comment by alxfrnr 32 minutes ago
New domains need to prove they can be trusted to get out of the sandbox and external links from trusted sources are the best way to do that and shorten the time spent in the sandbox.
Big wikis like this go online with thousands of pages on a brand new domain. Without any external trust signal it just looks like spam for Google
Comment by mcv 10 minutes ago
Comment by jimnotgym 6 hours ago
I launched a website that is niche a few weeks ago, but totally unique. After a couple of weeks search console wakes up to tell me that they have indexed one page in 80. And that the top search term that is finding my site says exactly the name of one of my pages, yet Google is showing my homepage in search!
Way to help your users
Comment by mrweasel 6 hours ago
Comment by swed420 39 minutes ago
Comment by SyneRyder 3 hours ago
While I don't know if this will help with Google, I have my own tiny dumb search indexer, and a sitemap is by far the easiest way for me to index an entire website. I'll discover the sitemap from reading your robots.txt file. RSS helps too, but my indexer uses that mostly to find fresh pages without going through your entire sitemap again. I know Kagi's tiny Teclis indexer also uses RSS files for discovery for their indie web index.
Comment by einpoklum 6 hours ago
Dear Jim,
At Google, our intent is to help our stockholders, not our users.
Sincerely,
Alphabet inc.
Comment by phyzix5761 6 hours ago
Comment by sph 5 hours ago
Comment by phyzix5761 2 hours ago
Comment by latexr 2 hours ago
We should really stop with this cynical view that every company is run by greedy bastards who have active contempt for their customers and using it to excuse the behaviour of the worst offenders.
Comment by saghm 27 minutes ago
Obviously people will disagree with how much harm is enough to be worth regulating, and I'm not trying to make a claim here about whether it's necessary for what we're talking about around search results. My point is that the fact that public companies can theoretically put social good above profits doesn't really change the fact that in practice few do, and the common pattern is worth taking into account.
Comment by miladyincontrol 8 hours ago
It sounds like the issue isnt for "independent wikis" but for wikis competing with an existing domain already hosting one on the topic, in this case fandom.com's.
Comment by cookmeplox 6 hours ago
Comment by accountrequired 5 hours ago
Comment by HexPhantom 1 hour ago
Comment by rao-v 8 hours ago
Comment by copper-float 6 hours ago
Comment by cookmeplox 6 hours ago
But it ended up being the only reasonable place we owned where we could put Fortnite, Overwatch and Valheim wikis that wouldn't get crushed by this new Google jail situation. Of course the alternative is what the parent comment suggests ("probably just need a good base domain for game wikis") - but you have to get THAT domain out of Google jail first. Chicken and egg.
Comment by sph 32 minutes ago
Well, at least you didn't go with fossilised dung.
Comment by yreg 4 hours ago
> Of course the alternative is what the parent comment suggests ("probably just need a good base domain for game wikis") - but you have to get THAT domain out of Google jail first.
So why not get such domain, wait out until it is let out of Google Jail and then put all the wikis on subdomains?
I imagine you would quickly gain trust and people would immediately recognize the links and know that this is the place to go. As opposed to the current state where the wikis have separate domain names and for each game we need to learn which one is the "good one".
⸻
Second question: Any thoughts on liberating non-gaming wikis from fandom, such as TV show wikis?
Comment by dianliang233 1 hour ago
Comment by yreg 1 hour ago
> There’s a bit of light at the end of the tunnel, though - based on our experiments so far, it seems like once we establish these new wikis fairly well on Google, it should be safe to move them off weirdgloop.org back to whatever the appropriate name was, while keeping the existing Google juice that the subdomain picked up. I’m hoping that in 6 months or so we can just 301 redirect overwatch.weirdgloop.org to overwatch.wiki, use Google’s change of address tool, and put these wikis to where they should have been in the first place.
Comment by dianliang233 1 hour ago
Comment by yreg 2 minutes ago
For people who regularly get into new games its much easier to learn just one easily distinguished domain with the high quality wikis.
overwatch.weirdgloop.org works better than overwatch.wiki in this sense, apart from the complaints about the odd name up in the comment chain.
Comment by Shank 4 hours ago
Comment by fakwandi_priv 7 hours ago
Comment by iamacyborg 5 hours ago
Comment by dreadpiratewiki 6 hours ago
Comment by 6thsurvivor 33 minutes ago
Comment by josephjrobison 7 hours ago
That answer is quality content (higher than AI or human medians), links and pr and social activity and mentions, and then user experience.
Comment by microtherion 6 hours ago
Comment by willtemperley 2 hours ago
Comment by basilikum 1 hour ago
Comment by charlieyu1 4 hours ago
Comment by watwut 6 hours ago
Comment by KingMob 8 hours ago
Comment by sph 30 minutes ago
"On February 4, 2026, Jay Sullivan was named the new CEO. In a public statement following the arrival of Fandom's new CEO, the company's president, Jimmy Wales, mentioned that they intend to incorporate additional AI tools into Fandom Wikis to adapt to internet searches". No comment.
Comment by rob74 6 hours ago
Comment by nephihaha 7 hours ago
The James Bond wiki is particularly bad, but Memory Alpha seems to be better run.
Comment by pndy 3 hours ago
Some few months ago I couldn't find actors credits for particular roles they had in Star Trek franchise. For whatever reasons these bits were removed but still were present within Wayback Machine.
Then again, in the past I've seen hostile take-overs of fan made wikis into wikia/fandom "infrastructure". In one case wiki was copied, in time articles were slightly edited and new content was added containing... unsolicited fan theories.
The other case: wiki was copied 1:1 due to some inside disagreements, then incorporated into fandom and in approx. 2 years abandoned. The original died as well but someone managed to restore it from Wayback - this time in read-only mode
Comment by rob74 6 hours ago
Comment by pndy 23 minutes ago
Comment by thaumasiotes 5 hours ago
Well, I guess that's true.
It's worth noting, though, that wikipedia's donation demands cover more than the entire screen on a phone.
Comment by account42 5 hours ago
Comment by thaumasiotes 5 hours ago
Comment by einpoklum 6 hours ago
Can you elaborate on what that means? i.e. what aspect of Wikipedia, and in what way Fandom is heading there?
Comment by bariscan 3 hours ago
make some social noise. signals has always good impact.
Comment by orbital-decay 48 minutes ago
https://www.google.com/search?q=Unlikely+Valentine
Fandom, Reddit, IGN, Eurogamer, everything is on the front page except fallout.wiki which is where the community is now.
Comment by EdwardDiego 37 minutes ago
Seriously, it does, but I'm very much sarcastic when I say "don't worry".
Comment by nephihaha 7 hours ago
Comment by LoganDark 6 hours ago
Comment by pwdisswordfishq 54 minutes ago
Comment by ButlerianJihad 6 hours ago
Entertainment studios can often turn a blind eye to use of their properties when it's by fans, for fans, and promotes those franchises. They are not sending C&D to people making avatars, or cosplaying, or memeing their screengrabs into notoriety.
It is weird to think that the WMF and Wikia/Fandom had a common founder and origin.
Comment by sylware 2 hours ago
Comment by amazingamazing 3 hours ago
This does not say. From how it is written seems to be a one man show:
https://weirdgloop.org/blog/why-were-helping-more-wikis-move...
Comment by sph 26 minutes ago
There are multiple game studios with their official wikis on weirdgloop's infra. Runescape players basically live off its wiki, it's that mandatory, and Jagex is well aware and OK with that.
Comment by dianliang233 1 hour ago
Comment by presango 1 hour ago
Comment by ahakoiro 4 hours ago
Comment by robin0716 11 hours ago
Comment by leonidasrup 7 hours ago
Field v. Google, Inc. (2006)
https://www.practicalecommerce.com/Search-Engines-Indexing-a...
Authors Guild v. Google, Inc. (2015)
https://www.flaglerlawgroup.com/a-new-era-for-fair-use-court...
Comment by fergie 6 hours ago
OTOH if the goal is to monetize other people's contributions then yes, I totally get why the "Google jail" would be bad for that, but I'm just not sure that its a cause worth fighting for- that road basically leads to a new Fandom.
Comment by tuetuopay 5 hours ago
I've hit this on a few wikis where, for some (at the time inexplicable) reason, I'd get only the crappy Fandom wiki with above queries, and a good wiki when searching only for "<game> wiki" after seeing a link to it on reddit.
When the reason is to move away from Fandom, get people what they are looking for, and perhaps attract contributors, this is an issue. Your average gamer won't route through the Wiki's main page, rather rely on the top Google search result for their issue.
Comment by billyp-rva 2 hours ago
Well, it's a "must" because over the years people have been trained to do this, instead of going directly to a quality website they trust. However, this has also trained website owners to neglect their brand and focus only on SEO, leading to the ad-infested wiki sites we have today.
Google went through all of this many (many) years ago with the "content farm" problem, so this is nothing new, really.
Comment by cookmeplox 6 hours ago
There's a ton of historical evidence of wiki migrations (1) starting out with all of the editors and none of the readers, (2) not doing anything to get those readers to the new site, and (3) ultimately losing the whole war because the reader->editor flow was still happening on the Fandom wiki. "Invisibility" is absolutely not desirable here, even if your only goal is maximizing the amount of good contributions.
Comment by fergie 3 hours ago
Comment by philipwhiuk 1 hour ago
Comment by jdranczewski 6 hours ago
Comment by sersi 5 hours ago
So yeah, when my goal is to search for content that would be in a wiki, I'm ecstatic when I get an independent wiki over fandom. The Google Jail is the opposite of what I'd want.
Nowadays, I use Kagi which is better than Google but it's still not to the level of Google in 2010.