Active incident at Dreamhost forcing password reset

Posted by greenchair 4 days ago

Counter165Comment71OpenOriginal

Comments

Comment by duskwuff 3 days ago

Reading the tea leaves: given that the outage started with "connectivity issues affecting the File Manager", which was an install of Monsta FTP [1] - a commercial PHP application - I suspect the file manager got popped by one of the recently published PHP vulnerabilities, and users' FTP credentials may have been exfiltrated.

[1]: https://www.monstaftp.com/

Comment by traeregan 3 days ago

20 year customer here. This morning I asked their support chatbot if this is related to the PHP vulnerabilities and it assured me that's not the case. I'm skeptical.

These status updates from the PHP incident would also have us believe that they were never directly affected: https://www.dreamhoststatus.com/pages/incident/575f0f6068263...

They're being awfully vague about this issue so far. 8 hours ago I submitted a ticket requesting more information and they've not responded yet. I guess that they're still in the thick of it.

Comment by pixl97 3 days ago

Why would their support chatbot know?

Having worked in support for years quite often higher layers of the company will give T1 support almost no information on what happened intentionally. That's pretty standard corporate CYA policy in litigious societies.

Comment by traeregan 3 days ago

I didn't think that it would know but their support UX funnels you through it so that's where I started. I followed it up with a ticket.

Comment by nkrisc 3 days ago

I would never expect first tier support to know, let alone ever admit, something like that.

Comment by nikanj 3 days ago

Asking the chatbot is as useful as asking the pet parrot.

Comment by paranoidrobot 3 days ago

You're being unkind to the parrot. Parrots are known for mimicking others in their environment.

One might reasonably expect it to scream/shout and swear if one mentions PHP.

Comment by MuffinFlavored 3 days ago

hopefully their hashed/salted passwords, right? not plaintext?

Comment by duskwuff 3 days ago

Internally, sure. But the application that was (probably) compromised was an FTP client, so it would have received the plain text of those passwords from users as they logged in.

Comment by spopejoy 3 days ago

> the application that was (probably) compromised was an FTP client

Thank you, this actually makes some sense. It's disturbing to get these emails with zero explanation what's going on, and very frustrating that user community has to guess. But PHP vuln in a web client intercepting user logins is the only thing that makes sense, otherwise that would mean passwords in the clear or some other 1990s-era nonsense.

Comment by trollbridge 3 days ago

For those customers who are still logging in with FTP…

As a precaution they reset everyone’s email passwords, presumably from an assumption people use the same ones for both.

I’m surprise they still let people use FTP.

Comment by jacquesm 3 days ago

I came across such a thing recently. The counterparty was a bank, and they point blank refused to update their processes so the people I talked to had no choice but to continue to use something they knew was hopelessly outdated. SFTP was released what, 3 decades ago or so? There is a good chance that whatever was on the other side of that upload was not capable of any kind of secure connection.

Comment by bawolff 3 days ago

I dont really see a problem with ftp over TLS. The protocol might be dated, but it still works fine.

Comment by Alex13192 3 days ago

[flagged]

Comment by 3 days ago

Comment by juliusdavies 3 days ago

DH patched all their PHP versions extremely quickly.

Comment by lexwhite 3 days ago

This has been a real annoyance for me lately since I run a lot of my music production business out of my website lexwhite.net hosted on Dreamhost and Dreamhost VPS. I love and trust Dreamhost a lot, though, as they have been an exceptionally solid company that I have worked with for, dang, nearly 20 years now I think! All of this new AI stuff has been exposing vulnerabilites at a rate that is hard to keep up with for anyone. We just have to really harden our cybersecurity as much as possible now.

Comment by joshstrange 3 days ago

I got the email about this and realized I’ve been using them for just shy of 20 years. It was one of the original places I hosted things and where I hosted projects for other people. I haven’t actively used it years but migrating the last few things off has never been a priority. This might be the push I need to re-evaluate everything hosted there and shut down or migrate it all.

Comment by zoidb 3 days ago

Yeah same here! I dont pay them any money but I still have an account there. For awhile I let my domains sit there until I realized that it was cheaper to move them to Cloudflare

Comment by paulbjensen 3 days ago

Dreamhost - gosh that brings back memories…

Comment by blocked_againnn 3 days ago

000webshost anyone? Also .tk domains.

Comment by LoganDark 3 days ago

Hey, I used both of those! Those were fun days. When DotTK moved to "Freenom" though, that's when it really started to go to shit.

Comment by phantompeace 3 days ago

Only real ones remember 110MB

Comment by blocked_againnn 2 days ago

Too little space :(

Comment by SteveNuts 3 days ago

Freewebs!

Comment by xmddmx 3 days ago

Super displeased with DreamHost's handling of this. I have multiple master accounts. One of them got the email around 12pm today (over 48 hours after the incident started) saying...

"As a precaution, we have expired the passwords for some email and FTP/SFTP users on your DreamHost account. Their old passwords no longer work, and no new password has been set."

But none of those email passwords were actually expired - all were working. This is confusing - did they only expire passwords that were under threat? If a password wasn't expired, should it be trusted?

The same email suggests you change all email passwords you control.

Another account received no email warning from DreamHost so I assumed it was fine, but 7 hours later it also received the same generic email.

My complaints:

- sucks that it took them 48 hours to identify the breach - perhaps understandable

- but once the breach was identified, how the fuck does it take them over 7 rolling hours to email their customers? That's terrible.

- confusing messaging: DreamHost are claiming that passwords are expired, but in testing multiple accounts, all of them are still working. Do I need to rotate these? (I probably will just for safety).

I'm a 25+ year DreamHost customer and generally satisfied, but this feels like amateur hour.

Comment by dannyw 3 days ago

For emails over 7 rolling hours, unless you're e.g. Google or Microsoft, you kinda have to, otherwise those systems will automatically shadowban (send-to-spam) you for excessive/abnormal email sending rates. It's not great.

It also doesn't help that a lot of scammers use this, or fake cyber incidents, for phishing; so it's not like they have an easy solution for exempting "security emails" either.

Comment by xmddmx 3 days ago

Sure, but it also took them 7 hours to email their own customers on the email infrastructure run by DreamHost themselves. Since they run the IMAP and SMTP servers, seems like they could have done that in seven seconds.

Of course, if they are nuking everyone's email password that doesn't help for notification purposes -- but as of this morning, all my email logins are still working fine, so it seems they didn't actually do that? Confusing.

Comment by pushcx 2 days ago

Yeah. And if it's not all passwords under the account, the email should've had a list of the affected ones. Instead Dreamhost says both that they've already deleted the at risk passwords, and also that customers need to reset all passwords. Maybe when they mean "reset all" they meant "reset all [the deleted] passwords", but being so unclear about scope and remediation steps is a painful mistake.

A single sentence like, "If a password wasn't reset as of (timestamp), we don't believe it could have been at risk." would go a long way to fixing these incident comms, if that's the case.

Comment by xmddmx 2 days ago

Agree. And this is all made worse by the fact that there seems to be a bug in iCloud keychain syncing - I can change the password on my mac, verify it works, but when I go over to iPhone or iPad, the old password is still being used. Not sure if there's a way to force the sync process on both ends?

Comment by hmokiguess 3 days ago

I migrated all my stuff from dreamhost about 5 years ago, got this email today and felt validated from having made that decision.

Comment by rationalist 3 days ago

Dreamhost is no longer the hosting company that I originally signed up for. I'm also looking for a different host.

I might have been willing to stick it out, but their panel/website-management is so slow.

Comment by elevation 3 days ago

I love their shared hosting; I have a $100/year account with dozens of little low traffic apps in their own subdomain/user isolation. They maintain the OS, I maintain the app, I can pull code with git and even compile a python interpreter if I need.

I love that I have infinite/unmetered email accounts. In practice I run 3-4 accounts on 2-4 domains, but they don't nickel and dime you on it, I've literally setup 1200 forwarders in the past. They have a good sending reputation.

What other provider does this as well?

Comment by hmokiguess 2 days ago

have you tried digital ocean? some friends I have swear by them as good

Comment by elevation 2 days ago

Yes, they're in my rotation as well (I never want to have just one provider.) But I liked that DH would maintain the OS, the shared hosting performance never bit me (my sites are all highly performant, no wordpress dogs) so all of my time would be in patching vulns. DH giving me no-root access to a hardened, performant box is ideal all around.

Comment by dieselgate 3 days ago

Porkbun has been fine, used dreamhost for years prior

Comment by mistersquid 3 days ago

> I migrated all my stuff from dreamhost about 5 years ago

May I ask to where you migrated?

Comment by hmokiguess 2 days ago

I re-architected most of what I had to fit in a serverless deployment model on AWS and now they run well within the free-tier in AWS + Neon DB

Comment by whyage 3 days ago

What's going on with Dreamhost lately? Is it falling apart?

Comment by AznHisoka 3 days ago

I havent seen anyone talk about it for the last 10-15 yrs. I would have guessed it fell apart already

Comment by PaulDavisThe1st 3 days ago

They abandoned all support for email lists a couple of months back. Will likely move after 20+ years on DH.

Comment by trollbridge 3 days ago

Chat support works well. Email is nonexistent.

Comment by PaulDavisThe1st 3 days ago

I don't mean "support for using mailing lists". I meant "they no longer provide the services (i.e. mailman) to run mailing lists"

Comment by trollbridge 3 days ago

Nobody wants to run a service like that anymore.

Comment by bigiain 3 days ago

i wouldn't be so sure. I know I move in privacy and "anti big tech" circles, but I (and a lot of people I know) am moving a lot of my personal/recreational internet time to rss feeds of personal websites, mailing lists, and Mastodon. And there's a fair bit of discussion and small but growing use of private and very small group federated Mastodon hosts.

Having said that, very little of that is being done by the sort of people who'd host it on a service like Dreamhost.

Comment by cowlevel 3 days ago

Mailing lists are inherently dangerous as you can be marked as a spammer.

Comment by LoganDark 3 days ago

Everyone (and I mean everyone) is a spammer by default these days, so this is no problem in practice.

Comment by james_in_the_uk 3 days ago

Topicbox.com, from the team behind Fastmail

Comment by danielmeskin 3 days ago

I mean, clearly the person you’re replying to does

Comment by trollbridge 3 days ago

No, they’re still doing well.

Support is primarily via chat these days.

Pricing wise they don’t beat Herzner anymore but are somewhat competitive with Digital Ocean.

I am gradually moving everyone to my own self hosting mostly for supportability reasons. We want to react in seconds, not hours, when a customer has an issue.

Comment by 99harry 2 days ago

I've been using Contabo, and so far, the service has been quite stable and professional. Updates are handled automatically, and I haven't encountered any major issues DigitalOcean seems a bit slower and more expensive in comparison, especially when Contabo offers a VPS with 4 CPU and 8 GB of RAM for just 4.50 euro/month.

Comment by beej71 3 days ago

I'm still at DH (have been for 20 years), but taking a peek at Hetzner... seems compelling at 30% cheaper than I pay now.

Comment by cowlevel 3 days ago

Just be warned Hetzner will delete your account at the slightest sign of controversy. It's fine for Minecraft but I wouldn't want to host a political website there.

Comment by forgotmypw17 3 days ago

Working pretty well for me.

Comment by donohoe 3 days ago

Same. Into my second decade with them. Great value, no real complaints.

Comment by monster_truck 3 days ago

I used them for entirely too long, and only continued to do so because a classmate in high school used my referral code to sign up for the account that would become his career of selling and managing websites for local churches.

I could not believe what the hell they were charging for a year of hosting with a dedicated ip (which they at the time required if you wanted https, real scumfuck move), so I didn't pay.

I guess I hope they actually deleted my shit because I sure didn't get this email

Comment by rstupek 3 days ago

How long ago was that because at the beginning you did have to dedicate an IP for https prior to SNI

Comment by xmddmx 2 days ago

[dead]

Comment by forgotmypw17 3 days ago

Ah, I noticed this yesterday, figured it was just me.

Comment by therein 3 days ago

Now that's a blast to the past. What's next, Rapidshare.de is having issues?

Comment by tuwtuwtuwtuw 3 days ago

Angelfire shut down this year so not them at least.

Comment by pixel_popping 3 days ago

Remember when they put that 25GB daily limit vs MU unlimited, that was the start of its death.

Comment by dvdyzag 3 days ago

Dang, memory unlocked

Comment by jamesfinlayson 3 days ago

Counting down until you can download again...

I do not miss those days.

Comment by dgiorgaras 3 days ago

[flagged]

Comment by hvalero2001 3 days ago

[flagged]

Comment by bone_421 3 days ago

[flagged]

Comment by kagouhui 3 days ago

[flagged]

Comment by sunjester 3 days ago

[flagged]

Comment by jawns 3 days ago

I've been a user for more than 20 years. With their shared hosting plan, which was about $5/mo. when I joined and is now $13/mo, I've been able to host my personal blog and professional site, as well as probably 3 dozen other sites over the years, including several that have hit the HN front page and never got hugged by the squeeze of death.

Honestly, compared to some of the professional environments I've worked in, it's a joy to have an idea, create a few PHP files, and in a matter of seconds have that idea live.

Comment by rpdillon 3 days ago

Been a customer for decades, love them.

Comment by 0898 3 days ago

Do you remember the cool newsletters they used to do?

Comment by ceejayoz 3 days ago

A shitload of small business sites that were built years ago and haven't been touched since.

Comment by SoftTalker 3 days ago

If they are static sites that shouldn't be a problem. Many are probably not though.

Comment by ceejayoz 3 days ago

My suspicion is at least half are unpatched WordPress installs.

Comment by Jgrubb 3 days ago

My suspicion is closer to 95%

Comment by Kwpolska 3 days ago

And the remaining 5% are unpatched Joomla and Drupal installs.

Comment by pockybum522 3 days ago

Me, for all the reasons listed so far in comments

Comment by Arainach 3 days ago

It's cheap and reliable with predictable costs. It hasn't caused me any problems in a decade. Most of the websites I want to host don't need cloud anything, they're statically generated.