About the compromised eMudhra intermediate CAs

Posted by birdculture 19 hours ago

Comments

Comment by magnio 18 hours ago

> On Mac, some command-line tools included with macOS or the Xcode Command Line Tools (including curl, git, python3, ruby, perl, and openssl) check certificates against /etc/ssl/cert.pem. For certificates that chain to a root CA certificate in this file, these tools don't check Apple's block or whether the certificate has been revoked. This file will be updated in a future macOS update. If you use these tools and don't want to wait for the update, you can remove root CA certificates from the file.

What a clusterfuck

Comment by dist-epoch 17 hours ago

> If you don't use Terminal as part of your workflow, you don't need to do anything.

they solved the issue for the people they care about

Comment by LoganDark 17 hours ago

This isn't entirely accurate. Plenty of apps run Terminal commands in the background (for some reason). For example Rewind.ai (before they got acquired and installed a kill switch, Adobe Flash-style) runs `top` every two seconds, presumably to figure out its own CPU usage. What the fucking shit? But I think it's not out of the question that other apps run other crap too.

Also, if they truly didn't care about Terminal users, they would not have even published this article!!

Comment by dns_snek 16 hours ago

> Plenty of apps run Terminal commands in the background

Most notably agents like Claude Code and Codex? Easy to check if you ask it to run:

    $ curl 'https://dv-g3b-emudhra.we-have-a-problem.com/'

    Your client successfully connected to the site. That's bad!

Comment by zahlman 13 hours ago

What is the dv-g3b- part supposed to mean? I.e. why was this specific name chosen for the subdomain?

Comment by Polizeiposaune 13 hours ago

eMudhra generated multiple weak private keys and included their public counterpart in multiple intermediate certificates. You need to check that all the intermediates are rejected.

See the lists in https://bugzilla.mozilla.org/show_bug.cgi?id=2083449; I believe that "dv-g3b-" corresponds to the intermediate CA certificate listed as "EM DV TLS CA - G3B"

DV/OV/EV are "Domain Validated", "Organization Validated" and "Extended Validation" which refer to different levels of checking that the CA should done before issuing a certificate with a particular name in it.

Comment by consonaut 13 hours ago

DV is probably "Domain validated" describing the type of cert and g3b is the specific root cert the chain starts from.

Comment by 16 hours ago

Comment by dspillett 17 hours ago

> This isn't entirely accurate. Plenty of apps run Terminal commands in the background (for some reason).

A lot do it because sometimes there is no convenient library wrapping what they are trying to check or execute that they are aware of, and running a cli command is generally cheap unless you are doing it in a tight loop. A lot of the time this is “it'll do until I find the right library or bindings” code that should have been fixed before release, but it isn't all as bad as running top as you describe.

Comment by curt15 17 hours ago

CLI tools are not to be feared or considered somehow hacky. However much big tech wants people to think that computing is all about some magic apps, Mac OS is still unix, and composing specialized programs together is very much in the spirit of the unix approach to computing.

Comment by Wowfunhappy 14 hours ago

> CLI tools are not to be feared or considered somehow hacky.

I agree, you're on a *nix OS, calling a CLI tool should be a perfectly valid way to do something.

IMO the real issue is that Apple's built-in CLI tools aren't properly integrated with the rest of the system. Apple's curl used to use the same certificate store as the rest of the OS, then Apple changed it.

Comment by krackers 6 hours ago

>used to use the same certificate store as the rest of the OS

It was technically curl that dropped SecureTransport but I guess because no one at Apple was willing to maintain it and it apparently is a frozen API that doesn't support newer than TLS 1.2 (!). For a while their inbuilt curl could use both but preferred LibreSSL anyway.

https://daniel.haxx.se/blog/2025/01/14/secure-transport-supp...

Comment by LoganDark 17 hours ago

You're right, that was just the example I had on hand. Maybe a better example could be something like a video downloader that bundles yt-dlp, that could end up using this certificate store rather than the Apple APIs.

Comment by 1-more 12 hours ago

> Plenty of apps run Terminal commands in the background (for some reason).

Huh, I tried to do this a long time ago and permissions bit me and I quit. I was making a nice UI that would run ffmpeg to stitch together some video files, so it would use native UI elements to pick the input files and output location, then generate the ffmpeg command and run it. It worked fine on my machine but it ran into permissions on my friend's machine who needed it and I gave up because they found something better.

Comment by dist-epoch 17 hours ago

They published the article for the people they care above, to make it clear to them they should not worry about all these reports about Apple being vulnerable to this issue.

Comment by notJim 13 hours ago

God I'm so sick of this "everything-is-a-conspiracy" mindset. Equally likely that they're incompetent, that they simply forgot, that it's difficult because of legacy code reasons.

Comment by 975357985 12 hours ago

Is this conspiracy you're talking about in the room with us now?

Comment by invokestatic 16 hours ago

Going against the grain here but I think this is decent security engineering from Apple. They patched the highest exploitable things immediately (Safari, Mail, etc). The residual risk is using these command line tools while someone is actively MITMing your connection.

How often are people “git clone”ing a repo without browsing to the GitHub site first? Safari would throw up certificate errors. And also, TLS on Git doesn’t even authenticate the code you’re downloading is legitimate at all. It could be modified by the Git host or a supply chain attack, etc.

Yea, it’s not great. Ideally the risk is reduced to zero from patching. But a lot of the times we accept some level of cybersecurity risk out of practicality. This is why we have responsible disclosure windows lasting months. It takes a while to do proper release engineering and testing for something like a whole OS update.

Comment by dannyw 15 hours ago

I really don't understand your justification.

One of the certificates literally has a private key of 4; it's as compromised as it could quite possibly be. This is not a "one of the intermediates have a weak and possibly crackable key" or "a bunch of incorrect certs were issued but the issuance was patched".

To a MITM attacker, git clones can be easily distinguished from visiting in Safari. You could choose to only poison git clones; without poisoning anything else.

Heck, you can probably build an exploit kit in a hour; camp out around a Starbucks or McDonalds with open wifi, and start compromising git clones and deliver malicious payloads.

Comment by invokestatic 15 hours ago

An attacker absolutely cannot “easily distinguish” between a git clone and visiting the website and swap the certificate.

To know whether a request is a git client or web browser, an attacker needs to be able to inspect the contents of the web request (to check the user agent or whatever). If the site is using a legitimate certificate, the attacker has no way to inspect the contents because it is encrypted with a key they do not control.

If an attacker used their own, eMudhra-signed certificate, it would throw the certificate error in Safari.

Comment by dannyw 15 hours ago

Yes, I'm assuming network MITM and DPI.

The whole point of TLS and certificates is to counter network MITM attacks.

Browsers and clients emit a lot of fingerprints, you can probably figure it out loosely from the TLS initialisation.

Comment by invokestatic 14 hours ago

So let’s take a step back and look at this threat model.

1) An attacker is MITMing your connection. This is already not easily exploitable by most threat actors short of being on a public Wi-Fi network, otherwise you are likely at the nation-state level attacker at this point.

2) An attacker knows exactly which Git repo you’re about to clone and has prepared a malicious version (noting they can’t snoop on your encrypted web traffic).

3) An attacker has developed a way to differentiate between Git over HTTPS and Safari, by using a side channel “fingerprinting” method that works with your exact versions of git and macOS.

4) You don’t pin the git clone to a particular hash or signature.

This is already a very low level of exploitability. All of these things need to be satisfied for the attack you’re describing to work.

Now let’s look at a more practical attack that exists regardless of if you have the patch from Apple.

1) A maintainer of an open source project you use has their GitHub account compromised due to credential stuffing and commits a malicious version.

2) You don’t pin the git clone to a particular hash or signature.

This attack is significantly more exploitable and has happened over and over again in supply chain attacks.

But look at the commonality: in both, no certificate pinning or signature checking was done. If you follow best practices, this stops both attacks dead cold.

This is the security engineering. It’s not just checking boxes. It’s evaluating risks.

Comment by dpark 13 hours ago

> This is the security engineering. It’s not just checking boxes. It’s evaluating risks.

This is absolutely not security engineering. This is nit picking a specific attack factor and trying to argue that it’s impractical. I suspect you are wrong, but it doesn’t matter, because you are ignoring the broader class of attacks and focusing on one specific example. The class issue is that on a Mac, you cannot trust any terminal-based application to honor the CRL. This means that any site accessed from a terminal based application cannot be trusted. There are a billion ways to exploit that.

This is not good security engineering from Apple. There is no way to justify this. Humans make mistakes, and that’s just life, but we shouldn’t pretend that this is an acceptable state or an acceptable response.

Comment by 1 hour ago

Comment by 15 hours ago

Comment by paimapi 15 hours ago

Some questions from a netsec novice:

1) Why wouldn't you have a SSOT for certs and instead rely on patching different apps individually?

2) Doesn't this violate defense in depth? Exploits for a widely disclosed issue now have turnaround activation within hours, not weeks. Not patching all surface areas and actively disclosing as such seems very much like drawing up a map for a jailbroken LLM to gin up a rudimentary attack.

3) Also doesn't this violate the principle of secure by default especially for power users who are more likely to be the ones with higher privileges across distributed systems?

Comment by dualvariable 12 hours ago

Historically, stuff like openssl and cli command tools used flat files like /etc/ssl/certs.pem.

Browsers came along and they wanted to maintain their own cert store because operating system flat files were often out of date.

Operating systems grew their own stores because they wanted some kind of registry that was more integrated with the rest of their ecosystem.

Nobody coordinated. CLI tools and unix libraries don't care about browsers or O/S registries, operating systems have requirements so that they can't use the flat files, everyone screams about stuff like systemd turning unix into windows.

And it will continue to proliferate. System management tools often ship their own certs since the O/S certs can be out of date and untrustworthy and may not be able to talk to AWS before you bootstrap the management tool and update the system certs because people still run ancient shit like RHEL5. This is more or less the same reason browsers ship their own as well. I've made the decision to ship an embedded cert store in a product before, and nothing has fundamentally changed that would cause me to rethink that decision (even though for people who have an up-to-date O/S it is strictly additional cost that they now have to track a cert store in another product--but you have to build for the lowest common denominator).

Comment by invokestatic 15 hours ago

1) I would argue that different trust stores are a carryover macOS’s Unix heritage. Having multiple roots of trust is actually very common. For instance Firefox maintains their own root of trust from the system, it’s called NSS. Different programming languages like Java and Rust also often use different roots. It’s not amazing, and has holes, but a lot of it comes down to historical reasons and a certain level of risk acceptance.

2) Yes, but actually, this is exactly why we have defense in depth. This is a temporary measure while a permanent patch is coming. Defense in depth means a single hole does not result in compromise. I would argue that this situation with Apple demonstrates defense in depth actually working as intended. If this was the permanent solution, and no further patch was coming from Apple, then yes, it would violate defense in depth.

3) Related to above, yes. But it’s not permanent.

Comment by Mogzol 13 hours ago

> How often are people “git clone”ing a repo without browsing to the GitHub site first?

People are going to be running pull or push much more than clone, and will often be running those without opening the origin on Safari.

Comment by yunwal 3 hours ago

> How often are people “git clone”ing a repo without browsing to the GitHub site first?

What’s the point of this question? Would going to GitHub.com in my browser somehow protect me from a MITM attack here?

Comment by mattstir 15 hours ago

It's wild to see people supporting this bizarre anti-security response. Not only are they dragging their feet in fixing the security issue, they're actively reverting any manual fixes silently with each new update. That's legitimately worse than doing nothing at all! It's also disconcerting to see folks advocate for the "let's wait to fix this critical security issue because we want to make it slightly easier in the future" approach, as if Apple is incapable of deleting ~12 lines from a file as part of the normal patch release? If they can revert your manual fixes, they obviously ship that file and so could have shipped a fix with that file. Genuinely disturbing to support any of this.

Comment by DetroitThrow 14 hours ago

Yes, a huge part of why I bought Macs for my company is for some of this to be automatically handled. Only now are we getting any communication and still no date on when this will stop being rolled back on updates.

Please, Apple, break the process on getting a fix out for this. I want to stop doing a morning check in with my team about certs.

Comment by 2OEH8eoCRo0 15 hours ago

I don't think it's controversial to say that there are a lot of Apple zealots here.

Comment by Sohcahtoa82 12 hours ago

Which is kinda wild considering there's also a lot of openness zealots, too. People advocating for open source hardware, software, right to repair, etc.

Comment by gwbas1c 17 hours ago

I'd really like to know why it's taking Apple so long to ship a fix. Especially if they can explain how to update an existing computer, what's the general holdup? Is it their release process? Is there some underlying complexity that's poorly understood?

Comment by icedchai 16 hours ago

Probably they didn't want to hold up the 27.2 release by throwing this in??

Comment by iAMkenough 2 hours ago

Did they forget about their “Background Security Improvements” feature??

https://support.apple.com/en-us/102657

Comment by jorisw 14 hours ago

Only CLI users are affected.

They can follow the 4 steps (3 was not optional for me) to remediate.

Comment by pixl97 13 hours ago

Many users have no idea they are cli users, hence they would never attempt to remediate.

No, this is just laziness or stupidity on apples part.

Comment by dualvariable 12 hours ago

No, this is probably just the result of Apple getting hammered over and over on every release for bugs slipping into the releases and updates.

People were just losing their minds on here over bugs in the 27.2 beta release that hasn't even shipped yet, with wild conspiracy theories about what Apple was doing.

So you wind up with a typical megacorp release process where nothing can happen if it isn't signed off in triplicate and two operators crack open the books with the missile codes and turn the keys at the same time to launch the update.

The result is that shipping a single file for a critical security update takes way too long.

Comment by pluc 17 hours ago

Related: Ten days later, Git on macOS still trusts the compromised eMudhra certificates https://news.ycombinator.com/item?id=50271640

Comment by rethab 16 hours ago

Wonderful AWK command there. For those with hobbies other than AWK, here's what it does:

  - set variable s to that string: -v s='=== /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3'
  - on lines matching the regex between //, set skip to false: /^(===|###) /{skip=0}
  - on lines matching s, set skip to true: $0==s{skip=1}
  - if skip is false, print the line: !skip

Note the last part where it just says !skip without a curly braces. AWK defaults to print in that case.

Comment by tomjakubowski 11 hours ago

AWK is a great hobby, if anyone's looking to get into a new one. If sed is map and grep is filter, then awk is reduce.

Comment by somat 8 hours ago

AWK is a great little language. But you know how some people get hung up on the useless use of cat, I get triggered whenever I see a grep piped into an awk.

And on the subject of the useless use of cat, I like the form because it makes input a normal command(replaceable by any other command), rather than a shell special form. I sort of wish that "tee" would not default to emitting on stdout to better act as cat's compliment.

Comment by tedivm 17 hours ago

I'm rally disappointed in the Apple response here. I can't understand why they wouldn't consider `/etc/ssl/cert.pem` in scope for a security update. This arbitrary boundary has no justification at all, and makes people less secure.

Come on Apple, do better.

Comment by mrpippy 15 hours ago

It is in-scope for a security update, and they say one will be coming. They just couldn’t modify cert.pem without a security update like they did for their own SSL APIs. Hopefully a security update will be coming soon.

Comment by pixl97 13 hours ago

Ah, taking weeks for updates in a security environment where minutes matter!

Comment by LoganDark 16 hours ago

There is justification for controlling the scope of security updates. Apple practices incredible defense in depth. This is one of the reasons they can claim such privacy guarantees compared to other vendors. But, I agree that they should have been able to update this file automatically, and hope they will add that capability for the next time something like this happens.

Comment by lazzurs 14 hours ago

This is some weak sauce. I've just ditched iPhone for GrapheneOS and Apple Watch for Pebble. I'm getting more convinced something like a Framework with Linux is the answer.

Comment by dorianmariecom 18 hours ago

that should really be a security update

Comment by boxed 18 hours ago

It is, isn't it?

Comment by LoganDark 17 hours ago

Apple states in the article that a full operating system update will be required to fix the file.

Comment by jimt1234 13 hours ago

A full operating system update, which includes auto-enabling a bunch of unwanted AI features and 50-GB of local models. Ugh.

Comment by hmartin 13 hours ago

[dead]

Comment by beaconflint 17 hours ago

[flagged]

Comment by xfcgv 10 hours ago

[flagged]

Comment by LoganDark 17 hours ago

Good on Apple for providing such a detailed article. I disagree that they're doing badly by not updating this file remotely on current software versions in the wild -- most likely their Rapid Security Responses / Background Security Improvements framework does not already support this file, which is why they state a full operating system update will be required. Most likely that will be fixed for the future, but in the meantime, being transparent and open about what administrators can do about this now is the RIGHT APPROACH in this situation, and no other vendor has been nearly as thorough about this.

Comment by agwa 17 hours ago

Google, Microsoft, Mozilla, several TLS libraries and language runtimes, and even many Linux distros run by volunteers had this issue comprehensively fixed over a week ago: https://emudhra.we-have-a-problem.com/#developers

Meanwhile, it's unsafe to run `git clone` on macOS unless you manually edit some files, and then edit them again after a macOS update puts the dangerous root certificates back.

In what world has Apple been the most "thorough"?

Comment by LoganDark 17 hours ago

Apple has been the most thorough in documenting this transparently and honestly. They didn't have the process for updating that file in the wild, and most likely that will be fixed.

Apple security updates, especially automatic ones, likely don't have the capability to simply write to arbitrary files on the system or run arbitrary commands at install time. Of all vendors, certainly from Apple I'd expect this.

Most likely, other vendors either already used a similar store as the root of trust system-wide (didn't have their own separate security stacks), or their security updates do have the ability to write anywhere or do anything.

And of course, Linux distros and other platforms where the component can be updated separately from the operating system have an even easier time.

This is not Apple intentionally leaving terminal users high and dry. The article literally is the proof otherwise while they no-doubt work on a better solution.

Comment by fg137 17 hours ago

There are so many "likely" in this comment that I don't know if I should care about a single word at all. Looks pure speculation if not fiction to me.

Looking forward to a comment that is much more grounded than substantial than this.

Comment by LoganDark 17 hours ago

The article states a software update will be required to fix the store:

> can still accept these certificates until a future software update

This suggests they don't have the capability to patch it with Rapid Security Responses / Background Security Improvements, given that they haven't.

Sure, I speculate that they likely would have already if they could have. Sure, I speculate that if they couldn't have, they might add the capability for the future. Sure, I speculate about why they wouldn't have already had a process for this. Whatever whatever. Disregard all that if you want, but the fact remains in the article.

Comment by thenewnewguy 16 hours ago

All of that is Apple's fault - they designed and created MacOS. It's not like they're publishing to a third party platform.

Why would I judge Apple only on the quality of this docs page (which is good) but not on their poor work that resulted in this situation in the first place?

Comment by LoganDark 16 hours ago

This is a legacy BSD store that macOS inherited. Apple has their own security stack that already responded to the incident within 2 hours in most cases, which means all first-party and most third-party apps were already covered. They seemingly just didn't have a mechanism to update the legacy BSD store out-of-band, which is more of an oversight than "poor work".

Comment by thenewnewguy 3 hours ago

Apple created and designed the update system for their operating system from scratch. The fact it doesn't include the ability to update a security-critical file quickly is entirely their fault.

Most (if not all) of the popular BSDs managed to ship the update ages ago. I would expect a trillion dollar company like Apple to have better and more thoughtful engineering than random volunteer projects.

Comment by antiframe 7 hours ago

They didn't inherit it. They chose it. If you choose to build on something, you should support it, and support it well.

Comment by Sohcahtoa82 12 hours ago

"They're not fixing the security issue, but at least they're documenting it" is a weird argument to make.

Comment by 17 hours ago

Comment by LoganDark 17 hours ago

My reply to a deleted reply:

> I'm sorry, but how is revoking a bad cert not supported in an easy way? Along with adding a cert and using it to establish a secure connection, that seems to be the core features of a cert store?

> Yet somehow, you manage to come to the conclusion that the vendor requiring manual work is the most thorough? I mean, in documenting their failure, maybe.

> With all their talk of caring about user privacy, this seems incompetent at best.

To be fair other vendors almost universally don't document their failures at all or provide any remediation steps to users whatsoever. I always enjoy the exceptions to this rule, but it's not the norm whatsoever.

You seem to be ignoring that Apple's security framework responded to the incident within about 2 hours in most cases. Safari and other macOS apps were protected right off the bat.

It's only this legacy BSD store that needs an OS update, almost certainly because they simply didn't already have a process in place to update it out-of-band. Because they're vertically integrated, they probably don't have an arbitrary do-anything feature that can paper over the lack of foresight here, like other vendors might.

I'd honestly consider that a feature, but sure, the benefits would outweigh the risks in this one particular case.

Comment by 1970-01-01 17 hours ago

They're doing badly because they have a market cap at 5 trillion USD ($5,070,000,000,000 to $5,110,000,000,000) and chose to ditch the first principals of a CA store. Doing impressive backflips to compensate for lack of understanding first principals is exactly that.

Comment by LoganDark 17 hours ago

Can you explain what first principles you mean? /etc/ssl/cert.pem has been a fairly standard convention on BSDs, and Linux distros usually have an equivalent somewhere too, like for example /etc/ssl/certs/ca-certificates.crt.

I don't blame Apple for overlooking it when they have their own security stack that all their own apps use, which already responded to the incident within two hours of the revocation. As another commenter pointed out, Terminal commands probably just weren't a priority of their threat model. I do hope they fix that, but mentioning their market cap is not gonna help here.

Comment by 1970-01-01 17 hours ago

It's on the front page right now. Their dev tools are a complete mess when it comes to actually using the trust store as designed. Why would you dismiss this outright?

https://emudhra.we-have-a-problem.com/apple.html

Comment by LoganDark 17 hours ago

I'm not saying it doesn't matter that dev tools are affected. Those are perhaps some of the most important tools and riskiest to leave vulnerable, because that could result in malware shipped to users from a trustworthy developer, or malware executed on the developer's machine, rather than simply phishing. I just don't think it's fair to assume they did it intentionally. It's a mistake, and one that should be fixed, don't get me wrong. But there's a reason it happened and that seems fair enough to me.

Comment by 1970-01-01 17 hours ago

In the same degree as involuntary manslaughter is not intentional murder. Sure they made a mistake. Fairness has limits when you're doing it wrong and know better.

Comment by LoganDark 13 hours ago

They may not have known better back then, even if by now it seems obvious.

Comment by longsword 17 hours ago

Where exactly is it good, that:

* apple told admins how to fix the vulnerability * apple shipped update 27.2, with no fix (today) * the update reverted the admin fix quietly

Comment by LoganDark 17 hours ago

The article warned of this already (re-applying the fix after updates). That update was likely already in the pipeline before this debacle. No one can say for sure though.

Comment by ForHackernews 17 hours ago

"AppleCar sometimes explodes on bumpy roads unless you manually disarm the bomb in the trunk before driving. If you do not drive on bumpy roads, you don't need to do anything. If you have disarmed the bomb, Apple Updates will automatically re-arm the bomb after every update. Please remember to manually disarm the bomb after each update."

It's right there in the article, look how proactive they're being!

Comment by LoganDark 16 hours ago

LOL'd, you're literally correct though. I could unironically see Apple doing that given just a few things. And I'd prefer it to a vendor who stays silent about it until they have a full update out, and then just calls the deaths unfortunate or something. Or like, a vendor who already has the update ready, but withholds it while people continue to blow up because an embargo is more important for some reason.

Comment by darkarmani 16 hours ago

> most likely their Rapid Security Responses / Background Security Improvements framework does not already support this file

Why would it support this file? Are CA trust stores really that important? /s

> and no other vendor has been nearly as thorough about this

1. That's totally not true. Apple has messed up something that is trivial to fix. 2. The EU's CRA is going to crush them if they don't get their shit together.

Comment by LoganDark 15 hours ago

They didn't foresee this, and that's on them, sure. But their response to this, given the situation, is just about the best I could expect from them given the circumstances.

Would it have been better if the next operating system update already shipped a fixed store rather than another broken one? Sure. Would it have been even better if they had the ability to remotely fix the store on the many millions of already online Apple devices? Absolutely. But if they couldn't do either of those things, which seems likely at this point, this is the next best thing while they get the rest of their shit in order.

I don't think it's particularly constructive to point out how everybody else has already been able to patch this in all places. Apple's clearly in a spot right now. Blame them for that, but don't blame them for picking the best option at their disposal right now.