Stop Breaking TLS

You know, the ones that really know about security. X-PAN-AUTHCHECK type of security.

The amount of CVEs some of the big firewall companies collect make it seem like it is a competition for the poorest security hygiene.

The real problem we have is compliance theatre where someone in management forces these solutions onto their IT department just so they can check a box on their sheets and shift all responsibilities away.

Say you work at a place that deals with credit cards. You, as a security engineer, have a mandate to stop employees from shipping CC numbers outside the org.

You can educate all you want, you can have scary policies and HR buy-in, you can have all the "Anomaly detection, Zero Trust network architecture, EDR, Netflow analysis" in the world, but exactly zero of those will stop Joe Lunchbox from copy/pasting a block with a CC number in the middle into ChatGPT. You know what will? A TLS-inspecting proxy with some DLP bits and bobs.

It sucks, yes. But it works, and (short of the fool's errand of trying to whitelist every site any employee needs) it's the only thing that works.

And yes, I'm aware PCI DSS has additional requirements for CDEs and whatnot, but really this can apply to anything -- a local government office dealing with SSNs, a school with student ID numbers, a corporation with trade secrets.. these problems exist everywhere, and implementing PCI-like controls is often a bridge too far for unregulated industries.

At this point in time, Microsoft is the bigger enemy here - some of their policies are just insane and none of this MITM will help [0][1]

[0] https://www.microsoft.com/en-us/microsoft-365/roadmap?id=490...

[1] https://techcommunity.microsoft.com/blog/microsoft365copilot...

Why do we all disdain local TLS inspection software yet half the Internet terminates their TLS connection at Cloudflare who are most likely giving direct access to US Intelligence?

It's so much worse as it's infringing on the privacy and security of billions of innocent people whilst inspection software only hurts some annoying enterprise folks.

I wish we all hopped off the Cloudflare bandwagon.

And some of the arguments are just very easily dismissed. You don't want your employer to see you medical records? Why were you browsing them during work hours and using your employers' device in the first place?

At another job I was handling a support ticket where a customer was asking, in so many words, "can I get HTTP headers of requests flowing through my Envoy TLS reverse proxy?" I said that they could terminate TLS at the proxy and redo things that way, but then that wouldn't be a TLS proxy it'd be a MITM or a gateway. They could log the downstream/upstream and duration of connections, but that wouldn't help.

One really nice win is troubleshooting: inspection appliances break decades of performance and reliability work which groups like the Linux kernel developers have done, and they make all errors have the same symptom so the security team will now be the only people who can troubleshoot network-level issues. Explicit proxying can’t fix your network but it makes it very clear where the problem needs to be investigated.

It doesn't matter if every certificate authority is compromised or just one. One is all that is needed to sign certificates for all websites.

And I find it hard to argue with that.

I've been using a VPN habitually on my phone and my (personal) laptop for a decade now. Work, home, travel. Doesn't matter. It's always on.

Now they don't have to worry about it anymore, they bought a product that sits in the corner and delivers Cybersecurity™

How do you propose compliance with their exfiltration protection requirements? (And “turn down $ from those customers” is not an answer)

Considering that CloudFlare has managed to MitM a huge part of the internet, I'd say that probability is not just non-zero, but greater than by a worrying margin.

So is the benefit worth it? Is there data to prove it? Or is it just authoritarian IT departments drunk on power implementing this stuff?

I'd love to know.

For those that don't know, its a MITM proxy with certificates so that it can inspect and unroll TLS traffic.

ostensibly its there to stop data exfiltration, as we've had a number of incidents where people have stolen data and sent it to competitors. (our c-suite don't have as much cyber shit installed, despite them being the ones that are both targets more, and broken the rules more....)

Now, I don't like zscaler, and I can sorta see the point of it. But.

Our cyber team is not a centre of technical excellence. They somehow managed to configure zscaler to send out the certs for a random property company, when people were trying to sign into our VPN.

this broke loads of shit and made my team (infra) look bad. The worrying part is they still haven't accepted that serving a random property company's website cert instead of our own/AWS's cert is monster fuckup, and that we need to understand _why_ that happened before trying anything again.

[1] this makes automatic pen testing interesting because everything we scan has vulnerabilities for NFS/CIFS, FTP and TCP dns.

I use Platform.IO for firmware development and can't build my firmware unless I hotspot my phone. I would say that's a PIO bug unless there is a flag I don't know about, but it's exposed by this nuisance of a firewall.

The devs where I am spend much of our time hotspotted to our phones with the corporate network never connected so IP goes out over the mobile network.

Whenever possible we use 'do not verify server certs' flags in libs and commands which is not ideal.

For example, I've encountered zscaler setups in the wild which close TLS connections if non-HTTP traffic is encountered. Presumably the traffic inspection fails since there is no HTTP request, and this failure path closes the socket.

It's hard to say whether it's due to the customer's IT dept's config, or zscaler itself -- but as far as the customer is concerned, it's my problem.

TLS inspection products can intercept the paste transaction before the data leaves the company network, hitting the user with a "No you didn't! Shame on you!"-banner and notify the admins how a user just tried to paste hundreds of customers' personal information and credit card details into some snooping website, or into otherwise allowed LLM chat which still is not allowed to be used with confidential information.

There can even be automations to lock the user/device out immediately if something like this is going on, be it the user or some undetected malware in the user's device attempting the intercepted action. Being able to do these kinds of very specifically targeted interceptions can prevent potentially huge disasters from happening while still allowing users more freedom in taking advantage of the huge variety of productivity tools available these days. No need to choose between completely blocking all previously unseen tools or living in fear of disastrous leaks when there are fine-grained possibilities to control what kind of information can be fed to the tools and from where.

There are plenty of organizations out there where it is completely justified to enforce such limitations and monitoring in company devices. Policies can forbid personal use entirely where it is deemed necessary and legal to do so. Of course the policies and the associated enforced monitoring needs to be clearly communicated and there needs to be carefully curated configurations to control where and how TLS is or isn't intercepted so employee privacy laws and regulations aren't breached either.

Enterprise control over company devices and user control over personal devices are not so different.

A few apps do use certificate pinning nowadays, which creates similar problems, but saying "you can never add your own MitM TLS cert" is not far from certificate pinning everything everywhere all the time. Good luck creating a new home assistant integration for your smart airfryer when you can't read any of the traffic from its app.

Imo: let's make it easier! Standardize TLS configuration for all tools, make easy cert configuration of devices a legal requirement (any smart device sold with hardcoded CA certificates is a device with a fixed end date, where the CA certs expire and it becomes a brick), guarantee user control over their own TLS trust, and provide good tools to check exactly who you're trusting (and expose that clearly to users). Not really practical of course (and opens all sorts of risky games with nation state interception as well) but there are upsides here as well.

Because the Framework laptop site at frame.work is malicious, of course.

God, I love CURLing crap from my workstation and not getting the files I needed but instead a bunch of mangled HTML telling me zScaler was going to scan what I was going to download.

Bonus points that it puts me in the wrong country because I’m closer to Montreal than any American locations so half the time I’m stuck in French Canadian on the web from my New York office.

Triple bonus points that I’m required to test speed at client sites and zScaler completely mangles our presentable results.

Quadruple bonus points that I put in "because I feel like it" into every elevation request I make on my corporate machine and our "cyber team" has literally never looked at elevation reports to ask what the hell I'm doing...

https://www.iana.org/dnssec/ceremonies

Plus, most of the internal threats are embezzlers who get away with it (for awhile, at least). I did work for one place that had a Chinese national attempt to make off with the entire customer database, but he did it by burning CDs (circa 2006).

That said, we are not a business dealing with highly sensitive data or legal responsibilities surrounding data loss prevention.

If you are a business like that, say a bank or a hospital, you want to be able to block patient / customer data leaving your systems. You can do this by setting up a regex for a known format like patient numbers or bank account numbers.

This requires TLS inspection obviously.

Though this makes it harder to steal this data, not impossible.

It does however allow the C-suite to say they did everything they could to prevent it.

He's also absolutely right about the architectural problems too, single points of failure, performance bottlenecks, and the complexity in cloud-native environments.

That said, it can be a genuinely valuable layer in your security arsenal when done properly. I've seen it catch real threats, such as malware C2 comms, credential phishing, data exfiltration attempts. These aren't theoretical; they happen daily. Combined with decent threat intelligence feeds and behavioural analytics, it does provide visibility that's hard to replicate elsewhere.

But, and this is a massive but, you can't half-arse it. If you're going to do TLS inspection, you need to actually commit:

Treat that internal CA like it's the crown jewels. HSMs, strict access controls, proper rotation schedules, full-chain and sensible life-span. The point about concentrated risk is bang on, you've turned thousands of distributed CA keys into one single target. So act like it. Run it like a proper CA with proper key signing ceremonies and all the safeguards etc.

Actually invest in proper cert distribution. Configuration management (Ansible/Salt/whatever), golden container base images with the CA bundle baked in, MDM for endpoints, cloud-init for VMs. If you can't reliably push a cert bundle to your entire estate, you've got bigger problems than TLS inspection.

Train people properly on what errors are expected vs "drop everything and call security". Document the exceptions. Make reporting easy. Actually investigate when someone raises a TLS error they don't recognise. For dev's, it needs to just work without them even thinking about it. Then they don't need to work around it, ever. If they need to, the system is busted.

Scope it ruthlessly. Not everything needs to go through the proxy. Developer workstations with proper EDR? Maybe exclude them. Production services with cert pinning? Route direct. Every blanket "intercept everything" policy I've seen has been a disaster. Particularly for end-users doing personal banking, medical stuff, therapy sessions, do you really want IT/Sec seeing that?

Use it alongside modern defences. ie EDR, Zero Trust, behavioural analytics, CASB. It should be one layer in defence-in-depth, not your entire security strategy.

Build observability, you need metrics on what's being inspected, what's bypassing, failure rates, performance impact. If you can't measure it, you can't manage it.

But Yeah, the core criticism stands though, even done well, it's a massive operational burden and it actively undermines trust in TLS. The failure modes are particularly insidious because you're training people to ignore the very warnings that are meant to protect them.

The real question isn't "TLS inspection: yes or no?" It's: "Do we have the organisational maturity, resources, and commitment to do this properly?" If you're not in a regulated industry or don't have dedicated security teams and mature infrastructure practices, just don't bother. But if you must do it, and plenty of organisations genuinely must, then do it properly or don't do it at all.

> what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously

Who cares? It's not like all CAs would have to be breached, just one. CA certs are not scoped, so the moment one CA gets breached, we're all fucked. CT helps, but AFAIK it's still not enforced everywhere yet

By day two I started validating their setup. The CA literally had a typo in the company name, not a great sign.

A quick check with badssl.com showed that any self-signed(!) cert was being transparently MITM'ed and re-signed by their trusted corporate cert. Took them 40 days to fix it.

Another fun side-effect of this is that devs will just turned off TLS verification, so their codebase is full of `curl -k`, `verify_mode = VERIFY_NONE`, `ServerCertificateValidationCallback = () => true`, ... Exactly the thing you want to see at a big fintech company /s