Cross-request KV cache exposure in the GLM-5.3 Workers AI model (155 responses)

This doesn't really have anything to do with GLM-5.3. This is what happens when you share models between users and don't do your bookkeeping properly, the actual model doesn't matter. What it shows is that there isn't enough isolation between the user sessions and that memory can overlap when that should never be possible.

This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don't see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title.

"We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. "

Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?

I like this RCA a lot more than most. I appreciate the ownership and clear explanation of why their testing did not catch this bug.

Also impressed with the speed to address the issue unplug the model and then pretty quick remediation, the speed to remediation matters less to me, but good to see they can act quickly on their platform to kill the serving of the model.

I kind of forgot about the shared tenancy attack model. I think I was assuming/hoping the software was magically isolating my tokens, when in reality that's impossible for shared tenants. There's whole classes of attacks on things like shared cache, before we get to the LLM-specific features and potential bugs. Enterprises with highly sensitive data should really be on non-shared hosts.

Please consider to update the title in: Cross-request KV cache exposure in the AI model workers

Other providers probably don't even know/care that this is happening

Read this last night. I was completely thrown off when I read this line:

> We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue

Surely “we expected and error but got a successful response” merits investigating rather than moving on?

Upshot: even though sglang is written in Rust, they still got burned by undefined behavior from uninitialized memory.

A much better title for this would be something like "Cross-Tenant KV Cache Exposure in Shared LLM Inference: Analysis and Remediation". As others have pointed out there is no need to mention GLM-5.3 in the title. As someone that has been using GLM-5.3 I immediately thought there is a cause for concern here, only to find out there isn't.

Even cloudflare being unable to deploy this technology correctly makes me think there are some serious failures of knowledge transmission happening somewhere between theoreticians and practicians.

What worries me is what was actually in those leaked prompts. This is a million-token context model, and people use that context to paste whole repos and configs, .env files included. So the realistic worst case isn't someone's chat text, it's someone's API keys. The write-up explains the root cause well, but doesn't say whether the customers whose prompts ended up in other people's responses were notified.

Reminds me of a bug I found in a very popular browser testing service that spun up iOS vm's/devices for on-device testing.

I copied the clipboard from the device, and my clipboard was full of another users credentials that they were obviously trying to paste into their own VM. Their devices were leaking clipboard data to each other.

I reported the vulnerability and asked if there was any bounty program (I wasn't looking for vulnerabilities but didn't hurt to ask), to which they said no, but they could send me a signed letter thanking me of my contributions. I said, great, thanks - and never heard back.

All pretty disappointing.

It seems more specific to the platform than the model. Maybe Cross-request KV cache exposure in the AI model workers would be a better fitting title?

Interesting. That might explain the few times I've received completely unrelated responses from a certain provider's API.

I'm just surprised more serverless providers haven't found (or publicly disclosed) similar issues, yet.

Six months ago I had a very basic prompt to Claude at work return very specific, unrelated information that looked very much like it could have come from someone else in the company. We raised an incident and were told by Anthropic (in a 100% Claude generated email) that this wasn't possible and that AI sometimes makes mistakes like that. I'm not sure how much they investigated but apparently their email was enough for legal and compliance so eh.

Is anyone even offering single tenant inference as a service?

Is that a new class of tenant isolation vulnerability?

> We were aware of this mismatch before launch. We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue.

This is insane? They thought it would return errors for long context, which didn’t happen, and they proceeded assuming it was fine? Didn’t even investigate the known misconfiguration and issue before going to production?

I should probably learn how inference is served at scale. Any have a zero to one reference for this?

CloudBleed huh?

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]