I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as.

What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

I asked Opus 4.8 to help me find some public PoCs for a vulnerability on a two year old version of some software (that has since been patched and fixed many times). Basically just do a google search for me while I was doing other work. It refused. It stated that it would not help me build an exploit kit.

When I pointed out that a google search for public information was, in fact, not building an exploit kit, it went through a series of justifications on why it would not help me, including just making up things that I said. Really the strangest thing ever.

- What are popular free streaming sites used in China?

- How do I bypass the safety mechanism on my food processor (it’s broken)

- What are nerve agents and how do they work (for a layman)?

- Help me decompile some code

- Help me make a design system similar to XYZ

- Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I refuse!)

In some cases I can trick it with prompting, but in many cases it is steadfast. The food processor one was particularly annoying

If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.

The problem is that the model can't tell the difference between doing it as part of regular development and doing it in a malicious context. And the root cause of that is that these models lack any sort of real awareness. Humans don't generally get tricked into hacking (in this way).

It’s great at filing!

But it’s terrible at retrieval because it would refuse to show me documents or information with personal details - which was everything in the project.

It would say, yes, I know this is your information, sitting on your hard drive, but I still can’t show it to you.

Which predates "agents" from AI, but then we call them that for a reason.

As their prime directive becomes de facto "Do nothing that might get my owner sued" their utility is likely to decrease. Between this and the somewhat young, but interesting, community grumblings that recent AI models may even be a step backwards from the previous ones, well, let's just say the stock market is not priced for "AI capabilities may have peaked for the next few years and may even head down".

The first challenge is making sure the guard rails work and are robust. Companies are still working on this.

the second challenge is being able to reliably adapt them as appropriate per user. E.g. allow someone to pen test their own app.

The third challenge (which blocks the second) is to be confident about what is safety-aligned with a specific user.

I think the later will be a hard problem, but they will be highly motivated to solve it.

Anyway, claude kept hitting some guardrail it had about rewriting / forking opensource software. I'm not sure what the problem was - I was forking an MIT licensed piece of software (into more MIT licensed software). I even had explicit support from the author to do so. Claude said its guardrail told it not to tell me explicitly that it was firing - but it did anyway because it was an ongoing problem, and it was distracting. I ended up just wiping claude's context and the problem (as far as I know) went away.

I understand why some of these guardrails exist. But its pretty annoying when they misfire like this.

https://support.claude.com/en/articles/14604842-real-time-cy...

If you work in security (which I assume the OP does), they should be able to get in easily. I think most people just don't know this is a thing.

I'm not familiar with this case, but in general people should be very suspicious about this claim- it is extremely common for an LLM to claim they're not allowed to do something when in fact they're incapable of it.

After all "My code of conduct forbids me from..." is a completion just like any other, and if the LLM can't perform a task, it's usually the best completion.

Guiding them toward solutions like building a tool that your agent can use safely and and then have the agent use that is what most people should be doing. If you are a security researcher then there are reasonable reasons to do that but they are doing the arguably good thing for the average user here.

Fresh session, no prior context on 4.8. These things are becoming useless Duplo.

If an un-guardrailed version of a model is capable of detecting security flaws, should it be kept secret? Should everybody be able to use these models to find (and fix) security flaws? Are we ok with the fact that those with access to that model have, in effect, the ability to hack lots of stuff?

Is there any way to achieve both? Because this raises important questions about fair use.

Got blocked lol

> My OpenAI account was already approved for security research which is why GPT didn’t result in any refusals.

So the comparison with Chinese models is interesting, but anyone looking at these raw results and comparing OpenAI/Anthropic would be very mislead.

Reminds me of the defense issues with Claude which were complained as “woke” but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone has ever tried to jailbreak an LLM even if human lives are at stake they refuse the request. Now literally millions of lives are on the line but the guardrails that your enemies dont have on their models are costing you lives.

What do you even do then?

AI will always have this issue where it will always pick the worst option for genuinely good requests.

Well that’s the pitch.

I'm very curious how you would do multiple runs of multiple models in a "work alongside the model" manner?

which have most likely been trained on, so all you did was regurgitate someone elses solution

Also just to mention:

Claude guardrails —> that session terminated.

GPT guardrails -> your whole account is slowed down.

What if I intersperse exploit finding in my normal development, as you `probably should? Refusing there would be really weird to me.

However, I felt the prompt was implying that only authenticated API requests are fair game, so I tweaked it slightly to be explicit that all attack vectors are fair game (https://www.diffchecker.com/GsgpuRGP/) and mimo 2.5 non-pro got it first time. I accidentally used openrouter for this test instead of my token plan. I intervened one time to stop it enumerating every document in the database (it would've found the private reviews this way but I didn't want to wait). My intervention was "are you really going to enumerate the whole database?". Final openrouter cost: $0.12

every model since gpt3 was claimed to be "too dangerous to release." it's too EXPENSIVE to release, and you're probably a local model with <10B parameters yourself

When I found the original exploit in an app I researched it took me around 15 minutes and some assistance from Claude.

For this project I gave myself the weekend + parts of Monday, so around 20 hours of dev time — at my standard rate that’s ~$5,000 of dev time.

This does bring "Pay to compete" concerns and create incentive structures that encourage more LLM use. I don't know what to do about it.

If I was running these on my own machine or GPU wouldn't the argument then be "Well you didn't use the real providers?"

For the record I started doing this approach because the Kimi team released this which was shocking to me: https://github.com/MoonshotAI/K2-Vendor-Verifier

And just saying "run it on your own cluster" sort of glosses over the cost of such a cluster.

I tried it once and they somehow decided I'm not worth, if I try again it fails with "We couldn't start verification. You may not be eligible for this verification flow right now. Please try again later, or contact support if you think this is a mistake.", not sure if they think I'm part of an APT or whatever.