Anthropic's open-source framework for AI-powered vulnerability discovery

0: https://redfloatplane.lol/blog/17-why-share/ (and related posts, I guess)

I've said many times that I believe "using the computer will transparently involve having it write and run code for you" (and if you're not technical you won't even know it!). What you're saying goes in that direction as well.

I feel that it's often better for us to create purpose-built tools for our lives, and with every model release, the complexity of those tools grows.

These are really personal tools: they solve a problem that other people might have, but are very tied to your own specific way of working, and would be hard to explain or adapt to someone else. So: shop jigs.

I have about 10 custom scripts and programs that are like this -- I haven't felt like this since college! Back then I had all the time in the world to customize my setup...now I have agents!

In a way, I want to show this to all my friends, but whenever I mentally trace how that would go, I realize they wouldn't really understand a bunch of the quirks they have, because they are _my_ quirks. They're reasonably complex pieces of tech that solve my problems very well, which are themselves particular versions of broader problems, and which I (at least for now) have no interest in supporting.

It's so clear we're heading in this direction, and yet so many people still believe code will be for the elites. Maybe production-code...As for the rest, I think soon your mom and dad are going to have their computer running code it wrote to serve them. Security-wise it's scary, but it's exciting to think about!

And even if you did… I spent months refining AI workflows that were just obsoleted by ultracode.

I am sure that in many organizations, teams responsible for this sort of work have less and less users coming to them.

We won't reuse open source libraries as libraries we import, but as design inspiration for the bespoke tools we make.

It's too cheap to make your own stuff and too expensive to be stuck with someone else primitives.

But grounding AI Coding in existing tools is incredibly powerful.

May even be an order of magnitude more

It's an estimate, so it might be wrong, but it gives the ballpark based on our experience. Happy to hear everyone's feedback.

But even this larger number, in turn, can be about 1/10th the cost of a formal engagement to discover the type of findings it seems to be going for: things that do not show up from PR reviews or even /security-review without the pre-work steps in the open-source framework guided by an expert. That's not counting the time and delay to figure out how to do that engagement.

Bluntly: if it matters, while this is a month's vibing budget for a single scan, it is also "pennies on the dollar" dirt cheap.

At the same time, its findings still need an expert. Its suggestions may be helpful, they may be actively harmful, depends on the prework quality.

Recommendation to IT department heads: spend a couple grand on this, use the scare page to rustle up the budget to build a relationship with a red team that can find, triage, help remediate if needed, and train your in-house team to be "security minded".

Reminiscent of the early days of tax automation where importing a W2 cost hundreds of dollars until people realized typing in 6 boxes worth of data was easy and paying the automation fee ate up their entire tax return.

https://github.com/space-bacon/SRT

Significantly improve every frozen model overnight. LFG.

Or they want to diversify

> If AI tokens were so magical in creating new value in developing software applications generally, they wouldn't be selling tokens directly.

That requires to build and sell a whole product they have little experience with, competing with their own customers. Not a great place for an AI vendor still trying to establish itself. It’s a lot of distraction, when you already have a lot to deal with the existing business. And strategically not too valuable

I don't understand this argument. I've ran and sold a semi-successful SaaS. The exhausting and frustrating parts are all the things an LLM cannot help you with. Coding the product is not the bottleneck or what grants you success.

This doesn't follow at all. Anthropic's revenue is growing 10x year over year selling tokens. Their tokens can be super magical, let them enter established industries and displace incumbents, and get 100% annual growth in those industries, and they would still be better off prioritizing selling tokens, because it's a great business.

What your argument shows is that there are limits. Their tokens are not quite powerful enough to make infinite money instantly in every area of software. Admittedly, that does seem true.

We started out with many companies forbidding their employees to use remote LLMs on their source code because of security concerns. Now many companies are starting to believe that they must analyze their all their source code with remote LLMs because of security concerns. When trusting Anthropic becomes normalized, that means they can sell more services that require access to the source code.

If hardware were so magical in creating new value generally, TSMC would be designing the chips instead of selling fabrication as a service.

That is what US chip companies used to do, by the way (back when there was silicon in Silicon Valley, before they got their lunch eaten by Taiwan). If TSMC had to design all of the chips they fabricate now, they would be doing a lot less business. Conversely, if any other company that wanted to design a chip had to build their own cutting-edge fab first, NVIDIA would not exist.

So, tokens are used to produce sloppy code, and then this thing uses more tokens to fix vulnerabilities in the slop ? Whats not to like in this business model ? Similar to microsoft's. Create an OS which is vulnerable, and then enable business models for anti-virus software. Everyone wins.

More seriously, linters are turned off in ci because the amount of time spent chasing false-positives is prohibitive.