A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.

The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.

Unfortunately LLMs are magic to them.

When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.

This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.

As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.

You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.

Bonus: after using them for a while, you get the beginnings of a training set if you change your mind.

You can do that with LLMs too - it's just a cost and latency question.

E.g. I use an LLM to label my e-mail. I do it because it's low risk enough that I don't care about it being incredibly precise - it does not automatically act on anything, it just labels - and it means I don't have to curate a dataset of what e.g. should get the "receipts" label.

An LLM zero-shots it well enough and cheaply enough to remove my incentive to train a model for my individual e-mail.

If I wanted to scale it up, Jev or similar would probably be better.

A dedicated classifier would almost certainly provide even better precision if I invested the time to curate a dataset.

But the threshold where that is worth it for me given the precision I get from a cheap LLM is pretty high, and it's also less flexible.

E.g. one of the nice things about using a generic model for this is that whenever I add a new tag, it starts using it immediately and odds are good that at most I need to add a line to a file with one-line descriptions of what the folder name means to get decent results.

So there's a space for both LLMs/Jev-likes and custom classifiers.

Use Jev-like models (or LLMs) when the effort to curate a dataset to train a dedicated classifier is too high, and switch to a dedicated classifier when/if you need to scale something to a point where the economics justifies the effort or when you already have the dataset anyway (EDIT: and if you start with an LLM or Jev, collect the data - it will provide a good starting point for your training set if/when you decide to train a dedicated classifier)

Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.

People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.

Universal classifier just seems nonsensical. It seems clear that it can't have accurate probability distributions on any possible question you might ask it. Moreover, giving only "maybe" answers to everything adds a layer of equivocation that makes it harder to catch it in a clear mistake.

That said, I've encountered plenty of fuzzy judgment questions now (ie is this case more like example X or example Y?) that reasoning models struggle with. I could see how, in lieu of a dedicated classifier, a model like jev, ostensibly trained to guesstimate the relative likelihood of natural language hypotheses, may do well.

But it still feels wrong to call it a classifier in the traditional sense, let alone a universal classifier.

The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).

What kind of dedicated classifier would be better? Genuinely curious.

Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/

At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow

Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well

I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.

An off-the-shelf API has way less friction, if the results are good enough why bother?

"at scale" or the above use case of "specialized domain"

Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.

System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.

Anomalies are very hard to detect on certain skin tones and creating data sets from image material is quite a lot of work, even if you have access to large clinics specialised on dermatology with diagnosed and undiagnosed material.

Not seeing LLM to perform very well here and hallucination would be a major issue (not at all a solved problem, even frontier models do hallucinate/match the most probable path. Perhaps a problem that cannot be solved without arbitrary rules.)

And I guess some SVM still outperform CNN, just that CNN provide a pretty good learning environment that can improve upon ingesting more data. Although there is a risk associated with that as well.

Still, if we had well documented data material, many algorithms and AIs would perform well. The work is getting the data in a dependable state.

More generally, continue to build systems. More complex, more robust, systems.

An LLM is just one of many tools available to integrate into a system, but it’s not going to replace every system: trivially, any system that requires physical manipulation of reality can’t be replaced by an LLM. I’m not going to eat my morning cereal using an LLM.

There are tradeoffs (cost, accuracy, all sorts of metrics) that play into which tool you use for which part of the system. There will be choices that are always more cost effective:

  - $ python -c “print(2+2)”
  - Claude, what’s 2+2?

Take the problem of determining if two people in a genealogy database represent the same person. It's a decision problem. There will be some data in structured formats you can count on, but which may be missing, or may even be wrong. There will be lots of fuzzy data (born in Newtown yes, but which Newtown?) and unstructured data (notes, etc.)

Now, do you want to try to assemble a dataset to train your own classifier on that, or do you want to throw a general purpose LLM-rooted decision model on it?

  > Do you have two pieces of metal that need to become one? 
  > But if it became  two pieces again it wouldn't be a big safety concern?
  > If that's you call Bob's OK welding!

Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.

I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?

The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.

If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.

While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.

Thinking about this every time someone shares their setup and there's LLMs doing simple periodical things directly instead of having the LLM write a deterministic Python script that will never try to hack your sandbox lol

I think its a combination of being new to classification models and not using in anger.

LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.

There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.

and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)

Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply

subnanosecond response times

model fits on CPU

Yes, this.

I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.

I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.

I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:

“It is difficult to get a man to understand something when his salary depends on his not understanding it.”

The bitter lesson tells us that general _approaches_ outperform domain-specific understanding.

That doesn't mean a general model will outperform a domain specific one.

https://en.wikipedia.org/wiki/Bitter_lesson

This is really what’s happening right now… LLMs are excellent computer users. Which makes one of the frontiers “amazing things computers always could have done, but we lacked the programmers to do”.

If basically anyone can train a high quality CNN for just about any purpose now, what new opportunities appear that computers can help with?

I believe the normal term is "gaming a benchmark test".

Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.

The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.

Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.

That's just good developer practice, not "benchmaxxing".

Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.

If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.

That is not overfitting, that is training for a specific task. The measure for overfitting is how well it works on previously unseen data, hence the 200 test images removed from training and validation. That set serves as a second order of validation for the researchers to check themselves they have not overfit the model to match only the training data.

The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is. A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.

I developed many models from scratch deployed in various fields.

When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.

This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.

Doesn't matter the domain or the task, the answer from some researchers is "LLM" always.

There's infinite front end code to train on. Virtually infinite python and shell scripts too. As I've worked my way away from such domains, the LLM outputs have gotten worse. It just makes sense.

So yes, this is an important finding that LLMs are not better than experience.