Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard

The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.

Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.

  5257408 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
  49215492960 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00002-of-00005.gguf
  49700372160 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00003-of-00005.gguf
  49466495968 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00004-of-00005.gguf
  13481997024 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00005-of-00005.gguf

  49842093280 GLM-5.3-Flash-Q3_XL-3.86bpw-00001-of-00004.gguf
  49644428576 GLM-5.3-Flash-Q3_XL-3.86bpw-00002-of-00004.gguf
  49653319264 GLM-5.3-Flash-Q3_XL-3.86bpw-00003-of-00004.gguf
  5694702080  GLM-5.3-Flash-Q3_XL-3.86bpw-00004-of-00004.gguf

  20437336576 qwen3_8_27b.ninfer

Local models cannot win. Local GPUs for gaming win over nvidia geforce now because of network bandwidth and latency and that cost is within the same order of magnitude as owning a GPU (though still substantially cheaper)

With LLMs, network latency hardly matters compared to overall response generation time, and GPUs processing multiple requests simultaneously have total tok/s throughout much higher than when handling a single request (unlike game streaming, where the main cost benefit of putting GPUs in the cloud is just keeping them active 24/7). Cloud inference is/will be multiple orders of magnitude cheaper than owning a GPU for any reasonable amount of usage over the life of the GPU.

But it's all for the good of humanity, so how can you complain?

https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...

>Anthropic restricting game reverse engineering

Which is literally the coolest thing to come out of this...

Did they get a letter from Nintendo's legal team or something?

Open Weight models have already reached level at which you can do a lot of productive work with them. They really don’t have to keep beating newer and newer models, of course it’s great for them to do things cheaper and faster, but we’re already past point where open models have to “catch up”

> With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.

This might be the undoing of the frontier models: if they're impinging on more or less legitimate use cases with their restrictions people will turn to open models that get out of the way and let them get on with what they need to.

Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.

I agree. I am thinking this can only happen if we make training dramatically cheaper and efficient. This is why we are building a new kind of hardware.

Yes it is extremely aggravating that Claude will often raise copyright objections even when doing something like putting images in a report for personal use. The AI companies are the most blatant copyright abusers in history.

> I won't want to be beholden to them and their whims forever.

Who's keeping you beholden to them now?

What I find legitimately insane about all that is that Anthropic obviously does not have these censorship restrictions internally. So it's like.. They're just the whole world's adversary now?

I got banned from the Hermes discord today not gonna lie.

https://gnosh.ai/head-to-head.html

Did we ever get back to 16GB feasibility?

AFAIK the timeline was:

  - AI was shit, models could fit in your GPU
  - AI was shit, models could no longer fit in your GPU
  - AI got good

Any chance a good model will ever be seen in a consumer GPU again?

> We

who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.

lol people would rather worship the hackertube wierdo and the cabal

if and when the AI bubble bursts, we'll know if local models will hold up. Assuming the cyclical pullback in investment is strong enough to see local machines with 512gb or more of RAM

Sonnet 5.5 refused to implement code using selenium/firefox to scrape a webpage on my laptop after we got a 403 from a cloud-based IP. I made that clanker generate datasets for the rest of the week to use up my weekly usage that would have been wasted.

[flagged]

What did you try to make generalized harness, and what kind of failures did you encounter?

Qwen 3.8 was distilled on Opus. Qwen Flash Next 3.8 even signs commits as Opus 4.6 some of the time. So, who knows :)

The title is about LiveCodeBench, not a creative writing benchmark.

Have some shame and astroturf better

> because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine

The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.

Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.

> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.

I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.

Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.

The bonsai quants are not that awesome. Yeah they're fast and they retain the "27B" but in practice the perform similarly

5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B

"with a bit of clever harnessing" I think perhaps you're minimizing the amount of effort that goes into the best moat AI companies still have.

you're probably better off trying to do this on Qwen3.8-Flash-Next on something like an R9700 or a pair of 5060 Tis, since you can run that at like 70 t/s and it's better. Then you just run multiple slots, assuming you can fit the KV-cache, which maybe you can't. You just need like 128GB system ram too.

I need to try this... it's on my list anyway.

Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.

Unsloth q3 works well with 16G vram if you can do with 64k context.

> one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet

I'm not sure I understand. Are you saying we could put a dinner-plate sized chip that costs millions of dollars and draws tens of killowats of power into every single computer on the planet? Or that a single of these chips could provide inference as a service for every computer on the planet? Because neither of those are even remotely feasible.

Happy hallowen! ::::

Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though

Other big thing for me is being able to see the thinking.

[dead]

I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is

Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.

What part of Claude offers the best team collaboration features in your mind?

That is only the Memdoor version. The original tested against the newest 100 questions: https://github.com/slee-persis/GVS5H

Well, Greg KH emphatically states that local LLMs are more than good enough for bug finding in the Linux Kernel. [0] Seems strong enough for the task to me. [1]

He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.

Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.

[0] IIRC, it is this video: <https://www.youtube.com/watch?v=NnV_cWeoo5Q>

[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)

Tbh I think it's mostly a matter of time. I use SOTA at work, but only local at home, and the performance improvements out of local models in the last few months have been substantial, and more are on the way. To add to that, people are starting get better at harness engineering (which is itself improving with things like Harness2 etc) and also now we see people doing things like fine-tuning a Qwen 4B model to do 3D modelling with some success etc. The latent capability for local models to do what you're talking about is there, but the in-the-trenches reality is people are still working through unlocking the full potential.

edit: I should point out that it's also starting to accelerate now in that you can get your agents to help you with making them better at these tasks. So, it matters significantly less if you yourself don't have the skills to progress things, so long as you can figure out just enough to get the models to do it themselves. I suspect we'll see this dynamic ramp up quite a bit in 2027 with Qwen4.

I've built an RSS reader and I'm doing a half assed reverse engineer / web version of Tapped Out (a silly little Simpsons mobile game that EA killed a while back). I've had to go to Claude a couple times when it got stuck on something curly, but I think that was mostly just loading a fresh context and I could have solved it I'd I really tried.

I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.

I am having pretty good success with DeepSeek 0731 locally. I have vibe coded some small things like an orchestrator. For a production product with real users, I will let it do first passes on code which I then review and polish up. I'm not oneshotting, but it's still a workflow benefit.

Well in large corp this takes time, even if you have an army of Einsteins at your disposal (just like we saw with Elons Doge) because there will be 16 different spreedsheets/dbs talking abt the state of the system that all contradict each other. While the 6 people who maintain their own state to keep the lights on, will have job titles that have nothing to do with system state maintenance.

Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.

[flagged]

Yes…and no. If you put 5 stupid people together, chances are you can get a smarter answer than individuals. If you put 5 smart people together, they will argue about the wording of the question.

I kid, but also not. If the smart model already has a smart plan, odds aren’t great more smart models are going to create an even smarter plan. Not entirely true because that is the goal of parallelizing discovery attempts, but the point is there is more room to gain benefit from adding additional inference-time compute with a smaller model than one that already had a ton of compute to begin with.

This is a benchmark that solves ten hard problems in five passes. It scores 9.8 against the authors' 9.6 and Fable 5's 8.0. Memdoor workflows are the harness that replicates the original GVS5H thread harness posted in HN last week. Raw results (i.e. each call's tokens, cost and host) are in the bench folder for verification.

will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx

I would not think in terms of "how much" context, but really just how the harness manages it. I've been fiddling around with harnesses earlier this year and the rabbit hole goes very deep. I eventually had to drop it because it consumed me, and I can only do one thing well. But what I learned is that there is a lot on the table at the harness level.

The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.

Plenty of people above are explaining why that wouldn't work. Basically 100x idiots don't always add up to one mediocre <noun>.

The gain comes from checking and restarting, not from reasoning. This harness is built using Mario orchestration which works pretty much like any DAG orchestrator. It's ideal for long repeatable thinking tasks.

"think", "believe" - maybe, but this is why we need better measurability.