Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
Posted by maxloh 1 day ago
Comments
Comment by throwa356262 1 day ago
Keep in mind, Anthropic started limiting access and introduced anti-distillation measures around 4.5-4.6 (?). So the majority of distillation should have happened on earlier models.
Maybe a better explanation is that they have access to the same training datasets? Which if private can again raise questions about theft, but on a very different level.
Comment by causal 1 day ago
K3-to-Fable is blue at 0.42. Is 0.42 meaningful, or did we set 0.4 as the lower bound because it makes 0.42 look significant?
Sol-to-Fable is 0.69. It's dark yellow, making this look VERY different from 0.42. But is it? What do these numbers mean in absolute terms?
Comment by causal 1 day ago
If you use the Claude Code harness on two models you will probably get very similarly styled output. I would not be surprised if K3 "stole" a lot of the harness that was leaked.
Notice how the only rows that even come close to being gray is GPT-5.4+Mini, diverging even from other GPT models. Is this because it has a wholly different training set, or (more likely IMO), did it just have a system prompt that leads to a different style?
Comment by c0n5pir4cy 1 day ago
Comment by kingstnap 1 day ago
So who's training on who's outputs?
Comment by orbital-decay 1 day ago
Worthless. Any LLM output similarity metric that uses n-grams as a source might as well measure the average temperature on Mars surface, no matter how much lipstick you put on it. It just can't have enough certainty. There used to be an n-gram benchmark popular on Twitter that showed extreme similarity of grok-3-beta to gpt-4.5-preview, while these models were trained on new base ones, came out 2 weeks apart, and were unmistakably different. Results were wildly inconsistent run to run. It didn't stop the crowd believing its creator in that DeepSeek R1 was trained on o1-preview (which was obvious bullshit as well, they were as different as two models can be). It's amazing how you can put anything on the web and everybody will believe you without checking or even understanding of what they're looking at.
K3 was trained on Claude's outputs, though - it repeats Anthropic's prompt injections 1:1 in its reasoning, which you should know if you ever tinkered with both models long enough. Good for them.
Comment by josh-paul 1 day ago
Comment by codelion 1 day ago
Comment by shiandow 1 day ago
Edit: No that doesn't seem to be what's happening here. I think it's some kind of word frequency analysis.
Comment by great_psy 1 day ago
There is an optimal answer to any question. Something that maximizes utility and minimizes tokens.
Comment by sebastianconcpt 1 day ago
Comment by pandoro 1 day ago
Comment by skybrian 1 day ago
Comment by tills13 1 day ago
Comment by skybrian 1 day ago
Comment by Laurel1234 1 day ago
Comment by applicative 1 day ago
Comment by parineum 1 day ago
It's a bit interesting how the open models are able to keep pace with the closed models except whole maintaining a steady following time.
Comment by abdullahkhalids 1 day ago
The interesting question is: will Anthropic release a Fable like model with an architecture similar to Kimi, and get the inference cost gains? They should surely beat Kimi because they can internally distill as much as they want.
Comment by orbital-decay 1 day ago
Comment by parineum 1 day ago
If using Anthropic models for distillation doesn't make them better, why do it?
Comment by orbital-decay 1 day ago
This entire story makes certain American AI shops look cartoonishly evil and Chinese ones relatively sane. Not only they want to grab without giving anything back, they also want to sabotage everyone else's AI research and do plenty of terrible things like media manipulation on the global scale and getting in bed with the government. This can't possibly end well, for the Americans in the first place.
Comment by nonethewiser 1 day ago
Comment by mosura 1 day ago
Comment by yandie 1 day ago
I don’t feel sorry for the model companies
Comment by mosura 1 day ago
Comment by nonethewiser 1 day ago
Comment by applicative 1 day ago
Comment by mosura 1 day ago
Comment by nonethewiser 1 day ago
Comment by shlewis 1 day ago
Comment by nonethewiser 1 day ago
Comment by shlewis 1 day ago
It's hard to feel sorry about the breach of their ToS, which ultimately is all Anthropic can argue, when they are constantly being sued by countless IP owners
Comment by pandoro 1 day ago
Comment by nonethewiser 1 day ago
Yeah.
LLMs
Comment by 8note 1 day ago
Comment by jayd16 1 day ago
Comment by titzer 1 day ago
Comment by verdverm 1 day ago
Comment by rvba 1 day ago
Comment by 8note 1 day ago
Comment by Fokamul 1 day ago
Comment by throwa356262 1 day ago
If I am working on something not sensitive and afterwards upload my chat log to a public dataset, can open models use that data for training?
Keep in mind, I paid for access to model and half of the work is mine (the prompts). But can the model provider forbid me from publishing my result as open data?
Comment by josefritzishere 1 day ago
Comment by refulgentis 1 day ago
But I don’t grok training enough to know that’s silly.
If my new prior is you can…that’s a pretty thin moat that’s essentially indefensible.
Comment by credit_guy 1 day ago
Fable 5 -> 0.42
Opus 4.8 -> 0.45
Sonnet 5 -> 0.45
Opus 4.7 -> 0.46
Grok 4.3 -> 0.52
There's an obvious jump at Grok 4.3, and it would not surprise me that the similarity there is because Grok used Anthropic models for training too (you can get the similarity list for Grok and it does look like the top most similar models for Grok are either Anthropic models or some Chinese models).
The damning evidence that K3 used Anthropic models for training is that K3 is more similar to those models than it is to K2.6. If you look at the Anthropic, OpenAI or Google models, they are most similar with their own other models. Not so with K3, where K2.6 is less similar than 15 other models.
Now, why is Fable 5 the most similar to K3 and not Opus 4.8. I think it's quite likely that K3 did some fine tuning at the end, when Fable 5 became available. They probably had all the infrastructure in place, and Mythos had been announced for months, so they were probably waiting for the second the newest Anthropic model was released to start using it for synthetic data generation.
Comment by throwa356262 1 day ago
Deepseek: 670B to 1.6T
Moonshot: 1.1T to 2.8T
Xiaomi: 310B to 1T
The new models also use a different architecture so I would assume in tests they will look different from previous generations.
Z.ai (GLM) and MiniMax on the other hand have continued training existing models (with some surgical changes to improve long context memory) so they should score similarly in these tests.
Comment by isoprophlex 1 day ago
Comment by nonethewiser 1 day ago
This looks very similar to the claim that distilling a model from Anthropic is the same thing as Anthropic distilling the model from information on the internet.
Which is very flawed, since distillation requires the thing to exist in the thing it’s distilled from. And no LLM model existed in the information Anthropic used to train the model. Instead the model was built using information and utilizing new technology including hardware, software, transformer architecture, etc.
Comment by wongarsu 1 day ago
In any case, I doubt Kimi was trained without "stealing" the same data. Assembling all of your training data from Claude responses seems infeasible. It's much more likely that Kimi's base model was trained similarly to any other base model, with terabytes of data from all imaginable sources. Then the model was fine-tuned with "high-quality" data, followed by reinforcement learning. Throwing in lots of chat transcripts from other chatbots into the "high-quality" dataset would be expected, and is done to some degree by everyone, but maybe a lot more for Kimi. And likely they did a lot of reinforcement learning against the Claude API
The model would exist without Claude, it just wouldn't be nearly as coherent or smart
Comment by maxloh 1 day ago
I find both claims unsound, though. Knowledge or model behavior itself is not copyrightable, so all these claims just boil down to the "I am not happy with that" argument. You cannot claim someone is stealing something you don't own in the first place.
Comment by nonethewiser 1 day ago
Comment by maxloh 1 day ago
From a strictly moral standpoint, it is illogical to state, "You stole things from my archive of stolen goods." LLM vendors need to morally own the knowledge before their accusations hold. It is unsound to claim ownership of something resulting from stolen property, regardless of the work you put into building it.
Either those accusations don't hold at all, all training is just "fair use", or the two processes you described are just different forms of "stealing."
Bottom line, even if we consider knowledge copyrightable, it is just stolen goods changing hands. You cannot claim ownership of a derivative work if you deny ownership to the sources your work was derived from. No one holds a higher moral standing than another.
Comment by jayd16 1 day ago
Comment by nonethewiser 1 day ago
Comment by jayd16 1 day ago
Comment by ghm2199 1 day ago
Comment by nonethewiser 1 day ago
Comment by yreg 1 day ago
If training on copyrighted data without authors consent is ok then distilling is ok as well.
Comment by nonethewiser 1 day ago
>If training on copyrighted data without authors consent is ok then distilling is ok as well.
Because building a model from distillation and building a model from raw data are not the same. You have to evaluate them independently. And legally its different as well. IP vs ToS (civil).
Comment by yreg 13 hours ago
They can hardly successfully sue a rival for breaking ToS.