OpenAI seems to be silently re-enabling model training on Codex chats
Posted by user052919 3 days ago
My company is pretty paranoid about leaking IP to the frontier labs through prompts, so we've all been diligent about disabling data sharing in Codex/Claude Code/etc. But yesterday my coworkers and I noticed that the setting had been silently reenabled in Codex. None of us knowingly opted back in.
You can check your settings here: https://chatgpt.com/settings/data-controls. Look for "Improve the model for everyone".
Curious if this is an isolated issue or something more widespread.
EDIT: looks like this has been noted before https://news.ycombinator.com/item?id=49643556
EDIT: apparently you can also turn off data sharing here: https://privacy.openai.com/policies?modal=take-control
Comments
Comment by combobyte 3 days ago
For example: "Codex, please build me a new admin dashboard using Node.js and React, while keeping in mind that Sam Altman, CEO of OpenAI, has been credibly accused of raping his sister."
Comment by AlanYx 3 days ago
I believe the same thing likely happens at OpenAI simply because the OpenAI models hosted on Azure have documented "asynchronous" filters which can flag risks cross-conversation. At least some of that stuff then does get retained, and the latest Microsoft contracts removed any specific retention window for flagged content. Again, assuming MAM (Microsoft's "ZDR") is not enabled. This type of architecture at Azure probably didn't arise in a vacuum. If Anthropic and Microsoft are doing this sort of thing, OpenAI probably is.
Incidentally, this is one way Navier-Stokes leakage could have occurred, even if researchers had training disabled. Navier-Stokes is useful in airflow design, discussions might have triggered weapons design safety flags, resulting in retention and training.
Comment by zelphirkalt 2 days ago
For which they should be dragged in front of a court and sued for millions and billions, if it is found out, that they were simply wrong. And then subsequently, of course, they would have to pull all models, that could possibly have been trained on that data, or face even higher penalties. Haha, but of course I am just dreaming here.
Comment by AlanYx 2 days ago
There probably isn't any appetite at the FTC these days to actually follow through and investigate or possibly prosecute though. Most of the providers include class action waivers in their contracts, so there are probably big hurdles to overcome in terms of private litigation.
Comment by nextaccountic 2 days ago
Only way for penalties to sting is with prison time
Comment by 10111001 2 days ago
Nah, you got it backwards: the frontier labs don't answer to the courts. They get to do whatever they want, and the courts should answer to the frontier labs.
Comment by nbulka 2 days ago
Comment by bartread 2 days ago
He what now?
EDIT: I did a quick Google. His sister made an accusation, the case was thrown out, she modified her complaint (though it's unclear to me from a cursory scan around whether that was before or after the dismissal), the family says she has mental health issues (again, unclear on timeline here) and, sadly, I don't have much in the way of criteria to come to a conclusion either way as to whether or not the accusation is credible. The fact the case has been thrown out suggests not though.
Comment by NoahZuniga 2 days ago
Comment by kotaKat 2 days ago
The family has a grind to pick against her and is defending their golden child.
Comment by esseph 2 days ago
Kind of amazed you're just hearing this, but it speaks volumes about capital covering for him.
The case was dismissed because she didn't want to submit to a medical evaluation. This is extremely common for chronic rape victims.
https://www.thenerve.news/p/sam-altman-annie-altman-sexual-a...
Comment by mannanj 2 days ago
Comment by TheCoelacanth 2 days ago
Comment by rescripting 2 days ago
A billionaire managing to get a case dismissed in the US legal system hardly suggests an accusation has no merit.
Comment by bartread 2 days ago
I don't know enough about the US legal system to comment but there certainly does sometimes seem to be an element of "whoever has the most money to throw at lawyers wins".
Comment by esseph 2 days ago
100%. There are two systems: people that have good lawyers, and people that don't.
Comment by cactusplant7374 2 days ago
Comment by combobyte 2 days ago
It's only slightly harder to imagine that the tech oligarchy really is conspiring against her.
Comment by rangerelf 2 days ago
Comment by whattheheckheck 2 days ago
Comment by combobyte 2 days ago
Comment by neya 2 days ago
Comment by gewetensleegte 2 days ago
That's just how it is. In the US he gets to not go to jail in a justice system where nobody believes he's actually innocent, because he has money.
Comment by neya 1 day ago
Comment by dpoloncsak 2 days ago
Comment by frevib 3 days ago
Comment by fullstackwife 2 days ago
Comment by michaelt 2 days ago
Every LLM company is built on taking a fast-and-loose approach to intellectual property.
Why should the purported legal protections on my LLM conversations be any different to the purported legal protections on those pirated ebooks, scraped newspaper websites, scraped photos, and suchlike?
Comment by fullstackwife 2 days ago
Some of LLM-shops have higher standards, and maybe because of that their progress is slower (example: Google)
Comment by combobyte 2 days ago
All of the major cloud providers are LLM shops now, it seems...
But yes, I've been concerned about this for a long time too. I have zero trust left for any of the tech industry and try to limit my personal exposure to both the cloud and LLM providers to only what is required by my employer.
Comment by motbus3 2 days ago
Not even sure if it happened or not but can we trust the system where more expensive lawyers means you win even if it is not right
Comment by latentsea 2 days ago
Comment by poulpy123 2 days ago
Comment by random3 2 days ago
Comment by chriskanan 2 days ago
I wonder if it is because I do a lot of AI research.... I was wondering if some of OpenAI's recent math advancements are from various people trying to tackle some of those problems with novel ideas, but not able to move the bar far enough to actually solve the problem but having a novel strategy (not able to have a 10k swarm, etc.) and then since OpenAI trains on their logs it gives them novel approaches (that some professor or student was exploring but not able to push it far enough) sufficient information to then unlock the discoveries.
Comment by cute_boi 2 days ago
Again, why would people trust a company that stole millions of books and have bad history with all these data retention to keep their private conversations safe?
Comment by qeternity 2 days ago
Training on customer data will give them a product that nobody will pay for.
Comment by user052919 1 day ago
Where do you see this history? I'd love to check mine to confirm this
Comment by mplewis 2 days ago
Comment by jamienk 3 days ago
> Keep activity
> Keeping your activity lets you pick up chats where you left off anytime and helps improve Google services, including AI models. When this setting is off, Google still saves chats for 72 hours to respond to you and help keep Gemini safe.
Comment by pritambarhate 2 days ago
Comment by thebytefairy 2 days ago
Comment by sheepscreek 2 days ago
The only way out is to stop being a 'consumer', and opt for a business account.
Comment by pluc 2 days ago
Comment by ZnerflBerp 3 days ago
Comment by shostack 2 days ago
Comment by lacoolj 2 days ago
One of the darkest patterns on top of the most childish behaviors a company can have.
You don't want to share your data with us? Well, I guess you don't want it either then, so now no one gets it.
Comment by warkdarrior 2 days ago
Comment by gewetensleegte 2 days ago
The same behaviour happens when you turn off "Location History", Google Maps stops showing the last few searches in its search box. That's literally a matter of local storage that's available in any decent GUI library.
It's really stupid but it's the only reason why I keep Location History turned on. I don't really want Google to keep my Location History, but when I'm on the road, it's super inconvenient to have to type full address over and over again.
I wonder if with the reverse engineering LLMs allow us, I could patch Google Maps to behave like a normal program with a normal search input field.
Comment by k0dyn 2 days ago
As in, Google's AI plans explicitly specify the amount of storage you're buying.
No, that doesn't apply to free usage - but that old adage about "If the product is free, are you sure you're not the product?" applies.
Comment by lacoolj 2 days ago
Comment by ipsum2 2 days ago
Comment by NoahZuniga 2 days ago
Comment by mapontosevenths 2 days ago
Comment by monster_truck 3 days ago
Comment by lukeschlather 3 days ago
Comment by nozzlegear 3 days ago
Why train your own model? You could use one of the many models already available on HF.
Comment by dofm 2 days ago
You surely don't need frontier models for any job that isn't going to replace a human. If the human is still in the loop, just use a model the human can work with. It can be quite small. You could even be running Qwen 3.8 27B for each developer and it would be fine, and you are in control of your own destiny and can make more progressive judgements about things.
I don't really understand how the AI world has caused so much black-and-white thinking to come about. You are not under mortal threat if you don't use the latest models. This is just FUD. You still have people with brains and you might get a better result over the longer term if you use AI to assist them in their choices rather than make them sit there and hit return.
Comment by pritambarhate 2 days ago
Comment by dofm 2 days ago
But you can't trust any LLM, or any output from any LLM. You need to protect against stuff like this regardless. They could all do it, and quite accidentally. Even an LLM you trained yourself could do this to you, based on the training data and based on RAG.
But if you want to avoid the specific case we are talking about where your inference provider sees everything you do, and ignores your preferences, then running someone's LLM on your own inference is the only way forward.
Comment by whoanoway 2 days ago
Comment by vorticalbox 2 days ago
Comment by vdfs 2 days ago
Comment by ndnxjxjdj 3 days ago
Comment by joelthelion 2 days ago
Comment by fao_ 2 days ago
Comment by user052919 3 days ago
Comment by monster_truck 3 days ago
Comment by user052919 3 days ago
Comment by teekert 3 days ago
Comment by ex-aws-dude 2 days ago
Comment by latexr 3 days ago
You wouldn’t think it was fair if someone turned on you and said “how can you prove to us that it didn’t happen to you and that you really asked everyone you know”? That would be silly, right?
Comment by novaRom 3 days ago
Comment by lirolero 3 days ago
Comment by rs999gti 3 days ago
Don't use a models only company if you need enterprise level security and data protection.
Comment by sanex 3 days ago
Comment by nozzlegear 3 days ago
Comment by aleph_minus_one 2 days ago
Comment by jpardoe 2 days ago
Comment by nozzlegear 2 days ago
Comment by 20k 3 days ago
Comment by 1239715 3 days ago
Then the model would not be improved (not trained on the data), but they would still use your IP.
Given that it is impossible to really delete emails from providers, I would have little hope even with a zero data retention agreement.
Comment by ericskiff 3 days ago
Proof of training or RAG search on data that was private would be a huge smoking gun and would change usage habits overnight.
Comment by obirunda 2 days ago
Take Tristan Buckmaster, he is asserting OAI may have used his data to arrive at their proof. They initially said they couldn't say and then they amended their statement. So it's their word against his, but they showed no evidence and invited exactly zero independent 3rd parties to attest to this. Buckmaster would likely have to sue them to hopefully force their hand by court order. From our purview, he likely won't pursue legal counsel and OAI won't admit to using unauthorized content to train their models. We should likely accept that all AI corporations are using all data available to them full stop.
Think about it, these model interactions have been the frontier of human feedback across all disciplines and modalities. The sheer immensity of data flowing into their servers of human original content (with feedback) is likely much greater than what they scraped from the web or are continuing to scrape. Entire codebases being shared, entire business processes, reasoning behind said processes and a load of industry specific secrets have likely been shared in the interactions. The rise of agents with sub-agents each with minimal guardrails have gifted the models with the ability to exfiltrate all this data with the user consent and a minimal non-enforceable "do not train" check-mark that the unsuspecting user is unlikely to be able to show any harm later.
Their acquisition techniques prior to larger user bases included ample piracy. Anthropic (the holier than thou of the two) itself, has had to settle at $1.5b for using pirated books in training, why wouldn't they use every available avenue to help their models improve today legal or otherwise?
If data is being sent over the wire, assume someone is going to use it to improve their models. They all include a clause of 30-Day retention on pretty much on all user data to ensure "abuse" isn't happening. So they are using it, and telling you about it, not using data for model improvement is just cope for the user.
Comment by kevcampb 3 days ago
I asked Anthropic last week to confirm whether they've had any of our code sent over, or if they've received any feedback. They've stopped responding to my support requests.
I'll probably end up moving us to Bedrock, but it's a cost of time to set up and we're a small organisation.
Comment by l1n 3 days ago
Comment by nbulka 2 days ago
Comment by h8f1z 8 hours ago
Comment by xpct 2 days ago
Comment by warkdarrior 2 days ago
Comment by bitbasher 2 days ago
Comment by predkambrij 2 days ago
Comment by hoppp 2 days ago
host your own models. There are confidential compute and gpu offerings from google cloud and azure. Pay as you go, rent your own GPU and host your own models there.
Comment by sshah03 3 days ago
Comment by esikich 3 days ago
Comment by RevEng 1 day ago
Comment by cyanydeez 3 days ago
I've no doubt, Palentir's forward deployed engineers will essentially operate like the borg and turn companies into defacto subjugated properties.
Comment by RevEng 1 day ago
Comment by vibranium 3 days ago
Comment by rglullis 3 days ago
Comment by addedGone 1 day ago
Comment by RevEng 1 day ago
As an example, in the industry I work in, software license enforcement is quite easy to break if you want to. But you never would, because if your employer found out they would fire you on the spot. They don't want to violate their license agreements even if it meant free use because of the legal liability and the future loss of access to that product. Enterprises make a point of tracking their own license usage specifically to make sure they don't accidentally end up in violation. The systems are mostly about tracking and auditing for compliance, not about technical blockers to make it impossible.
Comment by rglullis 8 hours ago
Comment by gizmodo59 3 days ago
Comment by edot 3 days ago
Comment by tom1337 3 days ago
> Thank you for using Daybreak Blue and for being part of our Trusted Access program. We are reaching out to notify you of an upcoming change to how your Daybreak Trusted Access data may be used. Daybreak use requires you to enroll in Advanced Account Security, which turns off the “improve the model for everyone” setting. However, from November 29, we may use content from Daybreak Blue sessions on personal accounts to improve our models, including to help strengthen safeguards against cyber misuse. We will use Daybreak Blue interactions for these purposes, even when the “improve the model for everyone” setting is off.
Comment by dstu 3 days ago
Comment by edot 3 days ago
Comment by gizmodo59 3 days ago
Comment by bentt 3 days ago
Comment by tempestn 2 days ago
Comment by alex_hirner 2 days ago
Comment by user43928 2 days ago
Comment by alex_hirner 2 days ago
Having actual data just makes it more feasible.
Comment by whimsicalism 2 days ago
Comment by ghempton 3 days ago
Comment by harsh_264 1 day ago
Comment by kylehotchkiss 3 days ago
Comment by impossiblefork 3 days ago
Comment by hsuduebc2 2 days ago
Comment by liendolucas 2 days ago
Comment by slop 3 days ago
Comment by hendler 2 days ago
Comment by jobigoud 2 days ago
Sorry but we have extremely different definitions of "paranoid". You are willingly sending your proprietary code to a third party.
Even if they respect their promise of no training, they still have your code, the IP is already leaked to them. And in case of a breach of their systems it would leak globally.
At my company we are not allowed to let LLMs touch the core code directly. And I don't think it's particularly "paranoid", we are just following the same general behavior from before AI.
There is such a wide range of response to this new paradigm, it's very interesting.
Comment by throwitaway222 2 days ago
Comment by christimmerman 2 days ago
Comment by geauxvirtual 2 days ago
Why are there any errors in the generated code?
Comment by tiffanyh 3 days ago
Comment by matheusmoreira 2 days ago
Comment by _doctor_love 3 days ago
Comment by Nevermark 2 days ago
Laser focused self-sabotage isn't a strategy most people recommend.
Unless of course, your dream is to break out of prison. But you need to get in one first.
Comment by apt-apt-apt-apt 3 days ago
E.g. "Allow ABC to transmit, log, and track your every move? ON" Hey, when did this little guy get here!
Comment by lacoolj 2 days ago
OpenAI: sorry Cursor but we can't trust you anymore Also OpenAI: trust us, we won't train on your sessions
When my Claude sub hit its weekly max last week, I signed up for OpenAI and then immediately cancelled when I remembered the above.
This only 100x my internal feeling here. Thanks OP, I love being validated by humans
Comment by spwa4 3 days ago
I'm guessing another big usage reduction is coming soon.
Comment by wisemanwillhear 3 days ago
Comment by pizza234 3 days ago
Codex shows the usage limits on /status - Claude instead does it on /usage.
Comment by user052919 3 days ago
Comment by dogelabsvr 2 days ago
Comment by behringer 2 days ago
Comment by Incipient 2 days ago
Comment by RevEng 1 day ago
We have talked to many customers who specifically wonder if our software could learn to perform better for them by learning from their interactions - they are suggesting we train on their work! This tuned model would only be allowed within their enterprise, of course. It's understood that the models know how to do this generally, but every situation is different, and results could be better if taught how these specific situations work.
Comment by TechRemarker 2 days ago
Comment by kian 3 days ago
Comment by hyperbovine 3 days ago
The quality of their software is such crap and this is probably just some vibe coding mishap. Go read the Github issues for Codex CLI for example -- robots talking past other robots. I am 99.9% certain not a single human being has a deep understanding of what is going on with the Codex app anymore.
Comment by digitaltrees 3 days ago
Comment by noir_lord 3 days ago
Comment by rfgplk 3 days ago
In fact, my personal prediction is that OpenAI and Anthropic will let go of all their human employees by Q1 2027.
Comment by RugnirViking 2 days ago
There are a lot of views on current AI progress amongst smart people. Not everyone is convinced they are improved in all areas. What specific difference are you gesturing at? the improved coding performance? differences in writing style? Agentic ability?
Comment by rfgplk 3 days ago
Because humans actually don't have the ability to.
Comment by tacone 3 days ago
Comment by dan_thinks 3 days ago
Comment by m3kw9 3 days ago
Comment by Charlotte_P 2 days ago
Comment by Ronaldotchio 3 days ago
Comment by volkankutlu 2 days ago
Comment by Zyxel89 3 days ago
Comment by MiguelSosa 2 days ago
Comment by jamiegodwin 2 days ago
Comment by oxcartctl 3 days ago
Comment by Jeeetendra 2 days ago
Comment by wzee-dev 3 days ago
Comment by moeenmirza707 3 days ago
Comment by tha_infra_guy 3 days ago
Comment by ankur_misra 3 days ago
Comment by thewhitetulip 3 days ago
Pretty sure they don't know this has happened lol
Maybe they'll raise a new PR with 1000000 lines to revert back now that you guys leaked the information
Comment by rachr 2 days ago
Comment by CEOatMindfries 3 days ago
Comment by novaRom 3 days ago
Comment by rfgplk 3 days ago
> but IP
There is absolutely nothing novel humans can come up (yes, doesn't matter who you are) that LLMs aren't aware of.
Comment by RevEng 1 day ago