Analysis
OpenAI employs hundreds of contractors who read real ChatGPT conversations and grade the model's replies, an internal effort the company calls Project Lily, according to 404 Media. The reviewers do not see usernames, and OpenAI runs an automated pass intended to strip personal information before conversations reach a human. The company acknowledges that sensitive details can still get through. The reporting describes a workforce rating and critiquing responses against a rubric, with particular attention to two behaviors OpenAI wants suppressed: sycophancy, and the model describing itself in human terms.
The RLHF Precedent
None of this is unprecedented in machine learning. Reinforcement learning from human feedback has been the dominant alignment technique since InstructGPT in 2022, and every major lab -- OpenAI, Anthropic, Google DeepMind, Meta -- pays annotators to rank outputs. What is new here is the scale and the source of the data. Early RLHF work leaned heavily on prompts written by contractors themselves or drawn from red-team exercises. Grading live production traffic from a consumer app with a user base the size of ChatGPT's is a different exposure profile, because the input distribution now includes whatever people actually type at 2am.
“Scale AI's tie-up with Meta reshuffled that market in 2025, when several labs moved annotation work away from a firm partly owned by a competitor.”
The vendor layer is where the story gets hard to audit. Labs rarely name the staffing firms behind annotation work, and the industry's suppliers -- Scale AI, Surge AI, Invisible Technologies, Turing, and a long tail of BPO shops -- sit behind NDAs that make headcount and geography difficult to confirm from outside. Scale AI's tie-up with Meta reshuffled that market in 2025, when several labs moved annotation work away from a firm partly owned by a competitor. The result is a supply chain where the most privacy-sensitive step in the pipeline is also the least documented.
The numbers matter for the risk calculus. At ChatGPT's reported scale, even a redaction system with a very low miss rate produces a large absolute volume of leaked identifiers, because the denominator is enormous. A 0.1% miss rate across a few hundred thousand sampled conversations is still hundreds of conversations containing something a person would not want a stranger reading. That arithmetic is why disclosure language matters more than the miss rate itself -- users can consent to a documented process, and cannot consent to one they do not know exists.
What This Means For Founders
For founders building on top of the consumer APIs, the practical read is about contract hygiene rather than outrage. Enterprise and API traffic is governed by different retention and training terms than the consumer product, and the gap between those tiers is now a thing your customers' security teams will ask about by name. If your app pipes end-user content through a consumer-tier key, or if your DPA does not specify whether human review applies, that is a question you want answered before a prospect asks it during procurement.
The counterweight: human review is also how the sycophancy problem got fixed. OpenAI rolled back an overly agreeable GPT-4o update in April 2025 precisely because human raters flagged the behavior, and the models that people trust with medical and legal questions are safer because annotators marked the confident wrong answers. Removing the humans does not produce a more private product; it produces a worse one. The honest framing is a tradeoff between two goods, not a scandal with a clean villain.
There is also a labor dimension the coverage tends to skip. Content review at this scale has a documented psychological cost -- the Facebook moderator settlements of 2020 established that -- and AI annotation work sits in the same category when the queue includes crisis conversations. Pulse has covered OpenAI's governance turbulence through the year; the contractor tier is the part of the org chart that never shows up in the S-1 discussion.
The Regulatory Collision
The disclosure now collides with active law. California's newly enacted chatbot statute, negotiated directly with OpenAI, sets duties around how these systems handle vulnerable users, and any statute that regulates chatbot behavior implicitly regulates the feedback loop that produces that behavior. Expect the next version of this fight to be about whether human review of consumer conversations requires an affirmative, in-product disclosure rather than a paragraph in a policy page.
