Hundreds of contractors read ChatGPT chats behind a filter tuned for identifier fields

A 404 Media investigation published September 14th reported that OpenAI is using hundreds of outside contractors to read and evaluate real ChatGPT conversations, including chats containing sensitive personal information.

OpenAI says it runs conversations through its Privacy Filter before contractors receive them; the filter is built to detect and mask names, contact details, addresses, account numbers, passwords and other identifiers.

OpenAI's own technical description of Privacy Filter says the model can miss uncommon identifiers or ambiguous private references and can over-redact or under-redact text when context is limited.

Reviewers are instructed to flag sycophancy, forced mimicry, patronizing assumptions, claims of personal experience and responses implying the model has human emotions, and to penalize clutter, "AI-speak" and inappropriate emoji use.

The FAQ notice about authorized personnel and service providers viewing user data has been online since at least 2023 and has only changed slightly over the years.

404 Media argues the disclosure is too hidden and too vague to make users aware of how much human review actually happens.

When 404 Media asked where users are explicitly told about human review, OpenAI initially didn't respond.

Anthropic confirmed to 404 Media that it uses human reviewers to check Claude responses and improve future models for users who have "Help improve our AI models" turned on, and anonymizes conversations before human review by stripping account details like email addresses.

Google uses human reviewers too and notes in Gemini that humans may review saved chats.

Users can opt out by opening ChatGPT's settings, selecting Data Controls and disabling "Improve the model for everyone"; the change prevents new conversations from being used for training.

The "Improve the model for everyone" setting is enabled by default and only applies to new conversations.

One reviewer told 404 Media he didn't think users knew that humans were reading their chats, and some reviewed conversations included users explicitly asking ChatGPT to keep the contents private.

Internal documents identify the work by the codename "Project Lily" and give reviewers access to anonymized prompts and, in some cases, entire conversations; usernames are removed but the text itself can expose personal details.

Some Project Lily tasks include a "user memories summary" describing previous interactions, location information and other context ChatGPT has retained about the user.

ChatGPT's usefulness increasingly depends on context while OpenAI's privacy protections depend on removing enough context to prevent identification; memory summaries make that tension explicit, because the same accumulated detail that helps personalize an answer can make a supposedly anonymized conversation easier to connect to a real person.

The contract workers are recruited through Crossing Hurdles and paid through AI training company Mercor, and one North America-based reviewer said they earn more than $50 an hour.

OpenAI offers a temporary chat mode that, according to the company, does not use input data for model training.

The documents do not establish how many users' chats were reviewed, how often the Privacy Filter failed, or whether a contractor identified any user.

Because the setting ships enabled and the opt-out applies only to new conversations, every conversation typed before a user disables it was already collected under the training setting.

The documents identify the training effort only as Project Lily and do not name the model being evaluated or establish whether the results are for a current or future system.

Each response receives a score from one to seven, where one represents an unusable answer and seven means the reviewer believes the response would be difficult to improve meaningfully; contractors grade for accuracy, usefulness, length, style and tone matching.