Two review gates, one nudify app: Meta sold the ads, Apple carried the product

Meta platforms recently ran ads for an AI porn-generation tool that seemingly encouraged users to create deepfaked videos resembling female US politicians, despite the company's policies against ads containing sexual material; it is the latest in a series of failures by Meta to keep ads for tools producing nonconsensual intimate imagery off its platforms.

A voice-over for one of the video advertisements viewed by WIRED promotes the tool as having "no restrictions," says "all the characters are real people," and describes it as "the AI that men actually use."

In one ad, a woman closely resembling a prominent female US politician is seen in front of a US flag and the flag of the president of the US above the caption "What if she moved?" She then winks, before the ad cuts to a scene of a woman with the same face performing in a pornographic video.

The account tied to the campaign ran 32 ads, according to Meta, which ran for between five and 46 hours; most received 10 or fewer impressions.

The tool is called Kromix and bills itself as an "AI image styler."

Apple has struggled to keep similar tools, which violate its policies against nonconsensual intimate imagery, nudifying, and sexually explicit content, off the App Store.

Ads for nudify or undressing apps have been appearing on Meta's platforms frequently over the past few years, with researchers finding thousands the company had failed to detect.

Katie Paul, director of the Tech Transparency Project, which discovered the ads, says all the ads seen so far on the nudify issue are exclusively sexualizing and nudifying women and putting them into AI-generated porn, and that this raises questions about how Meta will deal with such content, including when it comes to female politicians, while the company is actively running ads containing it.

Politicians and elected officials, overwhelmingly women, have been a target of nonconsensual deepfake images and videos for years.

In January, as Grok was used to create around 3 million sexualized images of women and girls, officials in Sweden, England, and Northern Ireland were targeted by people using the technology, often using official images as a starting point.

Benjamin Shultz, lead researcher at the American Sunlight Project, says that as production and distribution of AI-generated content have democratized, formerly benign media such as official portraits and video from a press conference have become raw material for low- and no-code tools that anyone with an internet connection can use.

According to Meta's public documentation, each ad is "reviewed" against its policies before being published on platforms including Instagram and Facebook; the advertising standards say "Our ad review system relies primarily on automated tools to check ads and business assets against our policies," and that review starts before ads begin running.

The page running the advertisements was created on August 3, not long before the ads began running, and had no followers, two things that might have indicated to Meta that the account was suspicious.

Meta has a policy stating ads cannot contain sexual imagery, including nudity, sexual activity, or sexually suggestive content; its broader adult sexual exploitation policies say nonconsensual intimate imagery is not permitted and the promotion of "nudify" or "AI kissing" apps is not allowed.

Thirty-two ads were approved by a review system Meta says runs before ads go live and against policies that already prohibit promotion of nudify apps and sexual imagery.

With most of the 32 ads drawing 10 or fewer impressions each, the campaign's measured delivery was on the order of a few hundred impressions at most for the majority of its ads.

San Francisco's city attorney sent Apple a cease-and-desist letter earlier this summer demanding the removal of 13 specific apps and that it stop "aiding and abetting" the sale of explicit deepfake images.

The same product passed two independent pre-publication review processes, Meta's ad review and Apple's App Store listing review, and was pulled from both only after a journalist's inquiry.