Big Tech’s Unwitting Role in the Spread of AI-Generated Nonconsensual ‘Nude’ Images

To review this article again, go to My Profile and then select View saved stories.

Damaging 'Nudify' Platforms Utilized Sign-In Services from Google, Apple, and Discord

Key tech enterprises such as Google, Apple, and Discord have facilitated access for users to malicious “strip” platforms. These sites deploy artificial intelligence to digitally strip individuals in photographs, creating images where the subjects seem to be unclothed without their approval. For several months, over a dozen of these deepfake portals have incorporated sign-in options provided by these technology giants.

An investigation by WIRED revealed that 16 major websites, known for offering services to alter and create nude images, utilize sign-in systems provided by major companies like Google, Apple, Discord, Twitter, Patreon, and Line. This method facilitates the account creation process on these deepfake platforms, lending them a semblance of legitimacy, prior to users purchasing credits to create images.

For years, platforms and automated systems capable of producing unconsented explicit photos of women and girls have been around, but their prevalence has surged following the advent of generative artificial intelligence. This type of violation, often involving the creation of explicit images without consent, has become disturbingly common, including reports of teenage boys generating such images of their peers. Critics argue that technology firms have not acted swiftly enough to address the magnitude of these problems, noting that these sites rank prominently in search engine results, are boosted by paid ads on social networks, and their corresponding applications are easily found in app marketplaces.

"Adam Dodge, a legal professional and the initiator of EndTAB (Ending Technology-Enabled Abuse), points out that this trend perpetuates the acceptance of sexual violence against females by major technology companies. He mentions, “Sign-in APIs are essentially facilitators for ease of use. However, the idea of simplifying acts of sexual violence is unacceptable,” he emphasizes. “Our approach should be to restrict access to these applications, but what's happening is that we're essentially laying out a welcoming bridge for perpetrators.”

The authentication technologies examined by WIRED, implemented via APIs and standard sign-in processes, enable users to access deepfake sites using their existing accounts. Google's sign-in mechanism was found on 16 sites, Discord's on 13, and Apple's on six. The sign-in option for X was present on three sites, while both Patreon and the messaging platform Line were featured on two of the same sites.

WIRED has chosen not to disclose the names of the specific websites involved, as they facilitate abusive behaviors. Many belong to larger conglomerates, under the ownership of either individual proprietors or corporate entities. These platforms continue to implement login mechanisms in violation of the overarching policies set by tech firms, which explicitly prohibit the utilization of their platforms in a manner that could lead to harm, harassment, or the infringement of privacy rights.

Upon reaching out to WIRED, representatives from Discord and Apple confirmed they have deactivated the developer accounts linked to their platforms. Google mentioned it would intervene with developers upon discovering any breach of its guidelines. Patreon declared it forbids any accounts involved in the production of explicit content, while Line acknowledged it is conducting an investigation but refrained from discussing individual websites. X remained silent in response to inquiries regarding the misuse of its systems.

Following the statement made to WIRED by Jud Hoffman, the Vice President of Trust and Safety at Discord, about the company cutting off API access for certain websites due to a breach of its developer policy, one of the affected websites announced in a Telegram group that its Discord-based authorization feature was "temporarily unavailable." The website also mentioned efforts to regain access. This particular service failed to reply to WIRED's inquiries regarding its activities.

Swift Growth

From its inception in late 2017, deepfake technology has led to a surge in the production of unsolicited explicit content. Although generating videos requires more effort, the proliferation of websites and applications designed to digitally disrobe or create nude versions of images is now widespread.

David Chiu, the city attorney of San Francisco, has firmly stated that the actions of certain websites and their creators should not be mistaken for innovation but should be recognized as sexual abuse. Chiu has initiated legal action against websites that offer services to undress and create nude images of individuals without their consent. He points out that the lawsuit his office has filed is directed at 16 websites, which collectively have received approximately 200 million visits in just the first half of this year. Chiu accuses these platforms of participating in the grievous exploitation of women and girls worldwide, claiming that the images generated are being used to intimidate, degrade, and threaten females.

Websites that facilitate undressing function as clandestine enterprises, deliberately disclosing minimal information regarding their ownership and operational mechanisms. Sites managed by identical proprietors frequently exhibit a uniform appearance and share strikingly similar terms of service. Their multilingual offerings, available in over twelve languages, underscore the global scale of this issue. Additionally, associated Telegram channels attract vast audiences, with some boasting memberships in the tens of thousands.

The online platforms are continuously evolving, regularly updating users about the latest advancements they're working on. One such advancement includes an AI capability that purportedly can alter the appearance of women's bodies and even supports importing images from Instagram. Typically, these sites levy fees for creating images and may offer referral programs to incentivize users to disseminate their content. Furthermore, some of these platforms have banded together to form a group with the aim of launching a unique cryptocurrency, which could be utilized as a payment method for acquiring images.

An individual going by the name Alexander August and holding the position of CEO for a certain website, communicated with WIRED to express their recognition and understanding of the issues related to the possible abuse of their technology. This individual stated that their platform has implemented several protective measures to block the generation of images involving minors. "Our team is dedicated to upholding our social obligations and is eager to work with regulatory agencies to improve the openness, security, and dependability of what we offer," they mentioned in an email.

Tech company sign-in options frequently appear during the account creation process or when users interact with features like image generation on websites. The exact number of users who opt for these login options remains uncertain, as many sites also offer the alternative to register using only an email address. Nonetheless, upon reviewing various websites, it was observed that most have integrated the login APIs from multiple tech companies, with Google's Sign-In being the most popular choice. Selecting this option triggers notifications from Google, informing users that the website will access their name, email address, language settings, and profile photo.

Google's authentication process can expose certain details regarding the developer accounts associated with a website. For instance, one Gmail account is connected to four websites, while a different account is connected to six. A representative from Google stated, "Developers are required to comply with our Terms of Service to implement Google Sign-In, which includes a ban on distributing sexually explicit materials and any actions or materials that slander or intimidate others." The spokesperson also noted that Google will enforce necessary measures if these guidelines are violated.

Following inquiries from WIRED, various technology firms with sign-in services reported that they have deactivated accounts.

Hoffman, representing Discord, has stated that in addition to dealing with the sites highlighted by WIRED, the platform will also tackle any other sites they identify that breach their guidelines. Shane Bauer, a spokesperson for Apple, mentioned that the company has revoked the licenses of several developers, rendering Sign In With Apple ineffective on those developers' websites. Adiya Taylor, who leads corporate communications at Patreon, emphasized that the platform bans any accounts that support or finance the use of external applications for generating adult content or explicit images. Taylor added, “We will act against any content or accounts on Patreon found to be contravening our Community Guidelines.”

In addition to their login mechanisms, numerous websites showcased Mastercard or Visa logos, suggesting that these payment methods might be accepted for their offerings. Visa did not reply to WIRED's inquiry for a statement, but a representative from Mastercard stated, “transactions involving nonconsensual deepfake content are prohibited on our network,” highlighting that they intervene whenever such activities are identified or brought to their attention.

Several times, technology firms and financial services have intervened against AI platforms that enable the creation of unauthorized images or videos, following journalistic investigations into their operations. Clare McGlynn, a legal scholar at Durham University with specialization in the laws governing pornography, as well as online sexual violence and abuse, points out that major technology companies contribute to the expansion of websites that undress individuals and other similar sites by failing to actively oppose them.

McGlynn expresses alarm over the lack of fundamental security measures and oversight, criticizing the current sign-in procedures as insufficient. He emphasizes that companies' responses are severely lacking, only taking action when issues are pointed out by journalists or activists. This, according to McGlynn, clearly shows a disregard for the problem, contrary to what they claim. "If they truly cared, they would have implemented these basic solutions to limit access," McGlynn argues.

Suggested for You…

Direct to your email: A selection of our top stories, curated daily just for you.

A flawed update from CrowdStrike brings global computer systems to a halt

The Major Narrative: When could the Atlantic Ocean possibly fracture?

Introducing the age of excessive online consumption

Additional Content by WIRED

Critiques and Manuals

© 2024 Condé Nast. All rights reserved. Purchases made through our website may result in WIRED receiving a share of the sales, as part of our affiliate agreements with retail partners. Reproduction, distribution, transmission, caching, or any other form of use of material from this site is strictly prohibited without prior written consent from Condé Nast. Ad Choices

Choose a global website