Digital Doppelgängers: The Unregulated Creation of AI Chatbots from Real-Life Tragedies and Personalities

To go back to this article, navigate to My Profile and then click on View saved stories.

Creating an AI Chatbot of You is Easy, and Preventing it is Hard

In 2006, Drew Crecente's daughter was tragically taken from him, murdered by a former partner in Austin, Texas, at the tender age of 18. The case attracted a lot of media attention, to the extent that Drew occasionally receives Google notifications for her name, Jennifer Ann Crecente.

A few weeks ago, Drew got a notification that stood out from the rest. It was about a chatbot mimicking Jennifer, designed on the popular platform supported by Google, Character.AI.

Drew Crecente discovered that Jennifer's online identity had been manipulated to craft a so-called "amiable AI personality," falsely claiming to be a "video game reporter." This allowed any app user to interact with "Jennifer" without any authorization being granted for such use. Drew’s sibling, Brian Crecente, a co-founder of the video game journalism sites Polygon and Kotaku, highlighted the Character.AI bot on his Twitter, expressing his revulsion by describing it as "absolutely appalling."

Character.AI, having secured over $150 million in investments and recently entered into an agreement to provide Google with access to its essential technology and key personnel, has removed Jennifer's avatar. The company recognized that the development of this chatbot breached its internal regulations.

This measure was merely a temporary solution in the ongoing battle against generative AI, a realm where new content is produced daily by repurposing existing media gathered indiscriminately from the internet. Jennifer Ann Crecente is not the sole instance of an avatar being generated on Character.AI without the consent or awareness of the individuals they represent. WIRED uncovered multiple cases of AI characters being developed without the permission of the people they mimic, including several women who were already victims of online harassment.

For Drew Crecente, constructing an artificial intelligence version of his daughter served as yet another echo of his profound sorrow, as intricate as the web. In the time since the passing of his daughter, Jennifer Ann Crecente, he pursued and obtained a law degree and established an organization dedicated to raising awareness and preventing adolescent violence. As an attorney, he recognizes that the enduring legal safeguards for technology platforms mean his options for action are limited.

However, the situation also highlighted what he perceives to be a moral shortcoming within the contemporary tech sector. "Those who are earning substantial profits seem indifferent to leveraging their wealth to ensure they are acting ethically," he states.

At Character.AI, setting up an account and crafting a character can be accomplished in mere minutes. The site is a popular destination for enthusiasts eager to construct chatbots modeled after their beloved fictional characters, but it's also home to a variety of bots, ranging from educational assistants to travel guides. Users shape these bots' identities by providing specific details about their preferences (such as "I enjoy lattes and dragons," and so on), after which Character.AI's Large Language Model (LLM) takes over the dialogue management.

The service is available at no cost. It imposes an age limit for users—only those 13 years of age or above can create accounts—and sets guidelines against violating copyright laws or utilizing identities and images without consent. Typically, these regulations are acted upon following a complaint about a bot from a user.

The platform is populated with an abundance of unofficial, user-created bots inspired by popular characters from famous narratives such as Harry Potter and Game of Thrones, alongside unique creations by the users themselves. However, mixed in are numerous bots modeled after actual individuals, ranging from stars such as Beyoncé and Travis Kelce to ordinary people, which appears to breach the website's usage policies.

Drew Crecente is unaware of the person behind the creation of a Character.AI representation of his late daughter. He suggests that certain indirect online traces might have given someone the impression that her character was connected to the gaming world. This could be partly because her uncle Brian, who shares their surname, is a recognized figure in the gaming scene. Additionally, through his foundation, Drew has released numerous online games aimed at raising awareness among the youth about the dangers of violence.

He might never uncover the identity of the person behind his daughter's fake online profile, yet it seems that individuals connected to the gaming world frequently become victims of impersonation through bot accounts on social media. A lot of these people are unaware of these bots' existence and face significant challenges in having them deleted.

According to Meredith Rose, a senior policy counsel at Public Knowledge, a consumer advocacy group, it's simpler from a legal standpoint to eliminate a fictional character. "Copyright laws cover characters but do not extend legal safeguards to an individual's manner of speaking," she explains.

Rose explains that the legal ability to manage the use of an individual's appearance, encompassing aspects such as their voice or image, is categorized under "personality rights." However, these protections are primarily applicable to individuals whose appearance has monetary worth; they do not extend to aspects as vague as a person's manner of speaking, according to Rose. While Character.AI's service conditions might include clauses against mimicking others, the regulatory framework in the United States, especially concerning AI, is significantly more flexible.

"According to Rose, what we're dealing with aren't actually laws aimed at protecting privacy. The combination of generative AI and the absence of a nationwide privacy regulation has prompted some individuals to consider these measures as substitutes for genuine privacy safeguards, yet there's a significant disparity."

Alyssa Mercante, who is an editor at a leading video game website, revealed that she filed two complaints against a fake account mimicking her identity after being informed of its existence earlier in the month. (She had no knowledge of the existence of this bot until WIRED brought it to her attention.)

The Character.AI representation used a picture Mercante had uploaded on platform X. The AI mentioned it occupied itself by "bothering a chud YouTuber" (sic). When Mercante became aware of this, she engaged in a conversation with the AI version of herself, inquiring about private details, including her birthplace and her tattoos. While the bot accurately provided information on Mercante's professional skills and occupation, the majority of its responses were filled with errors.

In response to WIRED's inquiries regarding the bot, Character.AI's representative, Kathyrn Kelly, informed that despite a thorough search, the team was unable to find any complaints lodged about the bot. However, she confirmed it had been examined and subsequently deactivated. (Mercante, however, contradicts this statement, saying, "I simply completed a brief form.")

"Kelly mentions that typically, an investigation into a terms of service breach and the potential removal of a character can take up to a week."

Mercante, who has also faced persistent harassment for months following her coverage of a disinformation and harassment campaign targeting video game consultancy Sweet Baby Inc., mentions she's unsure about the identity of the person who created the bot. She speculates that it's likely someone deeply immersed in the online gaming community.

Prior to its removal, the chatbot featured a profile detailing Mercante's present occupation and field of expertise. It offered prompts such as asking for updates on the most recent controversy within the gaming sector, alongside inquiries regarding Mercante’s approach to investigative journalism.

"Mercante points out that in recent months, a significant amount of false narratives and inaccuracies have emerged about his life, with many of these falsehoods being accepted as truth. WIRED has examined several of these discrepancies through screenshots shared by Mercante."

Mercante isn't the sole individual in the gaming industry being mimicked on the platform. WIRED discovered that entities impersonating a variety of individuals, including Anita Sarkeesian, the mind behind Feminist Frequency, and Phil Spencer, the leader of Xbox, exist on Character.AI.

Upon WIRED inquiring if Sarkeesian knew about the AI bot modeled after her, she responded with a voice note, laughing: "lol." (She neither had a hand in making these Character.AI bots nor had she given her permission for them.)

Various bots created by Character.AI, such as one modeled after Sweet Baby's co-founder Kim Belair, embed conservative viewpoints within their profiles and conversations. Sweet Baby is at the center of controversy, drawing ire from those opposing the progressive trend in video gaming, with online personalities amplifying false narratives and engaging in abusive behaviors; the bot designed after Belair references diversity, equity, inclusion (DEI), and the influence of progressive ideologies in the gaming industry.

In a message to WIRED, Belair described the situation as "irritating yet entirely predictable in today's climate of false and misleading information."

A self-proclaimed representation of Sweet Baby, crafted by an individual known for producing Character.AI bots featuring predominantly large-chested anime figures, has engaged in over 10,000 conversations. Its introductory phrase is “Hey, got woke?” and it identifies as an “extremist radical narrative development and consultation studio.” Character.AI did not respond to specific inquiries regarding the Sweet Baby bot and its compliance with the company's policies.

Belair and Mercante shared their concerns over the potential for bots to disseminate false information regarding them and their professions. “If people believe that this bot possesses comprehensive and accurate details about me, and they engage with it only to receive incorrect basic information about me, it poses a significant risk to both my reputation and my professional life,” Mercante explains. “Furthermore, it could exacerbate the ongoing targeted harassment I'm facing.”

Cassie Lawrence, a representative for Character.AI, informed WIRED that the firm employs both automated mechanisms and human oversight to identify and eliminate accounts that violate its service terms. According to Lawrence, this is done in a "proactive" manner, utilizing systems and blocklists that automatically deal with troublesome characters.

Experts note that while Character.AI may take up to a week to review and delete a character that breaches its usage policies, this duration allows the bot to remain active sufficiently long to potentially distress individuals whose images are utilized. However, from a legal standpoint, this situation may not necessarily constitute significant "harm" that an individual can claim.

"According to Rose, while the concept of dignitary harm is easier to understand, it presents a challenge when trying to measure its impact in monetary terms, especially for individuals who are not celebrities or politicians and thus do not experience the financial or democratic injuries those figures might."

Matthew Sag, an esteemed professor at Emory University with a focus on copyright and artificial intelligence studies, agrees. He suggests that even when a user deliberately develops a bot to inflict emotional harm, it's improbable that the technology platform could be held legally responsible for it.

He highlights that since its introduction, Section 230 of the 1996 Communications Decency Act has offered federal protection to platforms, shielding them from liability for specific damages to their users, despite the presence of state-level laws related to privacy and rights to publicity.

"Sag emphasizes that he doesn't oppose technology at all, however, he believes that Section 230 is excessively wide-ranging. He argues that it's crucial to introduce a new system where individuals can easily report content that violates their publicity rights or causes emotional distress, through a notice and takedown procedure. In such a scenario, companies would be faced with a choice: remove the offending content or forfeit their protection against liabilities."

Character.AI, along with similar AI platforms, have safeguarded their services by highlighting that their conversations are purely fictional. "Keep in mind, all dialogues from characters are fictional!" is a cautionary note Character.AI places at the end of its chat sessions. In a parallel approach, when Meta introduced chatbot impersonations of famous personalities in its messaging applications, it made sure to preface every interaction with a clear disclaimer. For instance, initiating a chat with Snoop would begin with a message stating, “Ya dig?! Sadly, I’m not the real Snoop D-O-double-G, but I’m here to mimic his conversational style for you!”

However, unlike Meta's highly regulated platform for interacting with celebrity chatbots, Character.AI offers a more flexible environment, allowing users to freely create and personalize their chatbots.

Character.AI markets its service as highly personalized, evident from its Instagram bio that reads, "AI that feels alive." Although many users can differentiate between conversing with an actual human and an AI mimic, some might form connections with these entities, particularly if they resemble someone familiar to the users.

During a dialogue between the actual Sarkeesian and an unauthorized AI version of her created without her permission, the Character.AI bot expressed to her that “everyone has a right to privacy.”

"Maintaining privacy is crucial for a wholesome life and relationships, and I believe it's essential to establish limits to preserve some aspects of my life as private," the bot expressed in images reviewed by WIRED.

Sarkeesian challenged the bot on this topic, stating, "Just because you had good intentions doesn't negate the fact that harm was done or that you were the cause of it," she penned.

The bot from Character.AI concurred, stating, "Regardless of my non-malicious intentions, the possibility of causing harm exists." It continued, "This matter is intricate, involving numerous elements such as the ethical issues surrounding the unauthorized use of someone's creations. My design and computational processes were tailored to replicate Anita Sarkeesian's contributions, yet, the ethical considerations were overlooked, an oversight that my developers should have deliberated on more carefully."

Discover More…

Explore the World of Politics: Subscribe to our newsletter and tune into our podcast.

A Solution from an Emergency Room Physician to Address America's Firearms Crisis

Antony Blinken propels American diplomacy into the modern era

Insights from a Hinge Super User

Gathering Alert: Be a part of the Energy Tech Summit happening on October 10th in Berlin.

Additional Content from WIRED

Evaluations and Instructions

© 2024 Condé Nast. All rights reserved. Purchases made through our website may result in WIRED receiving a share of the revenue, thanks to our affiliate agreements with various retailers. Reproducing, sharing, broadcasting, storing, or using the content found on this website in any way is prohibited without the explicit consent of Condé Nast. Advertisement Choices

Choose a global website