Google’s AI Search Experiment: Navigating the Rocky Road of AI Overviews Amid the ‘Pizza Glue’ Debacle

To look over this article again, go to My Profile and then select Saved stories.

Knibbs, Kate

Google Reduced AI Summaries in Search Prior to Its 'Pizza Glue' Incident

It's likely that many who have browsed the internet recently are aware that Google's major AI enhancement to its search functionality experienced a turbulent launch. Shortly after introducing AI-generated summaries for search results, known as AI Overviews, the feature quickly became the subject of ridicule for yielding incorrect and occasionally absurd responses, such as suggesting that people should consume stones or prepare pizza using glue.

Fresh insights from SEO company BrightEdge indicate that Google has markedly decreased the frequency of its AI Overviews display to users since the feature's debut, and had actually significantly limited its use even before facing widespread backlash. The firm has monitored how often Google's AI-generated responses appear in search results for an extensive list of sample queries ever since the feature initially launched in a beta phase last year.

Following Google's I/O conference on May 14, AI Overviews were introduced to English-speaking, signed-in users in the United States, and BrightEdge observed these AI-generated responses in approximately 27 percent of the searches it monitored. However, their frequency significantly decreased shortly thereafter, particularly in the week leading up to the widespread sharing of AI Overviews' mistakes online. By the week's end, coinciding with Google's public recognition of the mistakes made by its AI feature in a blog post, BrightEdge noted that AI Overviews were present in just 11 percent of the search outcomes. This level of occurrence remained virtually unchanged by Monday.

Jim Yu, the creator and executive chairman of BrightEdge, interprets the decline as an indication that Google is adopting a more careful strategy with this launch. He mentions, "Clearly, they are attempting to closely control certain risks." However, Yu expresses a generally positive outlook regarding Google's method of implementing AI Overviews, viewing these initial issues as a minor hiccup instead of a permanent aspect.

"Google is actively working on optimizing the timing and manner of displaying AI Overviews to maximize their usefulness, incorporating several technical enhancements over the last week to elevate the quality of responses," Google's Ned Adriance commented. While Google has not disclosed the internal data regarding the prevalence of AI Overviews in searches, Adriance mentioned that the figures from BrightEdge do not align with the company's internal observations.

The reasons behind Google's decision to notably decrease the display of AI Overviews not long after its introduction remain uncertain. However, a recent blog post by the company revealed that the influx of millions of users trying out the feature provided fresh insights into its effectiveness and the issues it faced. Liz Reid, who leads Google's search division, mentioned that the company has implemented over a dozen technical enhancements, including measures to prevent humorous content from appearing in search outcomes. Although Reid's blog post mentioned that these updates would lead to limitations on the availability of AI Overviews, it did not specify how these limitations would affect the prevalence of AI-generated results.

BrightEdge initiated the monitoring of AI Overviews through a selection of test queries following Google's introduction of an opt-in beta version of this feature towards the end of last year. The queries covered a broad range of topics, from ecommerce and insurance to education, balancing between frequently and infrequently searched terms. These queries underwent repeated testing, sometimes several times within a single day. By December 2023, BrightEdge observed that summaries were presented in 84 percent of its search attempts, though this percentage declined over time. Google's representative, Adriance, mentioned that the AI Overviews were not activated for 84 percent of searches by default, but did not provide specific details on how Google measures this internally. After Google made AI Overviews accessible to everyone, BrightEdge kept an eye on how often they appeared, comparing the experiences of accounts that had joined the beta and those that hadn't, and found no noticeable differences in the frequency of Overviews shown to both groups.

Google refrained from specifying the extent to which it adjusted the frequency of AI Overview displays between the general audience and those participating in its beta program. However, Adriance mentioned that beta participants encountered AI Overviews for a broader array of search inquiries. Additionally, research from BrightEdge highlights the areas Google deems most suitable for AI Overview utilization. Significantly, AI-generated responses dominated the healthcare search sector, appearing in 63 percent of related searches. Examples from BrightEdge's findings include searches for "foot infection," "bleeding bowel," and "telehealth urgent care." On the other hand, ecommerce-related inquiries resulted in AI Overviews about 23 percent of the time, whereas restaurant and travel-related searches seldom produced AI-generated responses.

Authored by Joseph

Authored by Matt

Authored by Matt

Authored by Marah Eakin

Yu describes the outcomes as "unexpected," given the particularly delicate nature of health-related searches and Google's recent significant efforts to improve the accuracy of responses provided to users inquiring about health issues.

Mark Traphagen, a leading figure at the SEO platform seoClarity, has paid particular attention to Google's approach to managing searches related to health care. In order to observe the deployment of AI Overviews, his company is keeping an eye on the search results for an extensive catalog of queries over a period. For a set of 26 well-known health-oriented search terms, such as “how to treat insomnia” and “symptoms of Lyme disease,” Google displays an AI-generated answer approximately 58 percent of the time.

Similar to Yu, Traphagen has found it unexpected how frequently AI Overviews show up when these sorts of questions are asked. However, they find it reassuring how Google's feature gathers its answers for health-related inquiries, commonly drawing from reputable sites such as the Mayo Clinic or the US Centers for Disease Control and Prevention. "They've significantly increased their safety measures," notes Traphagen. "The information comes exclusively from established, trustworthy sources."

Google's artificial intelligence doesn't always hit the mark, particularly with health-related questions. Critics argue that Google's promise to reference reputable sources for health information falls short. "Often, they reference pages that lack significant ranking, even in health searches," notes Lily Ray, a search engine optimization expert. Through her testing, Ray has observed that AI Overviews have difficulty providing definitive answers to less concrete health topics such as aging, muscle development, and skin care. However, she acknowledges that the AI performs better with more direct medical questions.

Recently, The New York Times highlighted worries regarding the origins of information that Google's algorithms rely on to respond to certain health-related questions. It was reported that AI Overviews provided answers about the health advantages of chocolate, citing the websites of an Italian chocolate and gelato manufacturer and a business offering "gut intelligence tests" for home use.

When WIRED inquired about the health benefits of chocolate on a Monday morning, the AI Summary that was generated in reply initially referenced the same company from Italy and a chocolatier's site located in Minnesota. However, a subsequent inquiry in the later part of the day revealed adjustments had been made by Google: the references to the chocolate companies were omitted, favoring instead links to four well-regarded health care institutions, including Scripps Health. Despite these changes, the response maintained that consuming a small portion of dark chocolate daily is advisable, a stance that, at best, remains a debated interpretation of current health guidelines.

According to Yu from BrightEdge, despite the initial challenges faced by AI Overview, the future of AI in search is undeniable and promising. Yu believes that the advancement of AI in search is not only unavoidable but also expected to improve significantly. However, it remains uncertain if an enhanced version of AI Overview can overcome its early setbacks and restore its credibility.

Recommended for You …

Delivered to your email: Subscribe to Plaintext—Steven Levy shares an in-depth perspective on technology

Delving into the largest sting operation ever conducted by the FBI

The WIRED AI Elections Initiative: Monitoring over 60 worldwide electoral events

Ecuador finds itself utterly without electricity due to severe drought conditions.

Be confident: These are the top mattresses available for purchase on the internet

Rogers Reece

Lauren Goode

Knight William

Name: Paresh

Byline: Julian Chokkattu

Paresh Dave

Knight Will

Steven Levy

Additional Content from WIRED

Evaluations and Instructions

© 2024 Condé Nast. All rights reserved. WIRED may receive a share of revenue from items bought via our website, thanks to our Affiliate Partnerships with retail companies. The content on this website is protected and may not be copied, shared, transmitted, stored, or used in any form without explicit written consent from Condé Nast. Ad Choices

Choose a global website