As anyone who so much as glanced at the internet in the past few weeks probably noticed, Google’s sweeping AI upgrade to its search engine had a rocky start. Within days of the company launching AI-generated answers to search queries called AI Overviews, the feature was widely mocked for producing wrong and sometimes bonkers answers, like recommendations to eat rocks or make pizza with glue.
New data from search engine optimization firm BrightEdge suggests that Google has significantly reduced how often it is showing people AI Overviews since the feature launched, and had in fact already substantially curbed the feature prior to the outpouring of criticism. The company has been tracking the appearance of Google’s AI answers on results for a list of tens of thousands of sample searches since the feature was first offered as a beta test last year.
When AI Overviews rolled out to logged-in US users in English after Google’s I/O conference on May 14, BrightEdge saw the AI-generated answers on just under 27 percent of queries it tracked. But their presence dropped precipitously a few days later, the week before screenshots of AI Overviews’ errors went viral online. By the end of last week, when Google published a blog post acknowledging its AI feature’s flubs, BrightEdge saw AI Overviews appearing on only 11 percent of search result pages. Their prevalence was essentially the same on Monday.
Jim Yu, BrightEdge’s founder and executive chairman, says the drop-off suggests that Google has decided to take an increasingly cautious approach to this rollout. “There’s obviously some risks they’re trying to tightly manage,” he says. But Yu adds that he’s generally optimistic about how Google is approaching AI Overviews, and sees these early problems as a “blip” rather than a feature.
“We’re continuing to refine when and how we show AI Overviews so they’re as useful as possible, including a number of technical updates in the past week to improve response quality,” says Google spokesperson Ned Adriance. Google declined to share its internal statistics about how frequently AI Overviews appear in search, but Adriance says that the BrightEdge numbers don’t reflect what the company sees internally.
It’s unclear why Google may have decided to significantly reduce the appearance of AI Overviews shortly after it launched, but the company’s blog post last week acknowledged that having millions of people use the feature provided new data on its performance and errors. The company’s head of search, Liz Reid, said Google had made “more than a dozen technical improvements,” like limiting satirical content from cropping up in its results. Her post noted that these changes would trigger restrictions on when AI Overviews were offered but did not detail how exactly those restrictions would change the frequency with which AI results appeared.
BrightEdge began tracking AI Overviews using its list of sample queries after Google allowed users to opt in to a beta test of the feature late last year. The test queries spanned nine categories, including ecommerce, insurance, and education, and were designed to span common but also rarer searches. They were tested over and over, in some cases multiple times a day.
In December 2023, BrightEdge found that the summaries appeared on 84 percent of its searches but saw that figure drop over time. Google’s Adriance said it did not trigger AI Overviews automatically on 84 percent of searches but did not clarify its internal measurements. After Google opened up AI Overviews to all, BrightEdge continued tracking their appearance using a mixture of accounts that had previously enrolled in the beta test and others which had not but saw no significant difference between what the two groups saw.
Google declined to share exactly how much it changed how many AI Overviews it showed the general public versus people enrolled in the beta test, but Adriance said that people who had opted in to the test were shown AI Overviews on a wider range of queries.
BrightEdge’s data also sheds light on the topics where Google believes AI Overviews can be most helpful. AI answers appeared on the majority of health care keyword searches, most recently at a frequency of 63 percent. Sample queries included in BrightEdge’s data included “foot infection,” “bleeding bowel,” and “telehealth urgent care.” In comparison, queries about ecommerce return AI Overviews at around 23 percent, while restaurants or travel very rarely trigger AI overview answers.