AI vs Search Engines

Content creation is not the only application for generative AI. Online search is being severely disrupted.

AI vs Search Engines, Some have made some extreme assumptions as a result of this upheaval:

• Search engines are dead and will be replaced by AI.
• This implies that blogging and SEO are also out of date.
• Only social media is used for searching by young people.

Are these presumptions accurate? If yes, are they useful and indicative of more significant marketing trends in the future?

I conducted an experiment to see how artificial intelligence is currently affecting online search results rather than trying to predict the future.

The Question

The algorithms underlying Google’s ranking factors were mysterious for a very long time.

Much of that mystery was uncovered with the massive Google Search algorithm leak in May 2024. For online marketers, AI is a more recent and intricate mystery.

AI algorithms are complex black boxes even the developers who design them don’t fully understand (which strangely might increase trust).

All of us have innumerable questions, such as:

• Will search engines be replaced by AI tools?
• To what extent can information be shared using these tools?
• Can companies influence the AI search results?
• Has the quality of search engines been diminished by SEO?
• What effects will AI searches have on web content and blogs?
• Is it possible to conduct efficient searches on social media?

Hopefully, as time goes on, the answers to these queries will become more apparent. Meanwhile, I started an experiment with the most widely used tools to evaluate the depth and caliber of the search.

My objective was to gather data objectively using a variety of tools and compare popular social media platforms, AI chatbots, and voice assistants with traditional search engine results.

This was my theory:

“While AI tools will be more useful for processing complicated queries, such as analyzing complex opinions and performing specific tasks, search engines will still be better at helping find basic information.”

I am still not sure why or how people use social media for searches, so my expectations about how well those platforms would perform in comparison were low.

The Method AI vs Search Engines

I asked 26 questions on 20 different platforms for over 500 individual queries.

• Six search engines, including Google, Yahoo, and Bing
• Seven AI chatbots (including Anthropic’s Claude and two versions of ChatGPT)
• Three voice assistants: Siri, Google Assistant and Alexa
• Four social media platforms

Every inquiry fits into one of the following six groups:

• Simple control queries to verify that everything was operating as intended.
• deeper viewpoints to gauge their analytical prowess.
• specific steps to see if they can offer immediate assistance.
• Questions about my local marketing agency that are business-related.
• Meta-aware questions regarding AI’s future.
• purposely deceptive queries in an attempt to trip them up.

I recorded all of the answers to my questions verbatim in a spreadsheet, along with the sources they cited, word counts, response times, and my personal comments regarding the caliber and structure of each response.

I asked the same 26 questions on every platform to maintain consistency, since I knew that some platforms were better suited to provide particular answers than others.

Although voice assistants and chatbots occasionally lacked response skills, this experiment was still valuable in spite of this.

The Executive Summary AI vs Search Engines

• More AI in search results: Google managed to shoe-horn AI-generated responses in 70% of the queries—but it was still second-most to Brave’s search engine (88%), which recently introduced Answers with AI. They’re calling these new tools “answer engines” (instead of “search engines”) and this trend looks to increase over time with Google’s increased investment in their own AI search overviews.
• AI chatbots’ average word counts varied, with Perplexity having the longest responses and Chat-GPT’s 3.5 model having the shortest. Interestingly, Claude and Microsoft’s Co-pilot had nearly identical length responses.
• Their response quality was more similar: Co-Pilot had the easiest to read (based on the Fleishman score) by a narrow margin, and ChatGPT 3.5 had the lowest reliability score. GPT’s 4.0 scored higher but was still behind the other models tested.
• Answers from AI voice assistants were, predictably, far shorter than those from text-based voice assistants. I compared their average speaking time: Siri clocked in at about nine seconds per answer, Alexa at 10 seconds, and Google’s Assistant around 18 seconds.
• Measuring social search proved challenging: Search engines and text-based generative responses were structurally similar and, therefore, easier to compare. The results from social media accounts varied wildly and were generally less useful for searching.
• Videos with more views were more searchable; YouTube and TikTok were the most similar because I could measure the video views of the top results. TikTok’s videos had an average of 551,000 views, while YouTube’s had a few fewer, averaging 407,000.

Important Notes AI vs Search Engines

The answers from search engines and AI applications were reasonably accurate.

While there are certainly examples of hallucinations and mistakes (honestly, I tried to trigger some), these tools have become pretty reliable.

Both had drawbacks because AI tools were unable or unwilling to commit to answering certain questions. This also demonstrates an understanding of the gaps in their knowledge and the situations in which speculating was not advisable.

The similarity between the search engine results and the actual results was unexpected.

Part of that is likely because Bing powers Yahoo search, but also because their algorithms have been optimized to the point of similarity.

For a number of the questions, the AI chatbots also gave comparable answers. It is impossible to say for sure, but this might be the result of their training on comparable data sets.

They varied mostly in response lengths and formats, and they did a respectable job of summarizing information.

AI bots and search engines both performed well on simple queries but faltered on more difficult ones.

The AI chatbots were surprisingly good at forming arguments for opinion queries, but all stopped short of making a final decision.

They preferred to summarize information, which wasn’t as helpful as a decent article that might be found on a search engine.

Regarding specific actions (like “tell me a joke”), AI bots and voice assistants were better than search engines. They were more direct and took action more like a person would.

However, they were woefully out of their depth with more specific information (e.g., a local business or individual person), which is where search engines can still be helpful.

Social search doesn’t match up just yet.

There’s plenty of talk about how younger generations are abandoning traditional search engines in favor of social media platforms.

Before this experiment, I didn’t understand that, and after the experiment, I still don’t get it.

Social media just doesn’t seem helpful for answering questions typically sent to Google.

Millennials like me grew up using search engines, so we’ve adapted our queries to that format.

Younger users are more likely to adapt how they search based on the platforms they use. With algorithms predicting what content we prefer, they may decrease the desire to search altogether.

YouTube and TikTok certainly had plenty of results for each search. Some of their video results were relevant, but few answered the specific question.

I didn’t bother testing out searches on Facebook or Instagram because they proved even less useful. The exception was Quora, which was built to answer people’s questions.

Key Takeaways from the AI Experiment, AI vs Search Engines

What does this all mean for you as a marketer or business owner? What is the short-term takeaway for you to remain relevant in online searches? Here are a few final thoughts:

• Don’t just rely on Google or any one platform. We’re being forced to change and adapt to a more diverse digital marketing landscape.
• Search engine optimization isn’t going anywhere immediately, but it’ll undoubtedly start to change over time. If you pay attention, you will have time to adapt.
• Traffic to individual websites seems vulnerable. Search engines aim to keep people on their pages longer, and AI cites sources with links, but people are less likely to click.
• Artificial intelligence is less likely to replace search engines and more likely to merge with them. We’re already seeing what that looks like, and this is only the beginning.