Das Lernen mit KI ist im Vergleich zur altmodischen Websuche unzureichend. Wenn Menschen sich auf große Sprachmodelle verlassen, um Informationen zu einem Thema für sie zusammenzufassen, entwickeln sie im Vergleich zum Lernen über eine Standard-Google-Suche tendenziell ein oberflächlicheres Wissen darüber.
https://theconversation.com/learning-with-ai-falls-short-compared-to-old-fashioned-web-search-269760
21 Kommentare
I’ve linked to the news release in the post above. In this comment, for those interested, here’s the link to the peer reviewed journal article:
https://academic.oup.com/pnasnexus/article/4/10/pgaf316/8303888
From the linked article:
**Learning with AI falls short compared to old-fashioned web search**
Since the release of ChatGPT in late 2022, millions of people have started using large language models to access knowledge. And it’s easy to understand their appeal: Ask a question, get a polished synthesis and move on – it feels like effortless learning.
However, a new paper I co-authored offers experimental evidence that this ease may come at a cost: **When people rely on large language models to summarize information on a topic for them, they tend to develop shallower knowledge about it compared to learning through a standard Google search.**
Co-author Jin Ho Yun and I, both professors of marketing, reported this finding in a paper based on seven studies with more than 10,000 participants. Most of the studies used the same basic paradigm: Participants were asked to learn about a topic – such as how to grow a vegetable garden – and were randomly assigned to do so by using either an LLM like ChatGPT or the “old-fashioned way,” by navigating links using a standard Google search.
No restrictions were put on how they used the tools; they could search on Google as long as they wanted and could continue to prompt ChatGPT if they felt they wanted more information. Once they completed their research, they were then asked to write advice to a friend on the topic based on what they learned.
The data revealed a consistent pattern: People who learned about a topic through an LLM versus web search felt that they learned less, invested less effort in subsequently writing their advice, and ultimately wrote advice that was shorter, less factual and more generic. In turn, when this advice was presented to an independent sample of readers, who were unaware of which tool had been used to learn about the topic, they found the advice to be less informative, less helpful, and they were less likely to adopt it.
We found these differences to be robust across a variety of contexts. For example, one possible reason LLM users wrote briefer and more generic advice is simply that the LLM results exposed users to less eclectic information than the Google results. To control for this possibility, we conducted an experiment where participants were exposed to an identical set of facts in the results of their Google and ChatGPT searches. Likewise, in another experiment we held constant the search platform – Google – and varied whether participants learned from standard Google results or Google’s AI Overview feature.
The findings confirmed that, even when holding the facts and platform constant, learning from synthesized LLM responses led to shallower knowledge compared to gathering, interpreting and synthesizing information for oneself via standard web links.
Well, yeah, of course. It’s the difference between someone giving you the answer and finding the answer out for yourself. The latter involves at least some critical thinking to determine if what you’re reading answers your question.
Yeah but both are unreliable though
Well, the bad thing is that the Google search results (the non AI part) nowadays are so much worse than 10 or even 5 years ago. Just use DuckDuckGo or something else.
Yeah, in my personal experience AI shines in unstructured search, and acting as a jumping point.
I reach to it as a way to find the keywords then I can use on a google search or equivalent.
The study falls short in that the users who used LLMs for creating summaries to subjects did not check the sources used by the LLMs. Versus the participants that did their work themselves creating the likelihood of unreliable information due to LLM errors in reliable information and sources.
“Old fashioned” is something of a stretch
Getting harder to tell, because the search results are increasingly dominated by the outputs of these LLMs. They are both too verbose and not sufficiently detailed, even when they are correct. They also tend to swamp results for *hard* problems with ones for related *easy* problems. This makes it hard to identify what even makes the hard problems hard, as you can’t find *any* information about them.
Just have ADHD and do deep dives into very niche topics until ungodly hours of the night like the rest of us. No AI necessary.
same with image gen. what its generating is the most generic, average, rounded-to-the-nearest-cliche pic in the whole worldwideweb
Remember when people would complain about a Google search being „not real learning“ because it’s so much less work than reading a book or multiple articles on the topic, or going to an actual library?
Wow. I’d have never guessed that reading the AI version of Cliff Notes would give you a superficial understanding of a topic. Thank goodness someone did a paper on this. Otherwise no one would have figured it out.
Soc did a number on Google search. Before ai.
I just cant stand it correcting my grammar and making assumptions about what I’m searching for.
It really depends on how deeply they would have researched the topic on their own. I’m finding web search more unless now that top-scoring web pages are 100% lengthy LLM-generated trash. This feedback loop of AI finding and summarizing its own content is the death of actual information.
This feels less like “AI is bad for learning” and more like “passive summarization is bad for learning.” If you let a model spoon‑feed you a neat overview, you skip the messy searching, comparing sources, and retrieval that actually build depth. I’d love to see a version of this study where people use AI in an active way instead (ask for multiple conflicting takes, get quizzed, force step‑by‑step reasoning) and see if the gap with web search shrinks or flips.
The experiment / paper uses self reported self learning assessments for determining the claim that AI falls short / has shallower knowledge? They didnt use some quiz or test on the subject matter?
i kind of doubt this is apples to apples. I’m a google search, you might read a primary document after the search. So it’s really search->read. For an llm it’s going to be a kind of light version of any document so it’s just not as deep unless you spend a lot of time chatting with the llm and even then you probably have to do some fact checking.
I mean when the study uses time spent on the task as a metric for learning, then compares Ai with manual google searches – of course its going to require more effort to manually search about information on a basic topic. Would of been a far more insightful to give all participants the same time constraints.
Try learning with a physical book; it’ll blow your mind.
Reminds me of when people would say that Google searching lowered our ability to retain information because the instant gratification of the Google Search didn’t promote long term memory storage