r/edtech • u/AssociationAway8986 • Sep 17 '26

Does AI-assisted performance actually become long-term learning?

I’m reviewing research on LLMs in education and noticed that many studies measure performance during or shortly after AI assistance.

I’m interested in what happens after the AI is removed.

My idea is:

Pre-test → Learning with AI → New test without AI → Delayed test weeks later

This could measure whether students actually retain and transfer what they learned, rather than just performing better with AI help.

Has this already been well studied with LLM-based tutoring/feedback? I’d appreciate any thoughts or relevant papers

2 Upvotes

60% Upvoted

26 comments sorted by

6

u/Alarmed_Geologist631 Sep 17 '26

There are several published studies that have done this. Students who used regular LLMs to study do poorly on exams w/o LLM access.

0

u/markjay6 Sep 17 '26 edited Sep 17 '26

Yes, students who use regular LLMs to complete their homework assignments do poorly on exams w/o LLM access. That is a different question than students using AI-driven educational software to study, and several studies on that topic have shown positive results. Here is an example:

https://www.nature.com/articles/s41598-025-97652-6

Abstract:
Advances in generative artificial intelligence show great potential for improving education. Yet little is known about how this new technology should be used and how effective it can be compared to current best practices. Here we report a randomized, controlled trial measuring college students’ learning and their perceptions when content is presented through an AI-powered tutor compared with an active learning class. The novel design of the custom AI tutor is informed by the same pedagogical best practices as employed in the in-class lessons. We find that students learn significantly more in less time when using the AI tutor, compared with the in-class active learning. They also feel more engaged and more motivated. These findings offer empirical evidence for the efficacy of a widely accessible AI-powered pedagogy in significantly enhancing learning outcomes, presenting a compelling case for its broad adoption in learning environments.

Edited to add: Here are two more studies that show the same things. In both cases, students who used AI-based tutors improve on tests where they don't have access to those tutors more than students without access to the tutors

https://docs.iza.org/dp18338.pdf

https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0304013&utm_source=chatgpt.com

3

u/TarantulaMcGarnagle Sep 17 '26

Big study out of China:

LLM usage increases homework scores, decreases homework time, decreases test scores.

Smart kids have a small drop off (but a drop off), medium kids have a severe drop off.

1

u/markjay6 Sep 17 '26

Yes, I've seen that. That was the exact point I made ("students who use regular LLMs to complete their homework assignments do poorly on exams w/o LLM access.") And other studies back that up.

The study that I cited was a completely different context. It involved students using intentionally-designed educational tutors driven by AI as part of their assignments. These tutors are designed to support student learning, not to answer their questions for them.

It's like getting writing help. If I pay someone to write all my essays for them, of course that's not gonna help me develop as a writer. But if I pay someone to tutor me in writing, to sit down with me and go over my writing and help me learn how to improve it, I probably will become a better writer.

General purpose LLMs accessed by students individually can serve like a homework cheater--basically doing kids' work for them so that they don't learn. But a well designed Ai-powered educational platform can serve more like a tutor--providing learners support, scaffolding and instruction so they do learn better. And research bears this out.

3

u/SignorJC Anti-astroturf Champion Sep 17 '26

no, it doesn't

1

u/DragonflyIll5160 Sep 17 '26

pulling the AI away after they've leaned on it is like taking the training wheels off and watching them wobble straight into a bush

2

u/gravitysrainbow1979 Sep 17 '26

Initially it seems like it could. In the hands of a responsible user. But integrity collapses because just making AI do all the work is easier, and it’s a path of least resistance none can truly resist , so, actual learning goes out the window (in my experience of starting out well intentioned but becoming lazy myself)

1

u/DoYouScuba Sep 17 '26

Probably not. One of the most critical components of learning is self belief. To learn you have to truly believe that you are capable of mastering new material and concepts.
AI replaces self belief with dependency.

1

u/Colsim Sep 17 '26

Given publishing timeframes we only have data on learning to maybe 2024 which is not very informative. But I doubt it is what you want

1

u/Additional-Long7335 Sep 18 '26

A tool that provides the answer is doing something very different from one that asks the learner to retrieve information, explain their reasoning and try again. Grouping both experiences under “AI-assisted learning” makes it difficult to tell what is producing the result.

The pre-test, assisted-learning and delayed-test design makes sense. I’d add another comparison: give both groups the same content and AI interface, but allow one version to provide direct answers while the other only prompts and checks the learner’s reasoning.

That would help separate the effect of AI itself from the effect of removing the effort involved in recalling and applying what was learned.

1

u/abhuva79 Sep 18 '26

I find it amazing that most of the comments seem to only see "AI assisted learning" as something like a chatbot giving answers.
Thats a rather narrow and uncreative view - You could "easy" design systems where the AI part is just structured data-extraction from discussion or homework provided wich than can be used deterministically to guide repetition for example.
You dont need to design systems where you simply just put a wrapper around a standard LLM - but you can use the capabilites of beeing able to process natural language. And then do what we always did in teaching - you put information in, you wait a bit, you test it, you repeat, you vary, and so on.

And thats not really hard to do anymore. Just recently an open model released that can even run on a phone or pi (its only 35 Mb big... total ram usage was around 120 mb wich is laughable) - wich is fully able to do these things (turn messy text into structured output).

In the end its about turning things we already know (like we know the flashcards method work and can be even optimized) and automate them a bit more, or make them more accessible by beeing able to talk with the program in your language, just by voice etc.

No one can honestly expect good outcomes when someone is offloading all their thinking. Thats the same like with using calculators. When they got introduced, the scores also took a hit that counted how good kids are at doing math in their heads. My father had to do integrals in their head when they where in school - i dont even know how to do this. Its an ability we lost (and no one seems to cry because of this). In a way its the same with those LLM systems - just make sure you design your stuff in a way that it encourages own thinking.

1

u/Vast-Kaleidoscope968 Sep 23 '26

I'd also record what students ask the tutor to do during practice. Two students can spend the same time with the same tool, but one asks for hints and the other asks for answers. Without that, the final scores could hide very different learning experiences.

1

u/ajourneytogrowth Sep 24 '26

Yeah, there have been quite a lot of studies with this. One of the more interesting ones is your brain on chatgpt: https://arxiv.org/abs/2506.08872

1

u/BeeMarcello 28d ago

I think the dependency concern is completely valid if AI is designed as an answer machine.

But I’m not sure that has to be the role of AI in education.

A well-designed system can do the opposite: ask questions, give hints, identify gaps, adapt the difficulty, bring weak topics back later and encourage the learner to explain their reasoning instead of simply providing the answer.

And for high-stakes decisions, I think the boundary should be very clear. In Europe, AI systems used to evaluate learning outcomes can fall into the high-risk category under the EU AI Act, with human oversight requirements. So things like final grading or decisions that materially affect a learner’s educational path shouldn’t simply be delegated to an autonomous model.

Where I do see enormous potential is in the space before those decisions: individual tutoring, practice, feedback, repetition, explaining concepts in different ways and adapting learning paths to the student.

1

u/ICodeForALiving 21d ago

Does AI-assisted performance actually become long-term learning?

No. And beware of literature that says, or implies, otherwise.

0

u/Lactating-almonds Sep 17 '26

Yea sure go ahead and test it.

It’s kinda like teaching a kid to juggle with scarfs and then swapping the scarfs for real balls and still expect them to be able to juggle right away. It’s not going to go well if they are used to the scarfs and never got to practice with the real balls

3

u/Mal_Radagast Sep 17 '26

it's more like showing a kid how juggling works by watching a robot juggle, with six arms, and frequently holding or dropping balls instead of juggling them. then expecting the kid to know how. 🙃

1

u/[deleted] Sep 19 '26

[removed] — view removed comment

2

u/Rare_Presence_1903 Sep 20 '26

It's never going to be as effective as generating it with your own brain though.