When the Tutor Leaves the Room

If an AI tutor helps a learner answer every practice question, what has the product proved? It may have made the session easier. Whether the learner can handle the next unfamiliar problem is still an open question.

That distinction belongs in the product brief, before a completion chart becomes a claim about learning.

Google's September 15 Gemini Notebook announcement describes new study experiences, including interactive overviews and additional quiz formats. These features expand the ways a learner can work with material. The announcement does not establish their effect on learning, and this article is not a hands-on evaluation of them.

The product decision is broader: if a feature promises to build a skill, its evaluation needs an unfamiliar task the learner completes without the tutor supplying the answer. Engagement and assisted performance remain useful measures. They cannot establish that capability on their own.

A better practice score can conceal a worse result

An earlier randomized high-school mathematics study makes the distinction unusually clear. Students using an unrestricted GPT-4 interface did better during practice, then scored worse than the control group on an exam without assistance. A tutor with teacher-designed safeguards largely removed that harm, but did not produce a positive exam effect.

This was a particular school, subject, and implementation. It does not establish that AI tutoring generally harms learning. It shows why successful assisted practice cannot substitute for the outcome the product promises.

There is encouraging newer evidence, too. A September 13 preprint on Medly reports a positive adjusted effect on GCSE science revision assessments after a four-week randomized trial. Hedges' g, a standardized effect size, was 0.33, with a 95% confidence interval of 0.18 to 0.48. Assessment questions were withheld from the intervention.

That result deserves attention alongside its limits. Medly commissioned and funded the evaluation; 644 of 929 baseline participants had post-test results. The assessments were not standardized external exams, and the study does not establish durable retention or broader transfer. It is a preprint, not yet peer-reviewed evidence. Its authors also distinguish the trial's treatment comparison from associations between platform use and outcomes.

The useful conclusion is conditional: product design and evaluation design matter. Neither “AI teaches” nor “AI prevents learning” is a sufficient release argument.

Build the assessment before polishing the tutor

Here is a fictional workplace-training example. The skill is applying a rule with two necessary conditions, including recognizing when information is missing. It is deliberately narrow enough to inspect.

The practice policy permits an expedited replacement only when an item is damaged and delivery was within the past 30 days. A damaged item delivered 12 days ago qualifies. A tutor can explain that example perfectly while leaving the learner with an unreliable shortcut: damaged means eligible.

The assessment should expose that shortcut. Keep the policy available as a fixed reference. Remove answer generation and hints. This tests applying the rule, rather than memorizing its wording or operating without accessibility support.

The distinction becomes visible in the answers. “Approve case A because the item is damaged” reveals the shortcut even though the learner succeeded in practice. “I need the delivery date” in case C demonstrates something an answer-completion metric could penalize: correctly declining to decide too soon.

The context-change probe adds a different setting while preserving the underlying conjunction. It is only a starting point. A learning specialist should review item difficulty and alternative interpretations before the team treats the score as a meaningful outcome.

What would earn a learning claim?

Compare the proposed tutor with the existing training approach, randomly assigning learners before they choose whether to use it. Keep training time comparable and measure baseline skill. Selecting only enthusiastic users would answer a different question. This tests the whole tutoring design; isolating the contribution of its hint policy would need a separate comparison.

For this brief, assess fresh cases the next day and again a week later. Those intervals are proposed checkpoints, not research-established defaults. The later test helps distinguish an immediate benefit from something that persists; it still cannot establish long-term retention.

Choose the smallest worthwhile improvement before collecting results, based on the cost of training and consequential errors in the workflow. Plan sample size around that improvement and report uncertainty. Record who misses each assessment and whether missingness differs by group. A positive result among the people who returned is not automatically a result for everyone assigned.

Keep assisted task success on a separate track. Measure completion time, requests for help, and learner frustration as well. A tutoring design that withholds every useful answer could produce a miserable experience without teaching anything.

The product hypothesis is that requiring an attempt and offering targeted hints will improve delayed, tutor-off performance relative to the current approach. If an adequately powered comparison showed no worthwhile improvement, the team should revise that design or withdraw its learning claim. A higher completion rate would not rescue the hypothesis.

Sometimes assistance is the whole point

There is a strong objection: people use tools at work. Why evaluate them under conditions the product is designed to eliminate?

For a permanent support copilot whose promise is faster, correct case resolution, that objection is right. The primary evaluation should include the copilot and the real workflow. Removing it tests a different product promise. Any fallback capability needs its own justification, tied to an actual operating requirement.

But a tool sold as training makes a claim about the person. If the intended outcome is independent judgment, the evaluation has to observe it. Teams can offer both assistance and training, provided they say which outcome their evidence supports.

This extends the distinction in Duolingo Designed for the Return: returning gives learning another opportunity; return itself is not the learning result. For AI tutors, even a correct answer can belong partly to the tool.

Before the next learning-feature review, put one successful practice answer beside a fresh, tutor-off assessment. Ask what would convince the team that the learner can apply the idea. That extends the value question in Adoption Is Not an AI Value Metric. Leave the review knowing whether the team is claiming better assisted work, a stronger learner, or both.