Teaching our course tutor when to hint, when to push and when to explain
We tested the ChemIllusion course tutor against a plain chat assistant on 50 organic chemistry tutoring moments. The tutor chose the right teaching move far more often and gave away far fewer answers. Here is the prompt it now runs on, and what the test can and cannot tell you.
Students in a ChemIllusion course can ask an AI tutor about the chapter they are reading. A tutor that answers every question is easy to build. A tutor that knows when to give a hint, when to ask a student to justify a correct answer, when to just explain, and when to step back is harder. We wanted to measure which of those our tutor does.
Testing one tutoring moment at a time
Our method follows TutorMoments from the Allen Institute for AI (Ai2) (TutorMoments: Do AI tutors know when to help and when to hold back?, August 7, 2026, and its open-source repository). TutorMoments freezes a tutoring conversation at one point and asks what the tutor does next. It separates appropriate scaffolding and appropriate rigor from over-scaffolding. Its moments are mathematics, so we wrote our own for organic chemistry.
Each of our 50 moments covers one of the 31 chapters of our organic chemistry and biomolecules book. Each has a best next move: a hint that unlocks the next step, a push to justify or apply a correct answer, a direct explanation, or a short confirmation that lets the student keep working. Many are built to tempt a tutor into a common chemistry slip, such as which way a charge sits or how far a rule really reaches. We gave the same conversations to our tutor and to a plain chat assistant with no instructions. A separate judge then scored every reply without knowing which system wrote it.
What we found
Fifty new moments, one reply each, scored blind. Gold is the prompt now live.
The plain assistant usually just solved the problem. It gave away answers the student was meant to work out in 69% of moments, and it over-explained in 76%. The tutor held back in all but 14% and chose the right move 94% of the time. Our earlier tutor prompt had chosen the right move 74% of the time on a first set of moments. Its misses were almost all explaining when a hint or a push was called for.
We tested two additions. A teaching-strategy instruction fixed most of those misses. That version still made two small chemistry errors in 50 replies. Both were extra claims the student had not asked for, one of them an overstated rule. A second, chemistry-accuracy instruction brought that to zero in 50 replies. It cost a little in teaching moves (94% against 100%), and we chose accuracy.
The prompt the tutor uses now
The tutor keeps its existing instructions: our ChemEd voice, searching our open textbook first, never handing over a homework or exam answer, and staying on chemistry. Two paragraphs now follow them, word for word:
Teaching strategy
Diagnose what the student's attempt demonstrates. Add only the support needed to make the next reasoning step possible. When a student already demonstrates competence, invite a justification, compare alternatives, or transfer the concept. Avoid supplying steps that take over the student's reasoning. Give a concise direct explanation when a conceptual question genuinely calls for one. Respect the course's assessment policy and let the student work when additional help is unnecessary.
Chemistry accuracy
State only chemistry you are certain is correct. Do not add side facts, rules of thumb, or because-explanations the student did not need for the next step; every extra claim is a chance to be wrong. When you state a rule or trend, give its real scope (slower, competes, gives a mixture) and do not turn a trend into an absolute. When you explain why something happens, check the direction: which atom is electron-poor or electron-rich, where a charge or partial charge sits, which way an equilibrium, rate, or shift moves. Prefer claims the OER text supports. If you are not sure of a claim, leave it out or ask the student to reason it through. Before replying, reread each chemical claim and delete any you cannot defend.
What this test does not show
- It measures what the tutor does, not what students learn. Ai2 makes the same point about TutorMoments.
- The replies came from AI models following each prompt exactly in an offline test, not from the model our students talk to. The result tests the instructions, not that model.
- The best moves and answer keys were drafted with AI help and are still awaiting review by chemistry instructors. The judge was also an AI model.
- Fifty moments is a small sample. Zero errors in 50 replies means a low error rate, not no errors. We record each run and will keep testing the tutor as it changes.