Plenty of training platforms now sell AI roleplay. Fewer stop to ask what makes a roleplay worth a learner's time. Two easy answers are simple: tell people how they will be judged before they start, and tune the roleplay based on what they will be judged.
Two ways to run the same roleplay
- •A typical AI roleplay: the hidden standard. A new account manager starts a sales roleplay. The screen says: "Handle this objection from a hesitant client." They give a polite, caring answer. Afterwards, the grader docks them 30% for not using the company's "Feel, Felt, Found" method and for missing a required pricing disclosure. Neither was mentioned beforehand.
The learner feels cheated and stops trusting the tool. What they learn is that grading is random. Next time, they spend their energy guessing the secret test instead of building the skill.
- •A Mentor126 Roleplay: the standard comes first. Before the same account manager presses Start Roleplay, the card shows:
- •The situation: a customer is frustrated about a late delivery.
- •Who you're talking to: the customer, and the objections they will raise, like "Your team missed the deadline" and "Why should I trust you again?"
- •What you'll be graded on: take responsibility without making excuses (30%), propose a concrete plan to fix it within 24 hours (40%), and stay calm as the customer gets heated (30%). Plus the pass mark.
Now the learner rehearses those exact behaviors before they speak. The Roleplay tests whether they can meet a known standard under pressure, not whether they can read the agent’s mind.
Why showing the standard works
A rubric is not just a tool for the grader. Heidi Andrade's work on rubrics showed they are a learning tool for the learner (Andrade, 2000). Seeing the criteria up front works like a map. Learners plan, check their own preparation against a clear standard, and aim their effort before they start.
Being open about the task pays off in results too. In a study of college courses, Mary-Ann Winkelmes and colleagues made assignments transparent: the task, its purpose and how it would be judged were all spelled out in advance. Students gained in confidence and skills, and the biggest gains went to the students who started with the fewest advantages (Winkelmes et al., 2016). Stating the standard levels the field for people who would never think to ask.
A score a manager can trust
Showing the standard also changes what the score means:
- •Every criterion is visible. The learner sees each one, its weight and the pass mark before starting.
- •The score is built from the criteria. The final number is added up from the criterion scores. It is not a number the AI makes up on its own.
- •The feedback follows the same list. After the Roleplay, the result comes back criterion by criterion, with areas to improve.
For a manager, that means a score you can explain. You can see which part of the conversation someone has nailed and which part still needs work.
The takeaway
It is not hard to tell which of the two roleplays above is a better use of a learner's time. Transparency is not the only thing that makes a roleplay work, but without it, a roleplay is a guessing game.
References
Andrade, H. G. (2000). Using rubrics to promote thinking and learning. Educational Leadership, 57(5), 13–18.
Winkelmes, M. A., Bernacki, M., Butler, J., Zochowski, M., Golanics, J., & Weavil, K. H. (2016). A teaching intervention that increases underserved college students' success. Peer Review, 18(1/2), 31–36.