Answer
AnswerHow do you measure whether AI training changed how people work?
Measure AI training at three levels: whether people can do a new task without help, whether they use AI tools in their real work afterwards, and whether a business number moves, because completion rates show attendance and nothing more.
Completion is the easiest number to collect and the weakest evidence. It shows that a lesson was opened to the end, not that anything was learned or used. Stronger evidence comes in three steps. First, can the person do a task they have not seen before, without the AI doing it for them? Second, are they using the tools in real work a month later? Third, does a metric the business already tracks change? Collect the first two for everyone and the third for one team.
At a glance
- Weak signal
- Completions and hours of training.
- Skill
- A new task done unaided, before and after.
- Adoption
- Use of AI tools in real work some weeks later.
- Business
- A metric the team already tracks, compared with a team that was not trained.
What are the steps to measure AI training?
Set a baseline
Run a short skill check and note current tool use before training starts. Without a before, the after means little.
Test on a new task
Check learning with a task the person has not practiced, done without assistance. Repeating the practice task measures memory, not skill.
Look at use after a few weeks
Ask managers, check tool usage data where you have it and survey the team. Skills that are not used fade.
Pick one business metric for one team
Choose something already tracked, such as handling time or first-draft turnaround, and compare a trained team with a similar untrained one.
Report mastery, not attendance
Give leaders mastery per skill and adoption by team. Those are the numbers that tell them where to act next.
Why are completion rates misleading?
Because activity and learning come apart. In a study of nearly 1,000 high-school students, those given a standard chatbot scored 48 percent higher than a control group during practice and 17 percent lower on a later test taken without it (Bastani and colleagues, PNAS, 2025).
A two-year trial of an AI tutor found that 96 percent of students tried it at least once, and that the median student used it in only 17 percent of the exercise sessions in which they made a mistake (Oreopoulos and Low, NBER, 2026). In both cases access and activity did not equal learning.
What does Scholé report?
Scholé reports mastery per skill and per person, team progress and adoption metrics, and lets an administrator ask for them in plain language. It can send progress to your own systems as xAPI statements and through an analytics API. It does not measure your business metrics for you, so pair its skill and adoption data with a number your team already tracks.
Where Scholé fits
- You want mastery per skill and per person instead of a completion count.
- You want adoption reported by team, and progress data sent to your own systems.
Where Scholé is not the answer
- You need an attributed return on investment figure. No training tool produces that alone, and it takes a comparison your organization has to set up.
- You need the audited record of mandatory course completions. Keep that in your LMS, which is the system of record.
More on this site: Scholé for teams, Integrations.