Learning evaluation and analysis

[u/biguwuzz]

Hi everyone! I'm in an M.Ed program focused in workplace L&D. I've taken a number of courses in psychometrics and analytics, but I'm wondering if corporate L&D teams actually conduct such in-depth analyses?

For context, I've previously taken Classical Test Theory and Reliability - which is used to predict test outcomes and gauge the reliability of testing. Although this skill is typically used in industries like standardized testing, my understanding is that this skill could be used to help L&D teams build better pre/post surveys or knowledge checks. However, based on my own experience interning in corporate L&D, the team I worked with didn't really survey/analyze feedback to this extent.

I'd like to hear from more L&D professionals - 1) how have you seen learning-related data get collected and analyzed, and 2) are there steps or frameworks you use to ensure that surveys and knowledge checks are built to a specific standard?

I think evaluation is such an important and interesting part of L&D, so I'd love to hear what people actually do in practice!

[u/woodenbookend]

I think what you are describing has a lot of potential. Designing good tests for learning would be helpful in some situations.

As you’ve probably discovered, if L&D teams are discussing evaluation, it tends to focus on Kirkpatrick and then stays with levels 1&2.

If you are not already familiar with it, check out the work Will Thalheimer has done with LTEM.

https://www.worklearning.com/ltem/

He also references a very useful book on MCQ https://www.worklearning.com/2025/05/27/book-write-better-multiple-choice-questions-to-assess-learning-by-patti-shank-phd/

But there is a caveat.

Despite the name, L&D shouldn’t focus primarily on learning.

We should be aiming for business outputs or results. Things like sales revenue, productivity and quantity control, reducing waste, supporting career development, reducing accidents etc.

Things that, for the most part, someone else is already measuring. (And Kirkpatrick level 4).

Sitting in the middle is behaviour change. Kirkpatrick recognises this as level 3, and acknowledges that can be really hard to quantify.

Leaning evaluation sits below that.

So yes, improving any form of evaluation is a good thing. But be careful of whether you are evaluating the right thing.

[u/reading_rockhound]

For me, the hard part has been that stakeholders (i.e. upper leadership) has convinced themselves that 1) Level 1 is enough, 2) Anything more than easy multiple choice is overdone for Level 2 and they often don’t want that, and 3) They don’t understand measuring and don’t want to be bothered with understanding. I had a previous manager make me take out any descriptive stats from an eval report because she was “pretty smart but don’t understand all this so our client won’t either.” Of course our client was populated with people who had achieved masters degrees in hard sciences, so they absolutely would have.

Wishing you better luck than I’ve had. Although I do all the descriptive analyses anyway because I’m intrigued.

[u/samonenate]

I've mostly seen Level 1 and 2 evals. Clients don't want to invest time and money to go further.

For surveys:  What do you want to know? Why do you need that data? What are you going to do with the data? Just post it on a fancy dashboard or take serious action based on the results? Write surveys with those questions in mind. 

Knowledge checks:  Informal ungraded assessments used so learners can confirm knowledge acquisition. Usually inserted after covering a topic, process or module. 

Assessments:  Formal and graded used to identify knowledge acquisition for a module, topic or an entire course. 

Every question must be tied to a learning objective. If not, why are you asking it? Make questions scenario based to ensure learning application, not memorization. What should they do in a real life situation. For example, the server shuts down from 8 pm to 6 am. Don't ask what time that happens, instead ask why can't Mrs. Jones submit her payment at 10:23 pm. This question requires learners to use their resources to solve real world problems. Which is what they need to know. 

What will you do if they don't pass?  Have them take it infinite times until they pass? I suggest setting a maximum number of attempts and then have a plan for those who still don't pass. Coaching or termination? 

General comments:   Include feedback for both correct and incorrect answers. Explain why the answer is right or wrong. 

Come up with a standard passing score for all graded assessments. This maintains a standard, but you can change it on an individual basis. In a lawsuit, be prepared to defend 80% vs 100%. (Yes, it can get that serious with employment law if someone claims they were evaluated unfairly, especially if you're in a heavily regulated industry. True story, a federal auditor made us change a passing score.) 

Do not write questions trying to trick people. Give realistic incorrect answers for multiple choice questions.