Coordinating and Evaluating Training Effectiveness, Explained

UPI Study › Courses › Healthcare Organization and Management › Free lessons › Lesson 13.4 Free with a free account

Coordinating and Evaluating Training Effectiveness

Healthcare Organization and Management · Chapter 13: Workforce Training and Development in HealthcareSection 13.4 · lesson 77 of 10510 min · free with a free account

← Types and Methods of Employee TrainingNext lesson: Employee Development, Retention, and Staffing →

Earn 3 college credits for Healthcare Organization and ManagementRecommended for credit by NCCRS · Health 302 · self-paced · one payment, lifetime access
Enroll in Healthcare Organization and Management · $250 →Download the syllabus (PDF)

In this lesson

Coordinating training and measuring whether it actually changed practice.

  • Explain why coverage planning and accurate training records come before evaluation
  • Describe Kirkpatrick's four levels and why reaction scores mislead
  • Measure behavior change through observation, record audits, and well-timed follow-up
  • Judge attribution and return-on-investment claims and use comparison groups to evaluate training

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,323 words). Read along, or search it.

Opening Case

A Lifting Course and a Vendor's Return

Last spring, forty of Larkfield's ICU staff took a four-hour course on safe lifting, using ceiling lifts to move patients. They rated it four point eight out of five. The next year, staff lifting injuries in the ICU fell from nine to five. The vendor's summary claims a two hundred percent return on investment. Helen Wu, the chief executive, has to decide whether to run it on every unit. So, did the training work? And how would anyone know?

Coordination

Scheduling, Coverage, and the Training Record

Before evaluation comes coordination. Four pieces of plumbing decide whether training happens at all. The training calendar's real limit is the staffing schedule, not room space. Coverage has to be arranged when a session is booked, not the week before. Then record keeping, and compliance reporting to the bodies that require it. The training record is a legal and regulatory document. After an adverse event, it's what shows whether a person was equipped for the task. A record showing attendance, where validation was required, is a liability.

Kirkpatrick Model

Four Levels of Training Evaluation

Donald Kirkpatrick set out the four levels of training evaluation in nineteen fifty-nine. Each answers a different question. Level one, reaction: what did people think of it? Usually a short form at the end of the session. Level two, learning: did they gain the knowledge or skill? A test, or an observed demonstration. Level three, behavior: do they work differently afterward? Observation, or an audit, weeks later. Level four, results: did the organization's performance change? An operational or clinical measure, before and after.

Evaluation Ladder

Cheapest Level, Least Information

Up the ladder, each level costs more, takes longer to answer, and is harder to pin on the training rather than something else. Each level also says nothing about the level above it. And that creates the field's central problem: the cheapest level tells you least, and it's where almost everyone stops.

Reaction Scores

Why Liking a Course Proves Little

Why stop at level one? It takes two minutes while everyone's still in the room. The number comes fast, it's almost always favorable, and nobody has to follow up. But studies find that liking a course barely predicts how much people learned from it. Worse, reaction scores reward comfort, entertainment and confirmation. Training that takes effort and challenges current practice, the kind that changes behavior, often scores lower. Manage a training portfolio on reaction scores, and you systematically select the training least likely to work.

Quick Check

A High Rating and a Perfect Skills Test

Quick check. A falls-prevention class for 4 South earns a four point nine rating, and every nurse aces the skills test at the end. What does that evidence show: that falls will drop, that they can do it, for now, or that unit practice has changed? Pause, and decide. They can do it, for now. That's level two: capability at the moment of the test, and nothing about what happens on the unit. Falls dropping is tempting, because the test was perfect. But that's level four, two rungs up, and neither level the class measured can speak for it.

Level Three

Measuring Behavior on the Job

Most managers should aim for level three. It's the first level that answers a question worth the money, and the last one you can attribute with any confidence. Direct observation is strongest, but people act differently when watched. So observe repeatedly, observe a sample without announcing it, and trust a high rate less than a low one. A record audit is cheap and unobtrusive, but measures documentation, which isn't always practice. A practice review works where the behavior leaves a trace. Timing matters too. The week after training captures a peak. The useful question is what the rate is at three months.

Worked Example

Slings Stored Down the Hall

Back in the ICU, a senior nurse quietly observes twelve patient transfers. Only two use the ceiling lift. That's seventeen percent. So the manager asks four staff who skipped the lift what stopped them. Three give the same answer: the slings are stored in a room down the hall. That isn't a training failure. It's a supply problem the course could never fix. The slings move into each patient room, and at six weeks, nine of twelve transfers use the lift.

Attribution

Several Changes in One Year

Level four runs into attribution. Injuries fell from nine to five, and the training happened. They're linked only if nothing else could explain the drop. But that year the ICU also added two new ceiling lifts, and started a lift team on day shift. And attention itself changes performance. Picking a problem for training focuses everyone on it. The honest statement is that injuries fell during a year when several things changed, and training was one of them.

Return on Investment

An Assumption That Quadrupled the Ratio

Jack Phillips added a fifth level: return on investment. Benefits minus costs, divided by costs, times one hundred. The cost side is workable if it includes released time and backfill. Say the lift course cost forty thousand dollars in full. The benefit side needs a dollar value for each injury avoided. Assume fifteen thousand dollars each, for four injuries, and the return is fifty percent. Assume thirty thousand each, and the same data give two hundred percent. The ratio quadrupled on an assumption. So state the cost accurately, state the benefit in its own terms with a baseline, and don't multiply by a guess to make a ratio look rigorous.

Quick Check

Questioning a Four Hundred Percent Return

Quick check. A vendor reports that its handover course returned four hundred percent, valuing each avoided handover error at fifty thousand dollars. What should a manager ask first: where the fifty thousand comes from, how highly staff rated it, or how long it runs? Pause, and decide. Where the fifty thousand comes from. The whole four hundred percent rests on that one value. Halve it, and the return falls to one hundred fifty percent. Asking about ratings is tempting, because it's easy. But ratings are level one, and they say nothing about errors avoided.

Evaluation Design

Comparison Groups, Timing, and Decay

Three decisions shape any evaluation. The strongest is a comparison group: measuring an untrained group alongside the trained one. It's often free. Rollouts are usually phased, so the units not yet trained are a comparison sitting in plain sight. Wasting it is the commonest missed opportunity. Timing is the second. Right after training captures a peak, at three months what lasted, and both together the decay curve. And third, expect decay. Plan the follow-up measure, and better still, a reinforcement point before it.

Using Evaluation

Ask Who Did Not Change

Evaluation is usually commissioned to justify a program. It's far more valuable used to improve one. Justifying asks whether it worked, which invites a defensive answer. Improving asks which parts worked, for whom, and what blocked transfer. The single most useful question is rarely asked: of the people who didn't change, what stopped them? The answers are usually specific, and about the workplace. Then decide: continue, redesign, or discontinue. Discontinuing almost never happens, because stopping means someone must say it didn't work.

Reporting

What Leaders Want to Know

Leaders want four things, and usually get one. They want to know what the training was for, what changed, what it cost in full, and what's recommended next. They usually get attendance and completion figures, which answer none of those. A level-three report changes the conversation. Observed practice moved from one rate to another, held at three months, and cost a stated number of released hours. It also makes the manager someone whose training requests come with evidence, and that shapes how the next request is received.

Case Resolved

An Honest Recommendation to Roll Out

So, did the lift course work? The recommendation to Helen doesn't repeat the vendor's two hundred percent. It reports observed lift use rising from seventeen to seventy-five percent after the slings moved, and the course's full cost. It says injuries fell during a year of several changes. And it rolls the course out unit by unit, with the units not yet trained as the comparison.

Key Takeaways

Evaluating Training Honestly

To recap. Coordination and accurate records come first; the record is a legal document. Kirkpatrick's four levels each prove less than the one above. Reaction scores barely predict learning; aim for level three. Attribution is hard, and ROI built on an assumed value proves nothing. Use a comparison group, and ask who didn't change, and why.

About Healthcare Organization and Management at UPI Study

Healthcare Organization and Management (Health 302) is a self-paced online course of 105 video lessons in 18 chapters, from health systems and the law to ethics, leadership, teams, strategy, informatics, finance, quality, human resources, marketing and evidence-based practice. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Health courses Every course with PRO →

Why take it here

Why students take Healthcare Organization and Management with UPI Study

How hospitals and clinics are runHealthcare Organization and Management covers how care organizations are led and run: health law and ethics, leadership and teams, strategic planning, health informatics, budgets, quality improvement, staffing and marketing. It is a core course in healthcare administration and health management degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Healthcare Organization and Management carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Coordinating and Evaluating Training Effectiveness

What are Kirkpatrick's four levels of training evaluation?
Donald Kirkpatrick set out four levels in 1959. Reaction asks what participants thought of the training. Learning asks whether they gained the knowledge or skill. Behavior asks whether they work differently afterward. Results asks whether the organization's performance changed. Each higher level costs more, takes longer to answer, and is harder to attribute to training, and no level proves anything about the level above it.
Why are training satisfaction ratings a poor measure of training effectiveness?
Reaction scores barely predict how much people learned. They tend to reward comfort, entertainment, and confirmation of current views, while training that takes effort and challenges current practice often scores lower even though it is more likely to change behavior. Managing a training portfolio on reaction scores can systematically favor the courses least likely to work, which is why managers should aim to measure behavior.
How do you calculate the return on investment of training, and why can it mislead?
Training return on investment, the fifth level Jack Phillips added, is benefits minus costs, divided by costs, times 100. A $40,000 course credited with four avoided injuries returns 50 percent if each injury is valued at $15,000, but 200 percent at $30,000. Because the result swings on an assumed dollar value, state costs fully and report benefits in their own terms with a baseline.
How can a hospital tell whether a drop in incidents was caused by training?
Use a comparison group, measuring untrained staff alongside trained staff. Phased rollouts make this easy, because units not yet trained can serve as the comparison. Without one, a drop may reflect other changes made the same year, such as new equipment or a new team, or simply more attention to the problem. The honest claim is that the measure improved while training was one of several changes.

About the Healthcare Organization and Management course

How many college credits is Healthcare Organization and Management?
Healthcare Organization and Management (Health 302) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Healthcare Organization and Management cost and what is included?
$250, paid once, with lifetime access: all 105 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Healthcare Organization and Management take?
It is self-paced with no deadlines. The 105 lessons total about 17 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Healthcare Organization and Management.

105 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Healthcare Organization and Management →Download the syllabusEvery course with PRO →
Healthcare Organization and Management
3 credits · NCCRSEnroll →