All blog posts ……..Start each work week with the latest issue. Scroll down to subscribe!

Scenario assessment do’s and don’ts

When we’re training skills, the graded and ungraded assessments we provide usually fall into one of two categories.

1. Knowledge assessments. In most cases, knowledge assessment should be brief, contextual, front-loaded (that is, provided very early in the training) and should assess only the knowledge required to perform skills.  When we’re training skills, knowledge assessments should represent only about 10 to 15% of the total assessments we assign.
2. Performance assessments. The bulk of assessments associated with skills training should be performance assessments; in other words, assessments that require learners to perform skills. 

Performance assessments can take one of two forms:

Performance chunks (formative)

In this type of assessment, we ask learners to perform one or two steps of a performance (or, complete just one or two steps toward a full work product). Assigning these small “chunks” builds learner confidence and helps to avoid overwhelm.  It allows learners to cement their understanding and ability step-by-step.  This approach is most useful when we’re training long, complicated processes or tasks.

Complete performance (summative)

After learners master the “chunked” or formative assessments we’ve given them, we need to assign “put-it-all-together” or summative performance assessments that require learners to perform an authentic skill in an as-authentic-as-we-can-make-it environment (or to complete an authentic work product) from beginning to end. This type of assessment is often referred to as a scenario assessment because it gives learner practice handling a real-life scenario similar to the ones they will be expected to handle post-training.

Scenario assessments drive critical thinking applied to a specific domain or skillset. The end result of a scenario assessment is always one of the following:

1. A work product (a physical artifact the learners created, such as a report), and/or
2. A performance (a set of steps the learners completed, such as a customer service interaction, that did not result in a physical artifact).

Apply the following 10 do’s and don’ts to create effective scenario assessments that scaffold and measure real-world training results.

1. DO acknowledge that creating sufficient effective scenario assessments is non-trivial.

Budget more time and staff than you think you need.

2. DO provide all backfill info learners need to understand your skills training.

Typically, any given learning audience is going to include some learners who are relatively familiar with the training topic, some who are unfamiliar, and some who think they’re familiar but have gaps in their knowledge.

To prepare all learners to master the same skill, we need to get all learners “on the same page” in terms of descriptions (e.g., of tasks and systems) and definitions (e.g., of any acronyms or domain-specific terms) that underpin the skills training.

3. DO start by describing boundaries, facts, and definitions for all the domains relevant to your training.

Training most skills requires us to provide much more detailed, contextual information than most of us realize.

4. DO support leaners post-training with bulletproof documentation.

Plan from the beginning to support your learners post-training with accurate, comprehensive documentation that supports learners in thinking critically about, handling, and responding to real-world scenarios. If you have a documentation team in-house, make sure to work closely with that team to develop training that aligns closely with documentation.

Documentation should include:

1. All relevant factors that potentially determine, control, or affect the scenario.
2. How these factors interrelate.
3. Which factors are “nice to have” and which are critical requirements of a successful outcome.

5. DO provide sufficient annotated scenario examples.

Provide many, many, many scenario examples. For each, include descriptive “think throughs” that explain the step-by-step thinking behind each decision and that tie back to the documented procedures and facts you’ll be providing to your learners after they’ve completed the training.

6. DO consider pairing training with post-training job shadowing.

This approach is especially critical for new hires as well as for all employees, regardless of tenure, in high-stakes industries.

7. DO consider providing at least one expert/learner one-on-one session.

One-on-ones with a SME or other industry expert may seem expensive, but if they’re timed correctly they’re actually the most efficient way of communicating critical thinking skills.

One-on-ones scheduled during the latter half of the training allow learners to practice with expert support and get high-quality, nuanced, point-of-need feedback at key decision points—all of which are necessary for them to be able to navigate complex, highly variable real-life scenarios.

8. DON’T provide practice that’s shallow, limited, or unrealistic.

In other words, don’t train the “happy path” and call it a day.

9. DON’T expect theoretical scenarios to substitute for real-life, authentic practice.

Explaining how they might approach riding a bike (or any other skill) is no indication of how learners will perform an actual skill in an authentic situation and environment.

10. DON’T make the mistake of assuming knowledge is equivalent to ability.

Knowledge assessments are easy to produce and administer, and they can be useful for measuring learners’ knowledge of skill-related rules and definitions. But knowledge assessments are of no value in assessing learners’ willingness or ability to perform.

The bottom line (TLDR)

Scenario assessments are the only way to drive skills acquisition.  Because creating and administering sufficient scenario assessments is relatively time-intensive, however, a lot of us low-ball assessments by providing one or two overly simplistic mocked-up examples and a few multiple-choice questions.

But there’s no such thing as a free lunch.

As instructional designers, we get to decide whether we want learners to practice and build skills:

1. By completing scenario assessments administered in a controlled environment, where learners can make the mistakes necessary to learn in a safe space; or
2. On live customers and data, which has the potential to harm the business irreparably in the long term.

Smart money says to choose #1.

What’s YOUR take?

Do you have a different point of view? Something to add? A request for an article on a different topic? Please considering sharing your thoughts, questions, or suggestions for future blog articles in the comment box below.

• LinkedIn