Lisa Everts, Test Developer, Business Development
August 22, 2016

When writing multiple choice test items, how do you make sure the answers aren’t “too hard” nor “too easy,” but “just right?” The author of this blog post recommends using a Goldilocks standard.


goldilocks

When a subject matter expert is crafting a multiple choice test question, the expert knows which answer the candidate is expected to choose. The real challenge for item writers is constructing item options that are not the correct choices. Sometimes even the most experienced SME can write items that do not perform statistically well because the distractors, or incorrect choices, are poorly written. How can the expert avoid making a question too easy or too hard by writing poor distractors?

Give item writers a “Goldilocks test” or “litmus test” to check test items for difficulty level. For most certification programs, the ideal set of options will be neutral; not “too soft” or “too hard”, but “just right”. Direct the group of SMEs through the item writing and review process and ask them to apply the “Goldilocks test” to each test item.

Too Easy
Do any of the incorrect options sound awkward? Read the question and each option separately. If an option is constructed so that it isn’t compatible grammatically with the question, candidates will be likely to eliminate the option based on construction and not content. Doing so gives unintentional cluing to a candidate who doesn’t actually know the answer.

Too Hard
Are any of the distractors common policy but not absolutely correct? While this approach is appropriate to test a candidate’s ability to identify a regulatory requirement or a minimum standard, it may create an item that is very difficult for candidates. They may have difficulty choosing the correct answer over an answer that they recognize as an industry policy.

Just Right
Is the format similar to the other distractors and key? Creating options that are parallel to each other should help ensure that the incorrect options might appear to be as correct as the key to a misinformed test taker.

Using a “Goldilocks test” standard for item writing will result in exam questions that identify competent exam candidates and not just good test takers.

Lisa Everts, Test Developer, Business Development
August 22, 2016

When writing multiple choice test items, how do you make sure the answers aren’t “too hard” nor “too easy,” but “just right?” The author of this blog post recommends using a Goldilocks standard.


goldilocks

When a subject matter expert is crafting a multiple choice test question, the expert knows which answer the candidate is expected to choose. The real challenge for item writers is constructing item options that are not the correct choices. Sometimes even the most experienced SME can write items that do not perform statistically well because the distractors, or incorrect choices, are poorly written. How can the expert avoid making a question too easy or too hard by writing poor distractors?

Give item writers a “Goldilocks test” or “litmus test” to check test items for difficulty level. For most certification programs, the ideal set of options will be neutral; not “too soft” or “too hard”, but “just right”. Direct the group of SMEs through the item writing and review process and ask them to apply the “Goldilocks test” to each test item.

Too Easy
Do any of the incorrect options sound awkward? Read the question and each option separately. If an option is constructed so that it isn’t compatible grammatically with the question, candidates will be likely to eliminate the option based on construction and not content. Doing so gives unintentional cluing to a candidate who doesn’t actually know the answer.

Too Hard
Are any of the distractors common policy but not absolutely correct? While this approach is appropriate to test a candidate’s ability to identify a regulatory requirement or a minimum standard, it may create an item that is very difficult for candidates. They may have difficulty choosing the correct answer over an answer that they recognize as an industry policy.

Just Right
Is the format similar to the other distractors and key? Creating options that are parallel to each other should help ensure that the incorrect options might appear to be as correct as the key to a misinformed test taker.

Using a “Goldilocks test” standard for item writing will result in exam questions that identify competent exam candidates and not just good test takers.