Doug Hieber, Senior Application Architect
October 16, 2018
In assessment exams, item enemies can undermine the validity of your exam. The author of this blog post says, use AI to separate item enemies.
If your enemies are united, separate them – The Art of War
In “The Art of War”1, Sun Tzu advises that if your enemies are united, you should separate them. He was advising 5th century BC Chinese generals, but the same is true for the assessment industry’s problem with item enemies.
To get on the same page, let’s talk briefly about what I mean by “item enemy” and why it is a problem.
Item enemies are two or more items that should not appear together on a single exam presented to a single candidate. There are several reasons this would be undesirable. The most obvious is that the “enemies” ask almost the same question, which places too much weight on one competency. It could also be that the wording of one item implies the answer to the other item, which gives the candidate an unfair advantage over candidates that were not presented with those two items. There are other reasons as well, but the overriding idea is that presenting both items to a single candidate is undesirable. It adversely affects the ability to effectively measure the candidates’ competency. So, follow Sun Tzu’s advice and ensure item enemies are separated into different exam instances.
Now for the hard part, identifying the items that are enemies. This is an industry-wide problem without a simple answer. Most exam programs rely on the content experts to know all the items in the bank and identify item enemies as they are created. This is difficult even if the number of operational items in the item bank is small. If the item bank is large, it is an almost impossible task, and most item banking software provides little or no automated help.
At Strasz, I have had the pleasure to work on a feature of Pro!2 that does help. To address the problem, we pulled together mathematics experts, assessment industry experts and developers who specialize in the assessment industry’s solution space to apply their expertise to create a unique solution. The team used a form of artificial intelligence called Natural Language Processing (NLP). It uses advanced vector mathematics, word analytics, frequency analysis and technical know-how to output a weighted Similarity Index between two items. The process was then fine-tuned through field experiments utilizing real world data. Finally, it was integrated into Pro! as a feature that provides a set of sensitivity and scope settings to enable test developers and content experts to control the results that are displayed. This provides a list of items that may be enemies. Content experts can then review this list of suggestions and mark the items that truly are enemies. Human intervention provides the last step, but this automated process helps lead the way. This saves valuable content expert time and improves the thoroughness of item enemy tagging.
The item enemy tag becomes part of the metadata for the item, and it can be used in processes such as report creation or test form assembly.
Sun Tzu would approve of using the Similarity Index to go on the offensive against item harvesting by targeting suspect content posted to the web. Tutorials or exam preparation programs may claim to have items that match operational items and the Similarity Index equips test sponsors with a tool to quickly identify items that may have been compromised.
Sun Tzu might not understand most of this blog, but he would certainly appreciate that the right tool lets you get the job done faster, or as he would say: “Let your rapidity be that of the wind.”
References:
1 https://en.wikipedia.org/wiki/The_Art_of_War
2 http://146.66.113.141/solutions-and-services/strasz-exam-management-suite/pro/
Doug Hieber, Senior Application Architect
October 16, 2018
In assessment exams, item enemies can undermine the validity of your exam. Use AI to separate item enemies.
If your enemies are united, separate them – The Art of War
In “The Art of War”1, Sun Tzu advises that if your enemies are united, you should separate them. He was advising 5th century BC Chinese generals, but the same is true for the assessment industry’s problem with item enemies.
To get on the same page, let’s talk briefly about what I mean by “item enemy” and why it is a problem.
Item enemies are two or more items that should not appear together on a single exam presented to a single candidate. There are several reasons this would be undesirable. The most obvious is that the “enemies” ask almost the same question, which places too much weight on one competency. It could also be that the wording of one item implies the answer to the other item, which gives the candidate an unfair advantage over candidates that were not presented with those two items. There are other reasons as well, but the overriding idea is that presenting both items to a single candidate is undesirable. It adversely affects the ability to effectively measure the candidates’ competency. So, follow Sun Tzu’s advice and ensure item enemies are separated into different exam instances.
Now for the hard part, identifying the items that are enemies. This is an industry-wide problem without a simple answer. Most exam programs rely on the content experts to know all the items in the bank and identify item enemies as they are created. This is difficult even if the number of operational items in the item bank is small. If the item bank is large, it is an almost impossible task, and most item banking software provides little or no automated help.
At Strasz, I have had the pleasure to work on a feature of Pro!2 that does help. To address the problem, we pulled together mathematics experts, assessment industry experts and developers who specialize in the assessment industry’s solution space to apply their expertise to create a unique solution. The team used a form of artificial intelligence called Natural Language Processing (NLP). It uses advanced vector mathematics, word analytics, frequency analysis and technical know-how to output a weighted Similarity Index between two items. The process was then fine-tuned through field experiments utilizing real world data. Finally, it was integrated into Pro! as a feature that provides a set of sensitivity and scope settings to enable test developers and content experts to control the results that are displayed. This provides a list of items that may be enemies. Content experts can then review this list of suggestions and mark the items that truly are enemies. Human intervention provides the last step, but this automated process helps lead the way. This saves valuable content expert time and improves the thoroughness of item enemy tagging.
The item enemy tag becomes part of the metadata for the item, and it can be used in processes such as report creation or test form assembly.
Sun Tzu would approve of using the Similarity Index to go on the offensive against item harvesting by targeting suspect content posted to the web. Tutorials or exam preparation programs may claim to have items that match operational items and the Similarity Index equips test sponsors with a tool to quickly identify items that may have been compromised.
Sun Tzu might not understand most of this blog, but he would certainly appreciate that the right tool lets you get the job done faster, or as he would say: “Let your rapidity be that of the wind.”
References:
1 https://en.wikipedia.org/wiki/The_Art_of_War
2 http://146.66.113.141/solutions-and-services/strasz-exam-management-suite/pro/