Rory McCorkle, PhD, VP of Business Development
March 28, 2024

Over the next couple of weeks, our blog explores the standard-setting process and Strasz’s standard-setting facilitation platform, SetScore!. This week, we will discuss the purpose of standard setting and its unique nature among test development stages, considerations in SME participation, and approaches to standard setting.

At one time, the standard-setting process was described as “the most controversial problem in educational assessment today” (Hambleton, 1998). While issues of equity and AI have likely moved into this position today, the pass rate issues experienced by many programs post-COVID highlight the degree of importance that standard setting still has in certification and licensure programs (PSI, 2023; Firth, 2023). So, discussing the unique place standard setting has in our test development lifecycle is important.

Purpose of Standard Setting

Outsiders often focus on the cut point or passing standard as the focus of standard setting. However, the primary focus is truly on establishing performance standards. For certification or licensing exams, this is the standard of competency, knowledge, or performance required to perform in the role being credentialed. This is the definition of minimal competency or qualification.

Second, it is also important to note that the standard-setting study does not identify the passing score but makes recommendations to the governing body of the certification or license. This governing body makes the final decision on the passing score or cut point based on a review of the recommendations and consideration of the diversity of the study participants. This turns the theoretical concept of the definition of minimal competency into an applied standard of the cut score, or in the words of Kane (1994), draws: “a distinction between the passing score, defined as a point on the score scale, and the performance standard, defined as the minimally adequate level of performance for some purpose.”

Given this distinction, credentialing bodies must consider several factors when setting a new standard:

  • Performance standards and the definition of minimal competence are incredibly important to ensure that the standard-setting panel has clear direction on applying this standard to examination questions. Sufficient depth of detail is critically important to create better reliability between participants in the study and will help the final recommendation accurately reflect this definition.
  • The translation of the performance standard to a passing standard or cut score is not as simple as a statistical technique. Rather, there will be effects based on the SMEs in the panel, depth of training, facilitation style, etc. Training, appropriate facilitation, and participant feedback through the study assume tremendous importance in this test development stage.
  • Subject matter expert (SME) diversity, particularly about their experience level, specialization, or other factors that may cause them to overestimate or underestimate the needed level of competency for the role, may help to diminish the need to adjust the recommended passing score after the study is completed.

Subject Matter Expert (SME) Considerations

Let’s discuss the backbone of all test development activities: our SMEs. As Cicek, Bunch, & Koons (2004) note: “participants in the standard-setting process are critical to the success of the endeavor and are a source of variability of standard-setting results.” To limit this variability, credentialing bodies must ensure enough SMEs are involved in the process to account for outlier SMEs and reflect the credentialing population. As mentioned earlier, special attention should be paid to reflecting the practice in terms of years of experience, practice setting, specialization, or other role factors that might influence how standard-setting participants view the minimal level of performance.

So, what are some important tips for your SMEs for standard-setting studies:

  • Diversity, diversity, diversity: During every stage of test development work, having a well-rounded group of SMEs is important to ensure the highest quality examination. However, it takes on greater importance during standard-setting, as ensuring that less experienced practitioners and disadvantaged groups have a voice becomes more critical in establishing performance standards and applying those to the exam items.
  • Number of SMEs: While studies may be done with fewer SMEs, having a larger group (at least 12) provides flexibility in terms of representation of diversity and allows for SMEs with outlier ratings to be removed in the final data.
  • Proper training: Whether the training is being done by your vendor or an internal psychometrician/test developer, SMES must understand the process, the importance of performance standards, and how they should be applied to the examination items.
  • Use data to re-calibrate raters: As discussed in the next section, it is also important to use data to help SMEs understand when they are trending high or low. From showing them how they compare to other SMEs to using candidate data, this input can help them better calibrate their application of the definition of minimal competence.

Standard Setting Approaches

Dozens of methods exist to set passing standards on credentialing examinations, but the Angoff method is by far the most common. Recognized extensively in both psychometric literature and legal cases concerning cut scores, the simplicity of explaining and training SMEs on this method, with its associated ease of data collection, has made it the ‘default’ method for most credentialing programs. Originally, the Angoff method was described as determining whether the minimally competent or qualified examinee could answer the item correctly (with SMEs giving those items a value of 1) or incorrectly (with SMEs giving those items a value of 0). This method is still used today in the Modified Angoff form of rating items with a Yes or No based on whether the minimally competent candidate would answer correctly.

However, the most commonly used methods today are the Modified Angoff methods where SMEs assign a probability to the item on whether the SME would get the item correct – done via a rating of 0 to 100. Different variants of this modification are used, such as only permitting ratings in multiples of 5.

One of the other common modifications is using rating rounds (typically 2) while providing feedback data to the panelists during the second round to adjust their ratings. This data could be normative, such as providing data from fellow panelists on their average, median, minimum, and maximum ratings, with standard deviation. It could also be impact feedback, showing the panelists how candidates performed on these items (Plake & Cicek, 2012). This feedback may also be provided on a set cadence of items, such as every fifth item in the sequence. Regardless of the feedback, it should be done within an appropriate context to ensure that ratings are not overly skewed toward the general examinee population, focusing on how the minimally competent or qualified candidates would perform on the item.

Regardless of the Angoff modification chosen, a skilled psychometrician and facilitator can help ensure that the method is structured specifically to the needs of the credential for which the passing score is being set and the data available to the credentialing organization.

So, where do I go from here?

As highlighted by pass rate issues, standard setting plays a crucial role in certification and licensure programs, especially post-COVID. Ensuring that the process focuses on establishing performance standards rather than just the passing score can help maintain appropriate standards for your program. Credentialing programs should ensure that they focus on several factors before undertaking a new standard-setting study:

  • Factors such as performance standards, SME diversity, and training are critical in setting new standards. SMEs should reflect the credentialing population and undergo proper training to ensure consistency in their ratings.
  • The Angoff method, particularly its modified versions, is widely used in standard-setting studies due to its simplicity and effectiveness. Modifications include assigning probabilities to item correctness and using rating rounds with feedback to adjust ratings.
  • Skilled psychometricians and facilitators are essential in structuring the standard-setting method according to the credential’s needs and available data.

About the Author

Rory McCorkle is the Vice President of Business Development for Strasz Assessment Systems. With 20 years of experience in the testing industry, he is a recognized expert in testing and assessment. Having worked with hundreds of licensure bodies, credentialing organizations, and higher education institutions on their testing programs, he has helped them navigate strategy, product development and management, test development, marketing, and accreditation issues in their programs. When not at work, he spends his time playing sports in local Kansas City community leagues, playing music and singing, as well as spending time with his wife and three stepchildren.

John DeFalco, SR Software Engineer
August 6th, 2021

Ken White is a Scrum Master for one of our Agile development teams. He’s also our Production Support Operations Manager for the same customer. I don’t believe combining these roles is a practice unique to Strasz. What really sets Ken apart from most others is, he is also currently the Fire Chief for the Liberty Corner Volunteer Fire Department1. So, it goes without saying that Ken has both an educational background and practical experience to bring teams of people together with a high likelihood of success. We’ve all heard of the chicken and egg paradox. So was the fire department the chicken and his college degree the egg? Or vice versa?

Ken (left) alongside the Chief (middle) and Deputy Chief (right) of the Liberty Corner Volunteer Fire Department.

Ken White is a Scrum Master for one of our Agile development teams. He’s also our Production Support Operations Manager for the same customer. I don’t believe combining these roles is a practice unique to Strasz. I’m sure there are plenty of other leaders in the field that are holding down both positions. What might be rarer, Ken has a degree in Management Information System & Operations Management that almost exactly aligns with his current job responsibilities. What really sets Ken apart from most others is, he is also currently the Fire Chief for the Liberty Corner Volunteer Fire Department1. So, it goes without saying that Ken has both an educational background and practical experience to bring teams of people together with a high likelihood of success. We’ve all heard of the chicken and egg paradox. So was the fire department the chicken and his college degree the egg? Or vice versa?

James Lipton from The Actor’s Studio is often fond of saying, “Let’s start at the beginning.” Back in the summer of 1986, Ken was working as a lifeguard and snack bar manager at a local pool when a friend approached him about joining the volunteer fire department. He hadn’t previously given it a thought. Yet, he immediately became fascinated by the inner workings of how the organization came together as a team. He was impressed that such a large group of volunteers could be coordinated to achieve great things in the community. The do-it-yourselfer in Ken was also fascinated with the department’s dizzying array of tools and equipment. 

Later that same year, he went off to college at the University of North Carolina at Greensboro. Ken conveys his choice of UNCG simply as “My parents could afford the school, and it was farther away than Rutgers.” As was previously stated, he pursued a degree in Management Information Systems & Operations Management, which was a natural choice, in retrospect. From early adulthood, Ken had a predisposition towards organizational thinking, technology, leadership, and management.

After graduating from college, Ken began his career at AT&T as a software developer and simultaneously became more involved with the fire department. He started his coding journey with an internal COBOL development program at AT&T. Ken rose through the organization over the next ten years. Ken eventually became a District Manager, with a staff of 80+ and 3 direct report managers. Concurrently, he rose through the ranks of the fire department. He became President, then worked his way up as Assistant, 2nd Assistant, then eventually Chief. At the fire department, Ken leads a multi-faceted team of 60 volunteers. 

The overlap of these two paths is significant. Both have a business and support side that require intense management, efficient organization, and experienced leadership at a high level. A software company’s business revolves around planning and scheduling releases, conducting regular status meetings, managing budgets, and interfacing with customers. The fire department is organized as a not-for-profit business and, as such, has a President that presides over the company’s business. This includes filing tax for

ms with the state, managing donations, fiscal planning, project planning, creating specifications, procurement, politics, and leading public meetings. Both positions require an individual at the top with stellar organizational and planning skills and a positive demeanor supporting customers.

For a software company, every product requires support. Users will encounter defects, and those defects must quickly be researched, verified, and remediated. Customers will occasionally have ad-hoc, high-priority requests in response to their own business’ stimuli, colloquially referred to as “fires” by the production support team. In parallel, the support side of the firehouse handles responding to dispatched 911 calls and extinguishing actual, physical fire alerts sent through an Incident Command System. When asked which fires are harder to control, Ken quipped, “The actual fires … usually”. 

On both fronts, teams are composed of individuals with specific roles and skills. For a software company, those roles are typically developers, designers, quality assurance, and IT. Team members use their varied skills and come together to create solutions. When a challenge arises, Developers will research the code base and provide technical solutions. Production support accesses the logs in production and applies their working knowledge of the system and the user’s workflow to determine how to recreate the issue. IT investigates network, security, and server-related issues. The fire department is similarly multi-faceted. The engine company performs fire suppression, the truck company provides ventilation and search capabilities, and others whose job is to provide a water supply. Clearly, both organizations need a respected and capable leader to coordinate the varied problem resolution activities in a responsive and professional manner.

In the summer of 2021, Ken celebrated his 35th year with the Liberty Corner Fire Department. I’d like to extend the celebration by adding to it Ken’s 35th year of applying, like Liam Neeson (Taken), “a particular set of skills,” both technical and managerial, to every aspect of his professional and personal life.

1 http://www.libertycornerfire.org/ – please help their cause by donating!