Rory McCorkle, PhD, VP of Business Development
April 4, 2024
Last week, I discussed key considerations around setting performance standards, facilitation, training, and SME selection for the standard-setting process. As a follow-up to that post, this week, we will discuss how some of these goals can be furthered by Strasz’s standard-setting application for this process, SetScore!. Let’s delve into how this tool can aid in test security, facilitation, data management, and centering the process around the performance standards.
Test Security
One of the crucial considerations in the standard-setting process is test security. Typically, SMEs are exposed to a full form of the examination or a larger item pool, which comes with significant security risks. Alongside volunteer management best practices, such as ensuring that SMEs have signed confidentiality and conflict of interest agreements, enhancing security is achievable through the use of a secured tool to access the items.
Historically, standard-setting processes involved loading items into a secured survey tool or having SMEs log into an item banking system while entering Angoff ratings into a separate tool. Both methods present challenges: the former creates additional security risks by placing content in a separate tool, while the latter requires SMEs to navigate between programs to answer questions and provide ratings, impacting the SME experience. With SetScore!, SMEs are shown individual items at a time, rendered directly from the item bank, while also collecting the SMEs’ answers and ratings. SetScore! is OWASP-compliant, offers two-factor authentication, and ensures appropriate access with role-based security.
Facilitation
Facilitating any standard-setting study is challenging for psychometricians or test developers, particularly when running a web conference-based process. SetScore! offers the facilitator extraordinary visibility into the SMEs’ activities, even tracking status through the answering and rating process down to the individual item. This helps facilitators understand what SMEs need to be reminded to complete rounds of ratings and provides real-time data on average ratings, range, deviation, and other key consistency measures. With tracking and reporting capabilities within the platform, facilitators no longer require external analysis tools, simplifying the study’s workflow.
Data Management and Reporting
Having facilitated numerous standard-setting studies as a psychometrician, I vividly recall the challenges of generating the final report and recommendations based on the data. SetScore! streamlines this process by producing off-the-shelf reporting and recommended cut scores based on the study’s data. This can expedite the final cut score determination process with the appropriate governing board for the credential.
The system also enables organizations to centralize their standard-setting data, encompassing SME characteristics and demographics, ratings, and response data. This data can be seamlessly exported at the end of the study for data retention or maintained within the system.
Performance Standards
In my blog last week, I discussed the importance of focusing standard-setting studies on performance standards, also known as the definition of minimal competence or qualification. SetScore! ensures that this definition remains front and center throughout the response and rating process, facilitating SMEs’ easy access to it and improving the alignment of ratings with the performance standards.
Interested in learning more about SetScore! and how your organization might benefit from its use? Please feel free to contact us to learn more or to schedule a demo.
About the Author
Rory McCorkle is the Vice President of Business Development for Strasz Assessment Systems. With 20 years of experience in the testing industry, he is a recognized expert in testing and assessment. Having worked with hundreds of licensure bodies, credentialing organizations, and higher education institutions on their testing programs, he has helped them navigate strategy, product development and management, test development, marketing, and accreditation issues in their programs. When not at work, he spends his time playing sports in local Kansas City community leagues, playing music and singing, as well as spending time with his wife and three stepchildren.
John DeFalco, SR Software Engineer
August 6th, 2021
Ken White is a Scrum Master for one of our Agile development teams. He’s also our Production Support Operations Manager for the same customer. I don’t believe combining these roles is a practice unique to Strasz. What really sets Ken apart from most others is, he is also currently the Fire Chief for the Liberty Corner Volunteer Fire Department1. So, it goes without saying that Ken has both an educational background and practical experience to bring teams of people together with a high likelihood of success. We’ve all heard of the chicken and egg paradox. So was the fire department the chicken and his college degree the egg? Or vice versa?
Ken (left) alongside the Chief (middle) and Deputy Chief (right) of the Liberty Corner Volunteer Fire Department.
Ken White is a Scrum Master for one of our Agile development teams. He’s also our Production Support Operations Manager for the same customer. I don’t believe combining these roles is a practice unique to Strasz. I’m sure there are plenty of other leaders in the field that are holding down both positions. What might be rarer, Ken has a degree in Management Information System & Operations Management that almost exactly aligns with his current job responsibilities. What really sets Ken apart from most others is, he is also currently the Fire Chief for the Liberty Corner Volunteer Fire Department1. So, it goes without saying that Ken has both an educational background and practical experience to bring teams of people together with a high likelihood of success. We’ve all heard of the chicken and egg paradox. So was the fire department the chicken and his college degree the egg? Or vice versa?
James Lipton from The Actor’s Studio is often fond of saying, “Let’s start at the beginning.” Back in the summer of 1986, Ken was working as a lifeguard and snack bar manager at a local pool when a friend approached him about joining the volunteer fire department. He hadn’t previously given it a thought. Yet, he immediately became fascinated by the inner workings of how the organization came together as a team. He was impressed that such a large group of volunteers could be coordinated to achieve great things in the community. The do-it-yourselfer in Ken was also fascinated with the department’s dizzying array of tools and equipment.
Later that same year, he went off to college at the University of North Carolina at Greensboro. Ken conveys his choice of UNCG simply as “My parents could afford the school, and it was farther away than Rutgers.” As was previously stated, he pursued a degree in Management Information Systems & Operations Management, which was a natural choice, in retrospect. From early adulthood, Ken had a predisposition towards organizational thinking, technology, leadership, and management.
After graduating from college, Ken began his career at AT&T as a software developer and simultaneously became more involved with the fire department. He started his coding journey with an internal COBOL development program at AT&T. Ken rose through the organization over the next ten years. Ken eventually became a District Manager, with a staff of 80+ and 3 direct report managers. Concurrently, he rose through the ranks of the fire department. He became President, then worked his way up as Assistant, 2nd Assistant, then eventually Chief. At the fire department, Ken leads a multi-faceted team of 60 volunteers.
The overlap of these two paths is significant. Both have a business and support side that require intense management, efficient organization, and experienced leadership at a high level. A software company’s business revolves around planning and scheduling releases, conducting regular status meetings, managing budgets, and interfacing with customers. The fire department is organized as a not-for-profit business and, as such, has a President that presides over the company’s business. This includes filing tax for
ms with the state, managing donations, fiscal planning, project planning, creating specifications, procurement, politics, and leading public meetings. Both positions require an individual at the top with stellar organizational and planning skills and a positive demeanor supporting customers.
For a software company, every product requires support. Users will encounter defects, and those defects must quickly be researched, verified, and remediated. Customers will occasionally have ad-hoc, high-priority requests in response to their own business’ stimuli, colloquially referred to as “fires” by the production support team. In parallel, the support side of the firehouse handles responding to dispatched 911 calls and extinguishing actual, physical fire alerts sent through an Incident Command System. When asked which fires are harder to control, Ken quipped, “The actual fires … usually”.
On both fronts, teams are composed of individuals with specific roles and skills. For a software company, those roles are typically developers, designers, quality assurance, and IT. Team members use their varied skills and come together to create solutions. When a challenge arises, Developers will research the code base and provide technical solutions. Production support accesses the logs in production and applies their working knowledge of the system and the user’s workflow to determine how to recreate the issue. IT investigates network, security, and server-related issues. The fire department is similarly multi-faceted. The engine company performs fire suppression, the truck company provides ventilation and search capabilities, and others whose job is to provide a water supply. Clearly, both organizations need a respected and capable leader to coordinate the varied problem resolution activities in a responsive and professional manner.
In the summer of 2021, Ken celebrated his 35th year with the Liberty Corner Fire Department. I’d like to extend the celebration by adding to it Ken’s 35th year of applying, like Liam Neeson (Taken), “a particular set of skills,” both technical and managerial, to every aspect of his professional and personal life.
1 http://www.libertycornerfire.org/ – please help their cause by donating!