Rory McCorkle, PhD, VP of Business Development
February 20, 2024

In the past two weeks, our blog has delved into the sessions and program we’re excited about for the Association of Test Publishers (ATP) Innovations in Testing Conference, scheduled for March 3 – 6. However, that’s not the only source of our excitement. At the conference, we’re also gearing up to share information about the ongoing evolution of our Pro! item development and banking software, our SetScore! platform for standard setting, and new AI features we’re rolling out.

In the past two weeks, our blog has delved into the sessions and program we’re excited about for the Association of Test Publishers (ATP) Innovations in Testing Conference, scheduled for March 3 – 6. However, that’s not the only source of our excitement. At the conference, we’re also gearing up to share information about the ongoing evolution of our Pro! item development and banking software, our SetScore! platform for standard setting, and new AI features we’re rolling out.

Pro! – A Powerful Platform for Item Writing, Review, and Banking

With over 20 years of experience in item development, banking, and delivery, Strasz comprehends the item development needs faced by small and large certification programs alike. More importantly, we remain responsive to our clients’ needs through our agile development process. This process enables us to research, scope, and rapidly develop new or enhanced features to address our clients’ needs. Having been in the testing industry for two decades, I am impressed by the responsiveness of this team and the efficacy of our developers’ work.

We recently introduced new features, streamlining bulk item updates, simplifying the process of importing item keys, and enhancing the tracking of item clones. If you require additional support with your item development and banking needs, visit booth #103 and ask any of our team for a demo. We would love to showcase our platform and discuss our current projects!

SetScore! – Adding Security and Enhanced Management to Your Standard-Setting Study

Standard setting is a critical process within test development, used not only to set the passing score for an examination but also as one of the two stages where subject matter experts (SMEs) review a full examination form of items. Including highly sensitive information presents a unique security challenge for certification bodies.

Strasz developed SetScore! to address this process’s management, collaboration, and security challenges. Having conducted numerous standard-setting studies throughout my career, I’ve encountered and used various approaches to provide SMEs items, collect Angoff ratings, and analyze this data. Whether using Excel sheets or customized survey platforms, many leave security challenges to trust – providing opportunities for bad actors to collect item information if desired.

SetScore! allows credentialing organizations and vendors to manage this process more effectively. Only vetted SMEs can enter the platform through careful user management to review and provide item ratings. The platform imports (via APIs) items from Pro! or via secure upload for the study, minimizing security risks. SetScore! also allows psychometricians and test developers to see real-time data on SMEs’ progress through the rating process while managing the rounds of ratings, better enabling SME management and reminders. Finally, SetScore! takes the analysis out of the back end of the process, providing calculations of recommended cut scores and deviations automatically at the end of the process, reducing the time to make a final cut score decision.

Learn more about SetScore! here or stop by ATP Innovations to get a demo!

AI in the Item Development Process

I have discussed AI and its impact on the testing industry for months. The use of AI in item development provides both opportunities and challenges. First, for organizations that want to ensure their intellectual property can be secured and copyrighted, it is important that content still has a human development component to ensure it meets U.S. Copyright rules (please note: I am not a lawyer, and you should consult with your counsel on this matter). Second, AI provides incredible opportunities to enhance the work done by your SMEs, making them more productive and producing higher-quality content.

With this in mind, we have been busy adding features to our platforms to enable credentialing organizations to use AI tools in their development process. At ATP’s Innovations conference, we will be unveiling some of our work and previewing new features that are coming soon in three critical areas:

  •         Item Analysis: AI tools can be a great co-pilot to our valuable SMEs. We can use private large language models (LLMs) to help our clients, and their volunteers analyze the items they produce. These models can also be customized and trained to your content, for example, your reference library. This model can analyze draft items to validate keys, analyze the attractiveness of distracters, and ensure proper English usage. It can even help SMEs with that last tricky distracter they are trying to write!
  •         Form Analysis: Building complete examination forms can be tricky to ensure that the content meets blueprint requirements while not having cueing or enemy items. We will be releasing features to allow organizations to automatically review forms for enemy items, increasing the productivity of your test development teams and increasing the accuracy of enemy item identification.
  •         Item Generation: For organizations looking to generate exam items, we will preview our item generation tool at the Innovations conference. This tool can generate an item for review or assist SMEs in their item writing process. Here, a custom LLM can also ensure that the content is based on your reference material, not simply public materials, while protecting your IP through a secure, private AI model.

Our work to bring our clients new features is never finished! Stop by our booth (#103) at the Innovations conference or schedule a demo today if you want to learn more.

About the Author

Rory McCorkle is the Vice President of Business Development for Strasz Assessment Systems. With 20 years of experience in the testing industry, he is a recognized expert in testing and assessment. Having worked with hundreds of licensure bodies, credentialing organizations, and higher education institutions on their testing programs, he has helped them navigate strategy, product development and management, test development, marketing, and accreditation issues in their programs. When not at work, he spends his time playing sports in local Kansas City community leagues, playing music and singing, as well as spending time with his wife and three stepchildren.

John DeFalco, SR Software Engineer
August 6th, 2021

Ken White is a Scrum Master for one of our Agile development teams. He’s also our Production Support Operations Manager for the same customer. I don’t believe combining these roles is a practice unique to Strasz. What really sets Ken apart from most others is, he is also currently the Fire Chief for the Liberty Corner Volunteer Fire Department1. So, it goes without saying that Ken has both an educational background and practical experience to bring teams of people together with a high likelihood of success. We’ve all heard of the chicken and egg paradox. So was the fire department the chicken and his college degree the egg? Or vice versa?

Ken (left) alongside the Chief (middle) and Deputy Chief (right) of the Liberty Corner Volunteer Fire Department.

Ken White is a Scrum Master for one of our Agile development teams. He’s also our Production Support Operations Manager for the same customer. I don’t believe combining these roles is a practice unique to Strasz. I’m sure there are plenty of other leaders in the field that are holding down both positions. What might be rarer, Ken has a degree in Management Information System & Operations Management that almost exactly aligns with his current job responsibilities. What really sets Ken apart from most others is, he is also currently the Fire Chief for the Liberty Corner Volunteer Fire Department1. So, it goes without saying that Ken has both an educational background and practical experience to bring teams of people together with a high likelihood of success. We’ve all heard of the chicken and egg paradox. So was the fire department the chicken and his college degree the egg? Or vice versa?

James Lipton from The Actor’s Studio is often fond of saying, “Let’s start at the beginning.” Back in the summer of 1986, Ken was working as a lifeguard and snack bar manager at a local pool when a friend approached him about joining the volunteer fire department. He hadn’t previously given it a thought. Yet, he immediately became fascinated by the inner workings of how the organization came together as a team. He was impressed that such a large group of volunteers could be coordinated to achieve great things in the community. The do-it-yourselfer in Ken was also fascinated with the department’s dizzying array of tools and equipment. 

Later that same year, he went off to college at the University of North Carolina at Greensboro. Ken conveys his choice of UNCG simply as “My parents could afford the school, and it was farther away than Rutgers.” As was previously stated, he pursued a degree in Management Information Systems & Operations Management, which was a natural choice, in retrospect. From early adulthood, Ken had a predisposition towards organizational thinking, technology, leadership, and management.

After graduating from college, Ken began his career at AT&T as a software developer and simultaneously became more involved with the fire department. He started his coding journey with an internal COBOL development program at AT&T. Ken rose through the organization over the next ten years. Ken eventually became a District Manager, with a staff of 80+ and 3 direct report managers. Concurrently, he rose through the ranks of the fire department. He became President, then worked his way up as Assistant, 2nd Assistant, then eventually Chief. At the fire department, Ken leads a multi-faceted team of 60 volunteers. 

The overlap of these two paths is significant. Both have a business and support side that require intense management, efficient organization, and experienced leadership at a high level. A software company’s business revolves around planning and scheduling releases, conducting regular status meetings, managing budgets, and interfacing with customers. The fire department is organized as a not-for-profit business and, as such, has a President that presides over the company’s business. This includes filing tax for

ms with the state, managing donations, fiscal planning, project planning, creating specifications, procurement, politics, and leading public meetings. Both positions require an individual at the top with stellar organizational and planning skills and a positive demeanor supporting customers.

For a software company, every product requires support. Users will encounter defects, and those defects must quickly be researched, verified, and remediated. Customers will occasionally have ad-hoc, high-priority requests in response to their own business’ stimuli, colloquially referred to as “fires” by the production support team. In parallel, the support side of the firehouse handles responding to dispatched 911 calls and extinguishing actual, physical fire alerts sent through an Incident Command System. When asked which fires are harder to control, Ken quipped, “The actual fires … usually”. 

On both fronts, teams are composed of individuals with specific roles and skills. For a software company, those roles are typically developers, designers, quality assurance, and IT. Team members use their varied skills and come together to create solutions. When a challenge arises, Developers will research the code base and provide technical solutions. Production support accesses the logs in production and applies their working knowledge of the system and the user’s workflow to determine how to recreate the issue. IT investigates network, security, and server-related issues. The fire department is similarly multi-faceted. The engine company performs fire suppression, the truck company provides ventilation and search capabilities, and others whose job is to provide a water supply. Clearly, both organizations need a respected and capable leader to coordinate the varied problem resolution activities in a responsive and professional manner.

In the summer of 2021, Ken celebrated his 35th year with the Liberty Corner Fire Department. I’d like to extend the celebration by adding to it Ken’s 35th year of applying, like Liam Neeson (Taken), “a particular set of skills,” both technical and managerial, to every aspect of his professional and personal life.

1 http://www.libertycornerfire.org/ – please help their cause by donating!