Creating Simple Rules

An assessment organization creates its own catalog of rules through user screens. Each rule has a unique user-assigned name, short text description to aid in its management, and the rule itself. There is no limit to the number of rules the catalog can contain or to the number of rules from the catalog used to test an item’s compliance with policy.

The rule itself is a four-part declaration. The first part identifies the portion of item content to which the rule applies: stem, answer choice, or both. The second part identifies whether the content identified in the first part must or must not contain something. The third part provides choices for contain, begin with, or end with a value contained in the fourth part. The fourth part is the value: a character, characters, word, or phrase. Different combinations of the four parts yield, for example, “The Stem must end with a ‘?’”; The Answer Choice(s) must not contain “true”. And so on.

To add a new rule to the catalog, an authorized member of the assessment organization’s staff enters a name and short description, makes selections for the first three parts, and enters a value for the fourth part.

Creating Compound Rules

Pro! expands the power of the simple rule by using coordinating conjunctions to string multiple simple declarations together. The coordinating conjunctions are And and Or. Each declaration in a compound rule uses the four- part form described above with a coordinating conjunction tacked on all but the last. An example of a compound rule is “Answer choices must not end with ‘?’” AND “Stem must end with ‘?’” AND Stem must not contain ‘not’”. There is no limit to the number of declarations in a rule.

All rules, simple or compound, have the following features:

Error Message

Every rule must have an associated error message. The staff member who creates (or edits) a rule also creates the error message. The error message is the text that Pro! will display to the item writer when an item does not comply with the rule. Pro! places no restrictions on the content of the error message.

Testing Rules

The person who creates a rule would naturally want to conduct a test of the rule’s efficacy before putting it into use. To that end Pro! provides a scratch pad for testing each rule. The scratch pad has a place to enter sample text for the stem and one answer choice. To test a rule, enter text into the scratch pad that represents the portion of item content that the rule is designed to test. The text should contain a violation of the rule, be it invalid characters, words, or phrase. Clicking a button runs the rule against the sample text and reports the result. A good policy is to conduct more than one test and to test that the rule does not report a violation in text where it should not.

The Rules Catalog

The rules catalog stores all rules. The number of rules in the catalog is unlimited. An authorized user can view the list of rules in the catalog and select a rule for editing, saving the changes back into the catalog. Pro! uses its tabular display feature to display the catalog’s contents. The tabular display feature provides powerful sorting, grouping, export, and reporting features that can aid catalog maintenance.

A cataloged rule can be used for any item type in any exam title unless its use has been restricted.

Using Rules in Item Writing Campaigns

A campaign manager chooses the rules for an item writing campaign. During campaign creation the campaign manager can select rules from the catalog, or, to simplify matters, can specify that the new campaign will use the same rules as an earlier campaign.

Checking Item Compliance

An item is tested for compliance with the campaign’s rules when the item writer tries to submit the item. All rules are applied each time an item is submitted. Rules and the declarations within them are applied in the order they appear. The logical relationship between each rule and each declaration within a rule is an AND.

Creating and Deleting Rules

There are two pathways to creating a new rule. The first way is by selecting the menu option, and the second way is during the setup of an item writing campaign. The features are the same regardless of the pathway chosen, except that when creating a new rule from within an item writing campaign, the campaign manager can restrict the new rule’s use to only the new campaign or allow its use in any campaign. Rules can be added and removed from an item writing campaign at any time.

Tracking Changes

Pro! records in the system activity and event log the creation of a new rule and all edits made to it during its lifetime. Activity and event log records are recorded in chronological order. Each log record identifies the user who performed the action that caused the log record to be written and the changes made to the rule. An authorized user can view the list of log records for a given rule, and so can determine who made the change and what was changed about the rule.