(Version 2.0, 12/31/94; updated March 2004)
Provided By The
of the
2200 East Devon Avenue, Suite 247
Des Plaines, Illinois
(708) 299-4227
(708) 299-4280 FAX
Compuserve 74040,3722
Copyright 1995 by the Institute for Certification of Computing
Professionals
All rights reserved
The purpose of this handbook is to assist the item writer in their preparation of questions for ICCP examinations. Since these exams consist entirely of multiple-choice types, other forms of item construction have not been included. We have chosen to include both general philosophy and specific mechanics of item construction. This will serve those concerned with question content as well as those who will construct final test items. Although these example items are actual items that have been used in previous ICCP exams, we have chosen them primarily for instructional purposes. They are not indicative of the specific content or level of difficulty which may be encountered in present or future ICCP exams.
The purpose of the words and syntax chosen in writing a test item is to communicate explicit meaning as efficiently as possible. Remember, few written words are read as carefully as those in objective test items. Skill in clear, expository writing are essential to good questions.
Item designers and writers must keep in mind that ICCP examinations are international and multi-cultural in scope. Therefore, writers must guard against constructing test items that contain elements of national, cultural, or regional bias. For instance, in some countries, the CRT--cathode ray tube--is referred to as a VDT--video display terminal. The five-digit postal Zip Code used in the United States is not a universal convention.
The ICCP exams test an applicant's grasp of the "body of knowledge" as defined in the ICCP Official Exam Review Outlines. These outlines define the scope of the exam into which exam items must fit. Writers should refer to the appropriate outline for the content of the exams (and thus the content of items). When submitting new items to ICCP, it is helpful if you classify the item according to the outline section (see the form attached to this guide).
Items should not be vendor-specific or product specific. For example, it is proper to write a question about features of spreadsheet programs in general, but it would not be proper to ask about a specific feature of Microsoft Excel (Registered Trademark of Microsoft Corporation) or Lotus 1-2-3 (Registered Trademark of Lotus Development Corporation).
In addition to classification according to the exam outline, it is also possible to classify the "depth" of the item, as follows:
D1 Recognition What a concept is called; name or title
D2 Differentiation The external differences between one concept
and a similar or neighboring concept
D3 Description The external characteristics of a concept; the
definition of the concept
D4 Usage How, when and why to use instances of a concept
D5 Structure The internal structure of a concept; its components
and the relationships of the components
D6 Construction How to construct instances of the concept from the
components, tailored to a specific purpose or
application
This classification is not absolute, but the idea of depth is useful to item writers in suggesting some different approaches to creating new items. Consideration of depth leads to such questions as "when should this particular method (concept) be used?" or "how can this method be combined with other procedures to achieve an objective?"
Item writers should indicate a target depth level on each question, as well as a classification by competency (subparagraph within the test outline).
What is a good question? In general, look for things that a computing professional -- a practitioner like you -- should know to be an effective contributor to his/her organization and to the profession itself. When a new person comes into your organization, what kind of general background things do you expect them to bring with them? What kinds of general things to you provide via your training program? Ask yourself, "Is this something that is important for every computing professional to know?" In the end, it's a judgment call, and your judgment as a professional is important in making this determination.
All ICCP exams consist of multiple-choice items, or questions. Each item is composed of a stem and four responses (or answers), one of which is correct. When writing an item, follow these rules of structure:
The examples in this guide follow these conventions.
Multiple-choice test items are currently the most highly regarded and widely used form of objective test item. Almost any understanding or ability that can be tested by means of any other item form can be tested by means of multiple-choice test items.
To develop multiple-choice test items, the writer must:
The function of the stem is to acquaint the examinee with the problem that is being posed. Ideally, it should state or imply a specific question. Although you can sometimes save words without loss of clarity by using an incomplete statement as the item stem, a direct question is often better. For the sake of variety, however, the item writer should also include some incomplete item stem questions. In designing these questions, take particular care to focus on a specific problem. Generalities will lead to ambiguity and confusing multiple interpretations. Below are three examples of good multiple-choice items.
Example 1: The item stem is a complete, direct statement.
At what time would a structured walk-through be most effective?
*A) Before program coding
B) At post-completion audit
C) While defining program specifications
D) Before generation of hierarchy charts
Example 2: The item stem is an incomplete statement.
Logical access to a computer system is BEST controlled by
*A) password codes
B) encryption of data
C) intrusion alarm systems
D) complex operating systems
Example 3: The item stem is internally incomplete
Records are _____ from an indexed sequential file by placing an
identifying code within the record.
A) added
*B) deleted
C) updated
D) replaced
Another variation of the multiple-choice question stem is the "Except" or "Not" type. In this construction, the test-taker is asked to select the option that is NOT appropriate. In this form, brevity is critical. Brevity in responses simplifies the task for the test-taker by removing an irrelevant source of difficulty. Brief responses also tend to focus attention on the essential differences among the alternatives offered. In general, this type of item construction should only be used for special situations, such as vocabulary knowledge or, perhaps, knowledge of an exception. Examples 4 and 5 demonstrate acceptable usage of this format.
Example 4: Knowledge of an exception:
Factors relevant to scheduling jobs in a multiprogramming environment
include each of the following EXCEPT
A) priorities
*B) programmer skill
C) memory requirements
D) input/output demands
Example 5: Knowledge of vocabulary
Which of the following is NOT a current asset?
A) Patents
*B) Inventory
C) Prepaid insurance
D) Marketable securities
Notice that in both Example 4 and Example 5, the item writer has emphasized the terms of exception, "except" and "not". This is to help the reader notice the word that is critical to understanding the question. Use upper case letters for emphasis where you want to reduce the possibility of confusion.
The item writer should adhere to certain writing practices when designing the responses. These practices are intended to keep the test truly objective.
1. Use parallel structure. If possible, begin each response with the same word, or part of speech. The test-taker will not be confused by multiple grammatical constructions. Neither will he/she be inclined to choose a selection simply because it is different and, therefore, stands out. This is depicted in Example 6.
Example 6: Use of parallel structure in responses
The MOST common technique for handling overflow records on
direct-access storage devices is
A) chaining
*B) buffering
C) overlapping
D) randomizing
By making all responses the same part of speech the item writer has ensured that the choices are clear and grammatically unbiased.
2. Avoid complex or unduly long responses. These types of response tends to confuse the test-taker. They frequently strain unnecessarily his/her power to concentrate. This particular difficulty is found in Example 7.
Example 7: Avoid complex or long responses
When we refer to the concept, "ones complement" arithmetic, what
precisely do we mean?
A) All numbers are taken to less than one
B) All arithmetic operations are performed twice for validity
checking
*C) All numbers are subtracted from one before performing any
arithmetic operations
D) Negative numbers are formed by inverting the bits in the
positive number such that all zero bits are ones and all
one bits are zeroes
After wading through answers C and D, the test-taker will be strained to remember the actual question.
Ideally, the intended answer to a multiple-choice question should be a thoroughly correct answer. It should admit no difference of opinion among adequately informed experts. This kind of absolute correctness, however, is difficult to achieve except in formal logical systems. Test designers must base many of their items on propositions that are not absolutely true, but are, nonetheless, strongly probable. Notice in Example questions 1 and 2 above that the item writer has stressed the words "best" and "most" in the item stem. This directs the test-taker not to look for absolute correctness, but rather the best possible option.
Another guideline to follow is that the stem of a multiple-choice item should ask a question that has a definite answer. Indeterminate questions may provide interesting topics for discussion, but they do not make good items for testing achievement. Example 8 below demonstrates an indeterminate question.
Example 8: Avoid indeterminate questions
The practice of structured programming is an outgrowth of
A) faster computers
B) increased computer storage
*C) escalating programmer salaries
D) rising costs of program maintenance
In this example, an adequate argument could be mounted for any of the options. Therefore, establishing one correct answer becomes a matter of subjective choice.
The purpose of a distracter in a multiple-choice test item is to discriminate between those who have command of a specific body of knowledge and those who lack it. To do this, a distracter must be a plausible alternative. In other words, it should be appropriate to the question or implied by the stem. One way of obtaining plausible distracters is to use true statements which do not correctly answer the stem question. Example 9 demonstrates this idea.
Example 9: Use of true answers that do not correctly answer the stem
In a memory-constrained virtual storage operating system, which of
the following types of interrupt is the MOST common?
A) Machine check
B) Page interrupt
*C) Timer interrupt
D) Program interrupt
In this example, all of the choices are truly types of interrupts. However, since the question specifies the most common form, only one response adequately answers the question: the timer interrupt. Multiple-choice items should make frequent use of this device for testing an achievement that is sometimes thought to be testable only by using essay examinations.
Another source of plausible distracters are familiar expressions and phrases. Because they have been used in common parlance, they are frequently attractive to students with merely superficial knowledge. See Example 10 below.
Example 10: Using familiar expressions as distracters
A group of characters that is read or written with each physical
read or write operation on a storage medium is called a
A) file
B) field
*C) logical record
D) physical record
It is highly likely that the test-taker would have heard the terms, "file", "field", and "physical record". But a person with a low-level understanding of the topic may well have not understood them. Therefore, these familiar terms provide excellent distracters at the elementary level of discrimination. On the other hand, obscure distracters are usually undesirable. Example 11 demonstrates this concept.
Example 11: Avoid the use of obscure distracters
The connections between the subsystems of an application system
are known as
*A) windows
B) asymptotics
C) permutations
D) catasterizations
If the stem definition and the word "windows" represent an appropriate level of difficulty for this vocabulary test, then the remaining terms used as distracters are obviously too difficult. It is unreasonable to expect the test-taker to know for sure that one of them might not be a better definition than the intended correct answer. Conversely, a distracter which is absurd or highly implausible will contribute little or nothing to the effectiveness of a test item.
A common device for adapting multiple-choice items to questions that seem to require several correct answers is to add as a final alternative the response, "all of the above" or "both of the above." A correct answer should not be wrong simply because there are other correct answers. In addition, the experienced test-taker who may have an incomplete knowledge of the subject will have an improved chance of guessing the correct answer. Example 12 demonstrates this type of question.
Example 12: Avoid the "all of the above" and "none of the above"
During a preliminary study to determine the number of invoices
prepared per month, you should
A) review a production log
B) ask a direct question
*C) do both of the above
D) do neither of the above
If the test-taker knows that option A is correct but is uncertain about B, he/she can eliminate options D and B, greatly improving his/her chances. In a similar vein, the item writer should also avoid the "none of the above" and "neither of the above" options. These item types are more suitable for measuring rote-memory skills such as arithmetic or spelling knowledge.
Go To Contents
Return to Reference Materials
Return to ICCP Home Page