Chapter 1 Measurement, scoring and classical item analysis
Learning goals. After this chapter you can
- turn raw responses into item scores with a key, and reverse-score negatively worded items;
- compute and read proportion correct, item-total correlations and coefficient alpha;
- use a distractor analysis to spot an item that is miskeyed or badly written;
- say what a sum score can and cannot tell you, and why we need IRT.
Key terms: classical test theory, sum score, p-value, item-total correlation, dichotomous item, polytomous item.
1.1 The ideas
A test is a sample of tasks that we use to make an inference about something we cannot see directly: A student’s reading ability, a nurse’s clinical knowledge, a patient’s level of anxiety. Before any model is fitted, raw responses have to be turned into item scores. For a multiple-choice item that means comparing the option chosen with the key (1 = right, 0 = wrong). For a rating scale it means coding the categories as 0, 1, 2, … and making sure that a higher score always means “more” of the trait, which is why negatively worded items are reverse-scored.
The simplest summary of a person’s performance is the sum score, the total of the item scores. Classical test theory (CTT) models the sum score as a true score plus error, \(X = T + E\), and gives us a small toolkit for judging items (Crocker and Algina 1986):
- Item difficulty (\(p\)): The proportion of people who answer correctly (for rating items, the item mean). Confusingly, a high \(p\) means an easy item.
- Item discrimination: The correlation between the item score and the total score. Items that good performers get right and weak performers get wrong have high item-total correlations. The corrected correlation removes the item from the total first, which matters for short tests.
- Coefficient alpha: An estimate of reliability, the proportion of sum-score variance that is true-score variance.
- Distractor analysis: For multiple-choice items, how often each wrong option is chosen, and by whom. A good distractor attracts low scorers. An option that attracts high scorers is a warning sign: It may be a second correct answer, or the key may be wrong.
CTT statistics are easy to compute and to explain, and you should always start with them. But they have two limits that motivate the rest of this book. First, \(p\) and the item-total correlation depend on the group that took the test: The same item looks easy in a strong group and hard in a weak one. Second, the sum score treats every item as equally informative and gives one standard error for everybody, even though a test measures some people more precisely than others. Item response theory addresses both.
1.2 Example 1: A multiple-choice science test
The SAT12 data in mirt hold the answers of 600 high-school students to 32
multiple-choice science items with five options (8 marks an omitted item).
The key is published on the help page, ?SAT12.
# Raw multiple-choice responses (options 1-5, 8 = omitted) and the key.
raw <- SAT12
key <- c(1,4,5,2,3,1,2,1,3,1,2,4,2,1,5,3,4,4,1,4,3,3,4,1,3,5,1,3,1,5,4,5)key2binary() scores every response against the key. Omits count as wrong.
resp <- key2binary(raw, key)
resp[1:5, 1:8]
#> Item.1 Item.2 Item.3 Item.4 Item.5 Item.6 Item.7 Item.8
#> [1,] 1 1 1 1 1 1 1 1
#> [2,] 0 1 0 0 1 0 1 0
#> [3,] 1 1 1 0 1 0 1 0
#> [4,] 0 1 0 1 1 0 1 0
#> [5,] 0 1 1 1 1 0 1 1
total <- rowSums(resp)
summary(total)
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 4.0 14.8 18.0 18.2 22.0 32.0itemstats() gives the classical statistics for every item in one table.
st <- itemstats(resp)
st$overall
#> N mean.total sd.total std.alpha alpha SEM.alpha mean.r sd.r
#> 600 18.2 5.05 0.795 0.798 2.27 0.108 0.075
head(st$itemstats, 8)
#> N K mean sd cor_if_rm smc alpha_if_rm
#> Item.1 600 2 0.283 0.451 0.300 0.158 0.793
#> Item.2 600 2 0.568 0.496 0.464 0.270 0.785
#> Item.3 600 2 0.280 0.449 0.371 0.184 0.789
#> Item.4 600 2 0.378 0.485 0.235 0.145 0.796
#> Item.5 600 2 0.620 0.486 0.340 0.197 0.791
#> Item.6 600 2 0.160 0.367 0.351 0.185 0.791
#> Item.7 600 2 0.760 0.427 0.289 0.200 0.793
#> Item.8 600 2 0.202 0.402 0.233 0.116 0.795The mean column is the proportion correct, cor_if_rm is the corrected
item-total correlation, and alpha_if_rm is alpha with the item removed. Let
us sort the items by their corrected correlation and look at the weakest.
items <- st$itemstats
items[order(items$cor_if_rm), c("mean", "cor_if_rm", "alpha_if_rm")][1:5, ]
#> mean cor_if_rm alpha_if_rm
#> Item.32 0.162 0.037 0.802
#> Item.12 0.415 0.076 0.803
#> Item.9 0.885 0.127 0.798
#> Item.21 0.915 0.136 0.798
#> Item.11 0.983 0.156 0.797A corrected correlation near zero, or an alpha that goes up when the item is removed, means the item is not measuring what the rest of the test measures. Before blaming the item writer, check the key. A distractor analysis splits students into groups by total score and shows which option each group chose.
distractors <- function(raw, total, item, groups = 5) {
band <- cut(rank(total, ties.method = "first"), groups,
labels = c("lowest", paste0("band ", 2:(groups - 1)), "highest"))
round(prop.table(table(band, option = raw[[item]]), 1), 2)
}
distractors(raw, total, "Item.32")
#> option
#> band 1 2 3 4 5 8
#> lowest 0.12 0.26 0.35 0.09 0.14 0.04
#> band 2 0.10 0.31 0.36 0.07 0.14 0.02
#> band 3 0.19 0.16 0.42 0.08 0.15 0.00
#> band 4 0.16 0.12 0.48 0.08 0.16 0.00
#> highest 0.06 0.07 0.62 0.04 0.22 0.00The keyed answer to item 32 is option 5, yet option 3 is the most popular
choice in every band, and its share climbs steeply from the lowest band to
the highest. Option 5 gains far less. When an “unkeyed” option behaves the
way a correct answer should, suspect the key. The
help page for SAT12 agrees: The key for item 32 is probably wrong and should
be 3. Re-scoring fixes the item:
key_fixed <- replace(key, 32, 3)
resp_fixed <- key2binary(raw, key_fixed)
st_fixed <- itemstats(resp_fixed)
c(old_r = items["Item.32", "cor_if_rm"],
new_r = st_fixed$itemstats["Item.32", "cor_if_rm"],
old_alpha = st$overall$alpha, new_alpha = st_fixed$overall$alpha)
#> old_r new_r old_alpha new_alpha
#> 0.0371 0.2026 0.7979 0.8023What to take away. A low item-total correlation is a symptom, not a diagnosis. The distractor table tells you why: A wrong key, two defensible answers, an option that confuses strong students, or simply an item that is too hard for everyone.
1.3 Example 2: A wellbeing questionnaire
Rating scales need the same care. Here we simulate a short questionnaire: 800 adults rate eight statements about their wellbeing from 0 (strongly disagree) to 3 (strongly agree). Items 3 and 7 are negatively worded (“I often feel tense”), so agreeing with them means lower wellbeing.
# Simulated 8-item wellbeing scale, 0-3, items W3 and W7 negatively worded.
set.seed(101)
a <- matrix(c(1.6, 1.4, 1.2, 1.8, 1.3, 1.5, 1.1, 0.5))
d <- matrix(c( 2.0, 0.5, -1.5,
1.8, 0.2, -1.8,
2.2, 0.8, -1.0,
1.5, 0.0, -2.0,
2.5, 1.0, -0.8,
1.7, 0.3, -1.6,
2.0, 0.6, -1.2,
1.0, 0.0, -1.0), ncol = 3, byrow = TRUE)
resp <- simdata(a, d, N = 800, itemtype = "graded")
colnames(resp) <- paste0("W", 1:8)
resp[, c("W3", "W7")] <- 3 - resp[, c("W3", "W7")] # as answered on the form
resp <- as.data.frame(resp)If we forget to reverse-score, the two negative items correlate negatively with the total, and alpha is badly understated.
itemstats(resp)$itemstats[, c("mean", "cor_if_rm")]
#> mean cor_if_rm
#> W1 1.64 0.348
#> W2 1.49 0.341
#> W3 1.24 -0.412
#> W4 1.44 0.326
#> W5 1.88 0.296
#> W6 1.60 0.356
#> W7 1.29 -0.433
#> W8 1.48 0.122
itemstats(resp)$overall$alpha
#> [1] 0.224Reverse-scoring maps 0 to 3, 1 to 2 and so on:
scored <- resp
scored[, c("W3", "W7")] <- 3 - scored[, c("W3", "W7")]
st <- itemstats(scored)
st$itemstats[, c("mean", "cor_if_rm", "alpha_if_rm")]
#> mean cor_if_rm alpha_if_rm
#> W1 1.64 0.543 0.710
#> W2 1.49 0.459 0.725
#> W3 1.76 0.451 0.727
#> W4 1.44 0.538 0.710
#> W5 1.88 0.443 0.728
#> W6 1.60 0.507 0.716
#> W7 1.71 0.473 0.723
#> W8 1.48 0.210 0.773
st$overall$alpha
#> [1] 0.753All items now point the same way. Item W8 still has a low corrected correlation, and alpha rises from 0.75 to 0.77 without it: It is a weak item, and an IRT model will give it a low discrimination too (Chapter 2). For rating scales the category frequencies are worth a look as well. A category almost nobody uses carries little information and may need to be merged with its neighbour (Chapter 8).
1.4 A first look with mirtverse
profile_data() gives a quick overview of any response file, and is what
mv_ask() uses before it writes an analysis. For each column it reports the
type, the number of distinct values, the range and the share of missing
values, and it guesses which columns are items and which are IDs or grouping
variables.
profile_data(scored)
#> Data profile: 800 rows, 8 columns (8 look like items)
#> column type n_distinct min max prop_missing role levels
#> W1 integer 4 0 3 0 item
#> W2 integer 4 0 3 0 item
#> W3 integer 4 0 3 0 item
#> W4 integer 4 0 3 0 item
#> W5 integer 4 0 3 0 item
#> W6 integer 4 0 3 0 item
#> W7 integer 4 0 3 0 item
#> W8 integer 4 0 3 0 item1.5 Your turn
- Score
SAT12with the corrected key. Which three items are now the hardest, and which three have the lowest corrected item-total correlation? - Run
distractors()on the item with the lowest corrected correlation. Is the problem the key, a distractor, or the difficulty of the item? - In Example 2, what happens to alpha if you drop W8? Would you drop it? Give one reason for and one against.
- Suppose 30% of students omitted the last five items of
SAT12because they ran out of time. Why would scoring omits as wrong distort the item statistics of those items?
Solutions
resp_fixed <- key2binary(SAT12, replace(key, 32, 3))
it <- itemstats(resp_fixed)$itemstats
rownames(it)[order(it$mean)][1:3] # hardest
#> [1] "Item.6" "Item.8" "Item.3"
rownames(it)[order(it$cor_if_rm)][1:3] # weakest
#> [1] "Item.12" "Item.9" "Item.21"- Use
distractors(SAT12, rowSums(resp_fixed), "<item>"). If no option is preferred by the top band and the correct option rises only slowly, the item is simply hard or vague rather than miskeyed. - Alpha rises only a little. Dropping W8 shortens the scale and may remove content the scale is meant to cover; keeping it costs little precision. A content review should decide, not alpha alone.
- Not-reached items would look harder and would correlate with speed rather
than knowledge. Treat not-reached items as missing (
NA) rather than wrong; IRT estimation handles missing responses naturally.
In the lab
The first two sections of Lab 1, run_lab("model-choice"), put you in charge
of scoring a certification exam with 1,200 candidates. You score the
responses, screen the items and decide what to do about anything suspicious.
See Teaching labs.
Further reading
Crocker and Algina (1986) remain the classic introduction to classical test theory and item analysis. The Standards for Educational and Psychological Testing (American Educational Research Association et al. 2014) set out what evidence a test score needs before it is used.