Chapter 1 Measurement, scoring and classical item analysis

Learning goals. After this chapter you can

  • turn raw responses into item scores with a key, and reverse-score negatively worded items;
  • compute and read proportion correct, item-total correlations and coefficient alpha;
  • use a distractor analysis to spot an item that is miskeyed or badly written;
  • say what a sum score can and cannot tell you, and why we need IRT.

Key terms: classical test theory, sum score, p-value, item-total correlation, dichotomous item, polytomous item.

1.1 The ideas

A test is a sample of tasks that we use to make an inference about something we cannot see directly: A student’s reading ability, a nurse’s clinical knowledge, a patient’s level of anxiety. Before any model is fitted, raw responses have to be turned into item scores. For a multiple-choice item that means comparing the option chosen with the key (1 = right, 0 = wrong). For a rating scale it means coding the categories as 0, 1, 2, … and making sure that a higher score always means “more” of the trait, which is why negatively worded items are reverse-scored.

The simplest summary of a person’s performance is the sum score, the total of the item scores. Classical test theory (CTT) models the sum score as a true score plus error, \(X = T + E\), and gives us a small toolkit for judging items (Crocker and Algina 1986):

  • Item difficulty (\(p\)): The proportion of people who answer correctly (for rating items, the item mean). Confusingly, a high \(p\) means an easy item.
  • Item discrimination: The correlation between the item score and the total score. Items that good performers get right and weak performers get wrong have high item-total correlations. The corrected correlation removes the item from the total first, which matters for short tests.
  • Coefficient alpha: An estimate of reliability, the proportion of sum-score variance that is true-score variance.
  • Distractor analysis: For multiple-choice items, how often each wrong option is chosen, and by whom. A good distractor attracts low scorers. An option that attracts high scorers is a warning sign: It may be a second correct answer, or the key may be wrong.

CTT statistics are easy to compute and to explain, and you should always start with them. But they have two limits that motivate the rest of this book. First, \(p\) and the item-total correlation depend on the group that took the test: The same item looks easy in a strong group and hard in a weak one. Second, the sum score treats every item as equally informative and gives one standard error for everybody, even though a test measures some people more precisely than others. Item response theory addresses both.

1.2 Example 1: A multiple-choice science test

The SAT12 data in mirt hold the answers of 600 high-school students to 32 multiple-choice science items with five options (8 marks an omitted item). The key is published on the help page, ?SAT12.

# Raw multiple-choice responses (options 1-5, 8 = omitted) and the key.
raw <- SAT12
key <- c(1,4,5,2,3,1,2,1,3,1,2,4,2,1,5,3,4,4,1,4,3,3,4,1,3,5,1,3,1,5,4,5)

key2binary() scores every response against the key. Omits count as wrong.

resp <- key2binary(raw, key)
resp[1:5, 1:8]
#>      Item.1 Item.2 Item.3 Item.4 Item.5 Item.6 Item.7 Item.8
#> [1,]      1      1      1      1      1      1      1      1
#> [2,]      0      1      0      0      1      0      1      0
#> [3,]      1      1      1      0      1      0      1      0
#> [4,]      0      1      0      1      1      0      1      0
#> [5,]      0      1      1      1      1      0      1      1
total <- rowSums(resp)
summary(total)
#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#>     4.0    14.8    18.0    18.2    22.0    32.0

itemstats() gives the classical statistics for every item in one table.

st <- itemstats(resp)
st$overall
#>    N mean.total sd.total std.alpha alpha SEM.alpha mean.r  sd.r
#>  600       18.2     5.05     0.795 0.798      2.27  0.108 0.075
head(st$itemstats, 8)
#>          N K  mean    sd cor_if_rm   smc alpha_if_rm
#> Item.1 600 2 0.283 0.451     0.300 0.158       0.793
#> Item.2 600 2 0.568 0.496     0.464 0.270       0.785
#> Item.3 600 2 0.280 0.449     0.371 0.184       0.789
#> Item.4 600 2 0.378 0.485     0.235 0.145       0.796
#> Item.5 600 2 0.620 0.486     0.340 0.197       0.791
#> Item.6 600 2 0.160 0.367     0.351 0.185       0.791
#> Item.7 600 2 0.760 0.427     0.289 0.200       0.793
#> Item.8 600 2 0.202 0.402     0.233 0.116       0.795

The mean column is the proportion correct, cor_if_rm is the corrected item-total correlation, and alpha_if_rm is alpha with the item removed. Let us sort the items by their corrected correlation and look at the weakest.

items <- st$itemstats
items[order(items$cor_if_rm), c("mean", "cor_if_rm", "alpha_if_rm")][1:5, ]
#>          mean cor_if_rm alpha_if_rm
#> Item.32 0.162     0.037       0.802
#> Item.12 0.415     0.076       0.803
#> Item.9  0.885     0.127       0.798
#> Item.21 0.915     0.136       0.798
#> Item.11 0.983     0.156       0.797

A corrected correlation near zero, or an alpha that goes up when the item is removed, means the item is not measuring what the rest of the test measures. Before blaming the item writer, check the key. A distractor analysis splits students into groups by total score and shows which option each group chose.

distractors <- function(raw, total, item, groups = 5) {
  band <- cut(rank(total, ties.method = "first"), groups,
              labels = c("lowest", paste0("band ", 2:(groups - 1)), "highest"))
  round(prop.table(table(band, option = raw[[item]]), 1), 2)
}
distractors(raw, total, "Item.32")
#>          option
#> band         1    2    3    4    5    8
#>   lowest  0.12 0.26 0.35 0.09 0.14 0.04
#>   band 2  0.10 0.31 0.36 0.07 0.14 0.02
#>   band 3  0.19 0.16 0.42 0.08 0.15 0.00
#>   band 4  0.16 0.12 0.48 0.08 0.16 0.00
#>   highest 0.06 0.07 0.62 0.04 0.22 0.00

The keyed answer to item 32 is option 5, yet option 3 is the most popular choice in every band, and its share climbs steeply from the lowest band to the highest. Option 5 gains far less. When an “unkeyed” option behaves the way a correct answer should, suspect the key. The help page for SAT12 agrees: The key for item 32 is probably wrong and should be 3. Re-scoring fixes the item:

key_fixed <- replace(key, 32, 3)
resp_fixed <- key2binary(raw, key_fixed)
st_fixed <- itemstats(resp_fixed)
c(old_r = items["Item.32", "cor_if_rm"],
  new_r = st_fixed$itemstats["Item.32", "cor_if_rm"],
  old_alpha = st$overall$alpha, new_alpha = st_fixed$overall$alpha)
#>     old_r     new_r old_alpha new_alpha 
#>    0.0371    0.2026    0.7979    0.8023

What to take away. A low item-total correlation is a symptom, not a diagnosis. The distractor table tells you why: A wrong key, two defensible answers, an option that confuses strong students, or simply an item that is too hard for everyone.

1.3 Example 2: A wellbeing questionnaire

Rating scales need the same care. Here we simulate a short questionnaire: 800 adults rate eight statements about their wellbeing from 0 (strongly disagree) to 3 (strongly agree). Items 3 and 7 are negatively worded (“I often feel tense”), so agreeing with them means lower wellbeing.

# Simulated 8-item wellbeing scale, 0-3, items W3 and W7 negatively worded.
set.seed(101)
a <- matrix(c(1.6, 1.4, 1.2, 1.8, 1.3, 1.5, 1.1, 0.5))
d <- matrix(c( 2.0, 0.5, -1.5,
               1.8, 0.2, -1.8,
               2.2, 0.8, -1.0,
               1.5, 0.0, -2.0,
               2.5, 1.0, -0.8,
               1.7, 0.3, -1.6,
               2.0, 0.6, -1.2,
               1.0, 0.0, -1.0), ncol = 3, byrow = TRUE)
resp <- simdata(a, d, N = 800, itemtype = "graded")
colnames(resp) <- paste0("W", 1:8)
resp[, c("W3", "W7")] <- 3 - resp[, c("W3", "W7")]   # as answered on the form
resp <- as.data.frame(resp)

If we forget to reverse-score, the two negative items correlate negatively with the total, and alpha is badly understated.

itemstats(resp)$itemstats[, c("mean", "cor_if_rm")]
#>    mean cor_if_rm
#> W1 1.64     0.348
#> W2 1.49     0.341
#> W3 1.24    -0.412
#> W4 1.44     0.326
#> W5 1.88     0.296
#> W6 1.60     0.356
#> W7 1.29    -0.433
#> W8 1.48     0.122
itemstats(resp)$overall$alpha
#> [1] 0.224

Reverse-scoring maps 0 to 3, 1 to 2 and so on:

scored <- resp
scored[, c("W3", "W7")] <- 3 - scored[, c("W3", "W7")]
st <- itemstats(scored)
st$itemstats[, c("mean", "cor_if_rm", "alpha_if_rm")]
#>    mean cor_if_rm alpha_if_rm
#> W1 1.64     0.543       0.710
#> W2 1.49     0.459       0.725
#> W3 1.76     0.451       0.727
#> W4 1.44     0.538       0.710
#> W5 1.88     0.443       0.728
#> W6 1.60     0.507       0.716
#> W7 1.71     0.473       0.723
#> W8 1.48     0.210       0.773
st$overall$alpha
#> [1] 0.753

All items now point the same way. Item W8 still has a low corrected correlation, and alpha rises from 0.75 to 0.77 without it: It is a weak item, and an IRT model will give it a low discrimination too (Chapter 2). For rating scales the category frequencies are worth a look as well. A category almost nobody uses carries little information and may need to be merged with its neighbour (Chapter 8).

round(st$proportions, 2)
#>       0    1    2    3
#> W1 0.19 0.23 0.34 0.24
#> W2 0.24 0.24 0.32 0.20
#> W3 0.16 0.22 0.33 0.30
#> W4 0.26 0.26 0.27 0.21
#> W5 0.12 0.21 0.35 0.32
#> W6 0.20 0.23 0.33 0.23
#> W7 0.17 0.22 0.34 0.27
#> W8 0.27 0.25 0.20 0.28

1.4 A first look with mirtverse

profile_data() gives a quick overview of any response file, and is what mv_ask() uses before it writes an analysis. For each column it reports the type, the number of distinct values, the range and the share of missing values, and it guesses which columns are items and which are IDs or grouping variables.

profile_data(scored)
#> Data profile: 800 rows, 8 columns (8 look like items)
#>  column    type n_distinct min max prop_missing role levels
#>      W1 integer          4   0   3            0 item       
#>      W2 integer          4   0   3            0 item       
#>      W3 integer          4   0   3            0 item       
#>      W4 integer          4   0   3            0 item       
#>      W5 integer          4   0   3            0 item       
#>      W6 integer          4   0   3            0 item       
#>      W7 integer          4   0   3            0 item       
#>      W8 integer          4   0   3            0 item

1.5 Your turn

  1. Score SAT12 with the corrected key. Which three items are now the hardest, and which three have the lowest corrected item-total correlation?
  2. Run distractors() on the item with the lowest corrected correlation. Is the problem the key, a distractor, or the difficulty of the item?
  3. In Example 2, what happens to alpha if you drop W8? Would you drop it? Give one reason for and one against.
  4. Suppose 30% of students omitted the last five items of SAT12 because they ran out of time. Why would scoring omits as wrong distort the item statistics of those items?
Solutions
resp_fixed <- key2binary(SAT12, replace(key, 32, 3))
it <- itemstats(resp_fixed)$itemstats
rownames(it)[order(it$mean)][1:3]                 # hardest
#> [1] "Item.6" "Item.8" "Item.3"
rownames(it)[order(it$cor_if_rm)][1:3]        # weakest
#> [1] "Item.12" "Item.9"  "Item.21"
  1. Use distractors(SAT12, rowSums(resp_fixed), "<item>"). If no option is preferred by the top band and the correct option rises only slowly, the item is simply hard or vague rather than miskeyed.
  2. Alpha rises only a little. Dropping W8 shortens the scale and may remove content the scale is meant to cover; keeping it costs little precision. A content review should decide, not alpha alone.
  3. Not-reached items would look harder and would correlate with speed rather than knowledge. Treat not-reached items as missing (NA) rather than wrong; IRT estimation handles missing responses naturally.

In the lab

The first two sections of Lab 1, run_lab("model-choice"), put you in charge of scoring a certification exam with 1,200 candidates. You score the responses, screen the items and decide what to do about anything suspicious. See Teaching labs.

Further reading

Crocker and Algina (1986) remain the classic introduction to classical test theory and item analysis. The Standards for Educational and Psychological Testing (American Educational Research Association et al. 2014) set out what evidence a test score needs before it is used.

References

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 2014. Standards for Educational and Psychological Testing. American Educational Research Association.
Crocker, Linda, and James Algina. 1986. Introduction to Classical and Modern Test Theory. Holt, Rinehart; Winston.