Chapter 7 Scoring people

Learning goals. After this chapter you can

  • compare EAP, MAP, ML and WLE scores and say when each is a sensible choice;
  • explain why maximum likelihood fails for perfect and zero scores;
  • convert sum scores to IRT scores with a conversion table;
  • read a Wright map;
  • estimate how often people are classified on the wrong side of a cut score.

Key terms: person score, EAP, MAP, ML, WLE, Wright map, classification accuracy.

7.1 The ideas

Once the item parameters are known, we can estimate each person’s \(\theta\) from their responses. The likelihood of a response pattern, viewed as a function of \(\theta\), is the product of the item probabilities: \(P_j(\theta)\) for each item answered correctly and \(1 - P_j(\theta)\) for each item answered wrongly. The four common estimators use it differently.

  • ML (maximum likelihood) takes the \(\theta\) that maximizes the likelihood. It uses only the responses, which makes it attractive for fairness arguments, but it does not exist for all-correct or all-wrong patterns (the likelihood keeps rising towards \(\pm\infty\)), and it is biased outwards for short tests.
  • WLE (weighted likelihood, Warm (1989)) adds a small correction that removes most of ML’s bias and gives finite estimates for perfect patterns.
  • MAP (maximum a posteriori) multiplies the likelihood by a prior distribution for \(\theta\), usually the population distribution, and takes the mode of the result.
  • EAP (expected a posteriori) takes the mean of that posterior. It is easy to compute, always finite and has the smallest average error, which is why it is mirt’s default. Like MAP, it shrinks estimates towards the population mean, more so when the test is short.

The standard error of an EAP estimate is the standard deviation of the posterior, and it comes for free.

For the Rasch model the sum score carries all the information about \(\theta\), so everyone with the same sum score gets the same estimate. For the 2PL and 3PL, two people with the same sum score can get different estimates, because correct answers to steep items count for more. Some programmes still want one score per sum score. EAP for sum scores (method = "EAPsum") gives exactly that, and produces a conversion table that can be printed in a manual.

7.2 Example 1: Four estimators, one short test

With only five items, the differences between estimators are easy to see.

# Five dichotomous LSAT items, one row per test taker.
resp <- expand.table(LSAT7)
mod <- mirt(resp, 1, itemtype = "2PL", verbose = FALSE)
patterns <- unique(resp)
est <- sapply(c("EAP", "MAP", "ML", "WLE"), function(m)
  fscores(mod, method = m, response.pattern = patterns)[, "F1"])
scored <- data.frame(pattern = apply(patterns, 1, paste, collapse = ""),
                     sum = rowSums(patterns), round(est, 2))
scored[order(scored$sum, scored$EAP), ][c(1:4, 15:18, 29:32), ]
#>     pattern sum   EAP   MAP    ML   WLE
#> 1     00000   0 -1.87 -1.82  -Inf -4.14
#> 13    00001   1 -1.53 -1.49 -3.12 -2.60
#> 32    00010   1 -1.51 -1.48 -3.07 -2.56
#> 173   10000   1 -1.41 -1.39 -2.71 -2.31
#> 264   10100   2 -0.65 -0.69 -1.17 -1.15
#> 107   01100   2 -0.61 -0.65 -1.11 -1.10
#> 230   10011   3 -0.75 -0.78 -1.31 -1.26
#> 100   01011   3 -0.70 -0.74 -1.25 -1.21
#> 145   01111   4  0.14  0.06  0.13 -0.22
#> 525   11101   4  0.27  0.18  0.43 -0.03
#> 661   11110   4  0.28  0.20  0.47 -0.01
#> 693   11111   5  0.73  0.64   Inf  1.09

Read the table from top to bottom. For the all-wrong pattern ML gives -Inf, and for the all-correct pattern Inf; WLE, MAP and EAP give finite values. EAP and MAP are pulled towards zero compared with ML and WLE. And patterns with the same sum score get different scores under the 2PL, because Item.3 (the steepest) counts for more than Item.5.

The sum-score conversion table:

fscores(mod, method = "EAPsum", full.scores = FALSE)
#>   Sum.Scores     F1 SE_F1 observed expected std.res
#> 0          0 -1.870 0.693       12     10.1   0.601
#> 1          1 -1.432 0.684       40     44.7   0.697
#> 2          2 -0.949 0.694      114    109.8   0.403
#> 3          3 -0.413 0.721      205    207.7   0.191
#> 4          4  0.152 0.759      321    319.2   0.102
#> 5          5  0.727 0.801      308    308.6   0.031

gg_scores() shows the distribution of the estimated scores, and gg_wright() puts people and items on the same scale. With five easy items the Wright map shows the problem at once: The items sit low on the scale, where few people need to be measured.

gg_scores(mod) + gg_wright(mod)
Distribution of the estimated scores (left) and the Wright map of people and items (right) for LSAT7.

Figure 7.1: Distribution of the estimated scores (left) and the Wright map of people and items (right) for LSAT7.

7.3 Example 2: A proficiency cut on a mathematics test

We simulate a 40-item end-of-year mathematics test for 2,000 students. The province calls students at or above \(\theta = 0.8\) proficient. Because the data are simulated we know each student’s true \(\theta\), so we can see how often the estimates put students on the wrong side of the cut.

# Simulated 40-item mathematics test; true_theta is known for checking.
set.seed(707)
true_theta <- rnorm(2000)
a <- matrix(rlnorm(40, 0.1, 0.25)); b <- rnorm(40, 0, 1)
resp <- simdata(a, matrix(-a * b), N = 2000, itemtype = "2PL",
                Theta = matrix(true_theta))
colnames(resp) <- sprintf("M%02d", 1:40)
mod <- mirt(resp, 1, itemtype = "2PL", verbose = FALSE)
eap <- fscores(mod, method = "EAP", full.scores.SE = TRUE)
wle <- fscores(mod, method = "WLE", full.scores.SE = TRUE)
c(cor_eap = cor(eap[, 1], true_theta), cor_wle = cor(wle[, 1], true_theta),
  sd_eap = sd(eap[, 1]), sd_wle = sd(wle[, 1]))
#> cor_eap cor_wle  sd_eap  sd_wle 
#>   0.945   0.945   0.947   1.049

With 40 items both estimators correlate about 0.95 with the true values, and they rank students almost identically. The EAP scores have a smaller spread than the true values (whose standard deviation is 1) and the WLE scores a slightly larger one: Shrinkage on one side, measurement error on the other.

Now the classification. A student is classified correctly if the estimate and the true value fall on the same side of the cut.

cut <- 0.8
truth <- true_theta >= cut
observed <- eap[, 1] >= cut
table(true = truth, estimated = observed)
#>        estimated
#> true    FALSE TRUE
#>   FALSE  1503   58
#>   TRUE     81  358
mean(truth == observed)
#> [1] 0.93

In practice we never know the truth, but we can estimate the accuracy from the scores and their standard errors (Rudner 2001). If a student’s posterior is roughly normal with mean \(\hat\theta\) and standard deviation \(SE(\hat\theta)\), the probability that their true \(\theta\) is above the cut is \(1 - \Phi\left((c - \hat\theta)/SE\right)\). Averaging the probability of the side they were classified on gives the expected accuracy:

p_above <- pnorm((eap[, 1] - cut) / eap[, 2])
expected_accuracy <- mean(ifelse(observed, p_above, 1 - p_above))
expected_accuracy
#> [1] 0.935

The estimate is close to the accuracy we observed against the truth. The misclassified students are, of course, the ones near the cut:

library(ggplot2)
ggplot(data.frame(true = true_theta, est = eap[, 1], wrong = truth != observed),
       aes(true, est, colour = wrong)) +
  geom_point(alpha = 0.5, size = 1) +
  geom_vline(xintercept = cut, linetype = 2) +
  geom_hline(yintercept = cut, linetype = 2) +
  scale_colour_manual(values = c(`FALSE` = "grey60", `TRUE` = "#C0392B"),
                      labels = c("correct", "misclassified")) +
  labs(x = "True theta", y = "EAP estimate", colour = NULL) +
  theme_mirtverse()
True and estimated ability on the mathematics test; misclassified students lie near the cut.

Figure 7.2: True and estimated ability on the mathematics test; misclassified students lie near the cut.

augment() returns the scores, their standard errors and the response data as one table, ready to export:

head(augment(mod)[, c("M01", "M02", ".F1", ".se_F1")])
#> # A tibble: 6 × 4
#>     M01   M02    .F1 .se_F1
#>   <int> <int>  <dbl>  <dbl>
#> 1     0     0 -1.70   0.384
#> 2     1     0 -0.246  0.304
#> 3     1     0  0.368  0.300
#> 4     1     0  1.18   0.319
#> 5     0     0 -0.746  0.319
#> 6     0     0 -1.09   0.336

7.4 Your turn

  1. In Example 1, find two response patterns with a sum score of 3 that get different EAP scores. Which items explain the difference?
  2. Fit the Rasch model to LSAT7 and repeat the EAP table. Do patterns with the same sum score still get different scores?
  3. In Example 2, compute the expected classification accuracy using only the first 15 items. How much does it drop?
  4. A parent asks why their child, who got 30 of 40 right, has a lower score than a classmate who also got 30 right. Write a short answer.
Solutions
subset(scored, sum == 3)
#>     pattern sum   EAP   MAP    ML   WLE
#> 65    00111   3 -0.41 -0.47 -0.82 -0.88
#> 100   01011   3 -0.70 -0.74 -1.25 -1.21
#> 114   01101   3 -0.26 -0.32 -0.59 -0.71
#> 137   01110   3 -0.24 -0.31 -0.56 -0.69
#> 230   10011   3 -0.75 -0.78 -1.31 -1.26
#> 278   10101   3 -0.30 -0.37 -0.66 -0.76
#> 329   10110   3 -0.29 -0.35 -0.63 -0.75
#> 440   11001   3 -0.60 -0.64 -1.09 -1.09
#> 465   11010   3 -0.59 -0.63 -1.07 -1.08
#> 507   11100   3 -0.13 -0.20 -0.38 -0.56

rasch <- mirt(expand.table(LSAT7), 1, itemtype = "Rasch", verbose = FALSE)
r_eap <- fscores(rasch, response.pattern = patterns)[, "F1"]
tapply(round(r_eap, 4), rowSums(patterns), function(x) length(unique(x)))
#> 0 1 2 3 4 5 
#> 1 1 1 1 1 1

short <- mirt(resp[, 1:15], 1, itemtype = "2PL", verbose = FALSE)
e15 <- fscores(short, full.scores.SE = TRUE)
p15 <- pnorm((e15[, 1] - cut) / e15[, 2])
mean(ifelse(e15[, 1] >= cut, p15, 1 - p15))
#> [1] 0.898
  1. No: Under the Rasch model every pattern with the same sum score gets the same estimate.
  2. For example: “The test gives more weight to questions that are better at telling apart students who have mastered the material from those who have not. Your child and their classmate answered different questions correctly, so their scores differ slightly even though their number correct is the same.”

In the lab

Lab 2, run_lab("decisions"), asks you to choose a scoring method for a certification board, estimate how many candidates are misclassified at the pass mark, and write a memo about it.

Further reading

Embretson and Reise (2000) compare the scoring methods; Warm (1989) introduced the weighted likelihood estimator; Rudner (2001) the approach to expected classification accuracy used here.

References

Embretson, Susan E., and Steven P. Reise. 2000. Item Response Theory for Psychologists. Lawrence Erlbaum Associates.
Rudner, Lawrence M. 2001. “Computing the Expected Proportions of Misclassified Examinees.” Practical Assessment, Research, and Evaluation 7 (14).
Warm, Thomas A. 1989. “Weighted Likelihood Estimation of Ability in Item Response Theory.” Psychometrika 54 (3): 427–50.