Teaching labs for measurement courses

Reading about item response theory is one thing; deciding what to do with a real test is another. The mirtverse labs put students in the analyst’s chair. Each lab is an interactive learnr tutorial that runs in the browser. Students write and run live mirt code, look at the results, and make the calls a working psychometrician makes. Every decision gets immediate feedback that explains why the choice holds up or doesn’t.

The first part of this page is for students. Instructors will find a guide and, at the very end, the answer key.

Start a lab in three steps

install.packages("learnr")   # 1. once, if you don't have it
library(mirtverse)           # 2. load mirtverse
run_lab("model-choice")      # 3. open Lab 1 in your browser

list_labs() shows every lab with a short description. In RStudio, the labs also appear in the Tutorial pane. Your progress is saved on your computer, so you can close the lab and pick up where you left off.

The labs

Labs 1 and 2: the certification exam

You are the new psychometrician for the fictional Northfield Certification Board. Its 32-item, five-option exam was taken by 1,200 candidates in English or French, and the board wants to move from number-correct to IRT scoring. The data (certexam) were simulated, so the truth is known, and a few realistic problems are hidden in them for you to find.

Lab	Run it with	You will
1. Choosing a model the board can defend (about 75 minutes)	run_lab("model-choice")	Score the responses, find a miskeyed item, check dimensionality, compare the Rasch, 2PL and 3PL models, check fit and local dependence, and justify your choice
2. From parameters to decisions (about 75 minutes)	run_lab("decisions")	Read the item parameters, check precision at the pass mark, choose a scoring method, estimate how many candidates are misclassified, test for DIF between languages, write a memo to the board, and see the truth behind the data

Lab 2 fits its own model at the start, so it can be done on its own.

Lab 3: is the test fair?

You work for the fictional Westvale Ministry of Education. A parent group says the Grade 6 numeracy test is biased against English language learners (ELL), who scored three and a half points lower than their classmates. The data (numeracy) cover 2,000 students, 500 of them ELL, on 30 items, and numeracy_items describes each item.

Lab	Run it with	You will
3. Is the test fair? Detecting DIF (about 75 minutes)	run_lab("fairness")	Separate a real group difference from item bias, run Mantel-Haenszel and IRT likelihood-ratio DIF tests, choose clean anchor items, judge how much the DIF matters for students’ scores, plan a content review, and see the truth behind the data

Lab 3 has its own storyline, so it can be done before or after Labs 1 and 2.

Lab 4: linking two forms

You are the psychometrician for the fictional Prairie Nursing Council. Its licensure exam has a new form, and the pass rate jumped from 50% to 72%. Are this year’s candidates better, or is the new form easier? The data (linking) cover 1,000 candidates on each of two 40-item forms that share 12 anchor items.

Lab	Run it with	You will
4. Linking two forms (about 75 minutes)	run_lab("linking")	Calibrate both forms, link them through the common items, catch an anchor item that drifted, compare separate and concurrent calibration, and carry the pass mark over to the new form

Lab 4 also stands on its own.

Lab 5: designing an adaptive test

You work in the testing office of the fictional Lakeshore College, whose 40-item math placement test is too long. Could a computerized adaptive test (CAT) from the college’s 180-item bank (catbank) do the job with fewer items? Students simulate it and find out what else a CAT needs.

Lab	Run it with	You will
5. Designing an adaptive test (about 75 minutes)	run_lab("cat-design")	Look at where the bank is informative, run one CAT by hand, simulate 500 students, and choose a stopping rule, exposure control and content balancing

Lab 5 stands on its own and needs no extra packages: the CAT is a short function written out in the lab.

Lab 6: scoring open-ended responses

You are the psychometrician for the fictional Northfield Grade 9 Science Assessment. Six open-ended tasks were each scored 0 to 4 by two human raters, with an expert third read when they disagreed by two or more points, and by an automated scoring engine (openended). Can the scores be trusted, and could the engine replace the second human reader?

Lab	Run it with	You will
6. Scoring open-ended responses (about 75 minutes)	run_lab("open-ended")	Measure rater agreement, find a severe and an inconsistent rater, resolve final scores, compare the PCM, GPCM and GRM, collapse rubric categories that do not work, and check the engine overall and for English language learners

Lab 6 stands on its own.

What it looks like

Each section mixes short explanations, code exercises with hints and solutions, and decision questions. A wrong answer becomes a teaching moment:

At the end of each lab, you get a table of every decision you made, which you can hand in as an assignment.

For instructors

Labs 1 and 2 follow one storyline, but Lab 2 also works on its own; Labs 3 to 6 stand alone. The tables below list what students do in each section and the decisions they make. The answers are in the answer key at the end of the page. For where the labs fit in a full term, see the course map, a suggested 12-week syllabus.

Lab 1: Choosing a model the board can defend

run_lab("model-choice"), about 75 minutes.

Section	Students do	Decision
The brief	Meet the board and the raw data
Score the responses	key2binary()	Why score before fitting?
Screen the items	Proportion correct, corrected item-total correlations and a distractor analysis	What to do with a suspicious item
How many dimensions?	One- vs two-factor 2PL, anova(), rotated loadings	Is one dimension defensible?
Compare the models	select_model() on Rasch, 2PL, 3PL	Which model to recommend
Check the chosen model	gg_itemfit(), gg_q3()	How to treat flagged item fit and local dependence
Commit to a model	Fit final, write a justification	Free-text justification, then a model answer
Your decisions	A table of every answer the student gave

Lab 2: From parameters to decisions

run_lab("decisions"), about 75 minutes. The board’s 2PL model is fitted in the lab’s setup, so students can start Lab 2 without finishing Lab 1.

Section	Students do	Decision
Read the items	coef(IRTpars = TRUE), gg_trace()	Which item is hardest and what its low slope means; reading a trace line
Precision at the pass mark	Map the raw cut of 18 to theta with the EAP sum-score table; gg_info(), testinfo()	Is the exam precise enough at the cut, and what items would help?
Choose a scoring method	Compare EAP, ML and sum-score scale scores	What goes on candidates’ score reports
Classification accuracy	Misclassification probabilities from each candidate’s score and SE	How to respond to measurement error near the cut
Fairness across languages	Raw means by group, dif(), gg_dif()	Impact versus DIF; what to do with the flagged items
The memo	Free-text memo, then interpret() and item_card()
The reveal	lab_truth(), true versus estimated difficulties	Where the 2PL departs from the true 3PL

Lab 3: Is the test fair? Detecting DIF

run_lab("fairness"), about 75 minutes. Three exercises fit 31 models each and take about a minute; the lab warns students before each one.

Section	Students do	Decision
The brief	Meet the complaint, numeracy and numeracy_items
Impact or bias?	Proportion correct by group for every item	What a raw group difference shows
Match on the total score	Mantel-Haenszel by hand for one item, then ETS A/B/C categories for all	What DIF that favours the focal group means
The IRT approach	dif(), gg_dif(type = "curves")	Why IRT flags an item that Mantel-Haenszel misses
Choose clean anchors	A multiple-group model with only unflagged items as anchors; DIF(scheme = "add")	Which impact estimate to report; which flags were artefacts
How much does it matter?	Differential test functioning from the expected test score curves, and each item’s share	What to tell the deputy minister
Content review	Join the flagged items to the item descriptions	What the review panel should do
The memo	Free-text memo, then interpret()
The reveal	lab_truth("numeracy"), then a gender DIF analysis	Reporting a null result

Lab 4: Linking two forms

run_lab("linking"), about 75 minutes.

Section	Students do	Decision
The brief	Meet the registrar’s question and the NEAT data
Raw scores	Raw means and pass rates by form, and the anchor score	Form, group, or both?
Separate calibration	A 2PL for each form; anchor difficulties side by side	Why the anchors shift; which anchor drifted
Link the scales	Mean-sigma constants with and without the drifted anchor; displacement	What the constants mean; what to do with the drifted anchor
Concurrent calibration	One multipleGroup() model with the clean anchors	Separate or concurrent calibration
Carry the pass mark over	IRT true-score equating with the test characteristic curves	The Form B pass mark; what the drifted anchor would have cost
The memo	Free-text memo to the registrar
The reveal	lab_truth("linking")

Lab 5: Designing an adaptive test

run_lab("cat-design"), about 75 minutes. Each simulation of 500 students takes a few seconds.

Section	Students do	Decision
The brief	Meet the dean’s question and catbank
Where is the bank strong?	Bank and fixed-form information curves	Where a CAT will struggle
One CAT by hand	Find the first item; watch a CAT for a strong and an average student
Simulate the CAT	500 students, CAT versus the fixed test, overall and by ability band	What to tell the dean
Protect the items	Exposure rates; randomesque selection	Why exposure matters; what to do after randomesque
Keep the content balanced	Share of items by area; cycling through the areas	Is balancing worth the extra items?
Choose a stopping rule	SE 0.30, SE 0.25 and a fixed length of 20	The stopping rule
The memo	Free-text memo to the dean
The reveal	lab_truth("catbank")

Lab 6: Scoring open-ended responses

run_lab("open-ended"), about 75 minutes. Each polytomous model takes a few seconds; select_model() fits three.

Section	Students do	Decision
The brief	Meet the director’s question and openended
Do the raters agree?	Exact and adjacent agreement and QWK by task	What to do about T6
Rater by rater	Each rater’s mean and difference from their partner, and exact agreement	Raters R5 and R8
The final scores	Resolve two reads with the third-read rule; build the student-by-task table
A polytomous IRT model	PCM, GPCM and GRM with select_model(); step difficulties and category curves; collapse T4’s categories	Which model; what to do about T4
Can the engine replace a reader?	Engine-human QWK and SMD by task, then SMD for ELL and fluent students	Is the engine ready?
The memo	Free-text memo to the director
The reveal	lab_truth("openended")

Tips for running the labs

• Before class: ask students to install mirtverse and learnr and run list_labs(). Fitting the models needs no internet connection.
• Order: Labs 1 and 2 belong together. Lab 3 works on its own and fits a unit on fairness or validity; Lab 2’s short DIF section is a good warm-up for it. Lab 4 fits a unit on linking and equating, Lab 5 a unit on item banks and adaptive testing, and Lab 6 a unit on polytomous models or performance assessment.
• Speed: most exercises run in under about 15 seconds on a laptop; the two-factor model uses fewer quadrature points so it stays quick. The full DIF analyses in Labs 2 and 3 take up to a minute.
• Hosting: to run a lab in the browser without any installation, deploy the tutorial folder (system.file("tutorials", "model-choice", package = "mirtverse")) to shinyapps.io or Posit Connect. learnr’s event recorder (options(tutorial.event_recorder = ...)) can log students’ answers there.
• Assessment: the free-text justification at the end of each lab makes a good short assignment. Ask students to paste it, together with their decision table, into your course site.

Answer key

This section gives the answers away.

The planted problems

certexam is simulated from item parameters based on a 3PL calibration of mirt’s SAT12 data, so the items behave like real ones. Because the data were generated, the truth is known, and a few problems were planted on purpose:

Planted problem	Where students meet it	What a good analyst does
Q14 is miskeyed in certexam_key	Lab 1, item screening	Spots the negative item-total correlation, checks the key with a distractor analysis, fixes it and rescores
Q21 and Q22 share a reading passage	Lab 1, dimensionality and local dependence	Recognises a two-item “factor” as local dependence, not a second trait; treats the pair as a testlet
Guessing on harder items (the true model is the 3PL)	Lab 1, model comparison	Weighs AIC against BIC, convergence and sample size; 2PL and 3PL are both defensible, Rasch is not
Q09 (uniform) and Q27 (non-uniform) show DIF against French candidates	Lab 2, fairness	Flags both for content review
French candidates score 0.35 SD lower on average	Lab 2, fairness	Separates a real group difference (impact) from DIF

lab_truth() returns the generating values for instructors.

Lab 1

• Why score first: the 2PL and 3PL model the probability of a correct answer.
• Negative item-total correlation: check the key before anything else.
• Real key for Q14: A. About 90% of the top 20% of candidates chose it; the published key says C.
• Dimensions: one. The likelihood-ratio test and AIC favour two factors, BIC favours one, and the second factor is made only of Q21 and Q22.
• Model: the 2PL (lowest BIC, converges, M2 RMSEA about 0.015) or the 3PL (lowest AIC, but it does not converge with default settings and its guessing parameters are poorly determined at N = 1,200). Both are accepted; the feedback names the trade-off. Rasch is rejected by every criterion.
• Item fit: a couple of items have p < .01, but after a false-discovery-rate correction none is flagged at that level, and the misfit is small (RMSEA about 0.03).
• Q21 and Q22: Q3 of about 0.27, more than twice any other pair. Combine them into a testlet score, model the passage, or keep one item; at minimum, report it.

Discussion prompts

• The likelihood-ratio test rejected one dimension. Why is that not enough to conclude the exam measures two traits?
• The true model is the 3PL, yet BIC prefers the 2PL. Is BIC “wrong”? What is model selection trying to find?
• A candidate who failed by one point appeals. Which parts of your analysis would you want to show the appeal panel?

The decision table at the end of Lab 1, for a student who made these choices:

Lab 2

• Hardest item: Q32 (b about 3), with Q12 close behind. Both have low slopes (about 0.45), the 2PL’s way of absorbing guessing.
• Trace line: at theta = 0, Q32’s probability is about 0.2.
• Pass mark: raw 18 corresponds to theta of about 0.05. Information there is about 6 (SE about 0.41), close to the exam’s peak at theta of about -0.5. Medium-difficulty, highly discriminating items would help most.
• Scoring: scale scores from the sum-score table (every candidate with the same raw score gets the same score) are the most defensible for certification; pattern EAP is accepted with the caveat about explaining it. ML is undefined for perfect and zero scores.
• Classification: about 145 of 1,200 candidates (12%) are expected to be misclassified, and about 364 (30%) are within one SE of the cut. Report it, and consider a retest policy or more items near the cut; don’t move the cut.
• Impact: French candidates are estimated 0.38 SD lower (true value 0.35). That is a group difference, not item bias.
• DIF: only Q09 (uniform; about 15 percentage points harder for French candidates at the same ability) and Q27 (non-uniform; the curves cross) are flagged. Send both to bilingual content review.
• Truth: the 2PL departs most from the true difficulties for the hardest items, where guessing matters.

Discussion prompts

• Two candidates both answered 18 items correctly. Under pattern scoring one passes and one fails. How would you explain that to them? Would you?
• Q27’s average (signed) DIF effect is small. Why is that misleading?
• The board’s 2PL was “wrong”, yet most of the decisions it supported were right. When does using a simpler model than the truth matter?

Lab 3

The planted problems in numeracy (lab_truth("numeracy") returns them):

Planted problem	What a good analyst does
ELL students are 0.5 SD lower on average	Treats it as impact, not bias: the clean-anchor model estimates it at 0.55 SD
N07, N18, N20, N25 and N30, long word problems, are harder for ELL students at the same ability (uniform DIF)	Flags them with both methods and sends them to content review
N02, a word problem with an idiom, discriminates poorly for ELL students (non-uniform DIF)	Notices that Mantel-Haenszel misses it and the IRT test catches it
N21, a symbolic item, is easier for ELL students	Flags it for review too, rather than keeping it to offset the others
N03 has a tiny shift	Leaves it alone
N10, N12 and N28 are just as long as the DIF items but show no DIF	Concludes that length alone does not explain the DIF
No gender DIF and no gender impact	Reports the null result with the sample size

• Raw gaps: ELL students do worse on 29 of 30 items (by up to 35 percentage points on N30); only N21 goes the other way. That says nothing about bias yet.
• Mantel-Haenszel: D for N07 is about -2.2. Category C: N07, N20, N25 and N30 (negative) and N21 (positive, about +2.6). Category B: N18 and N15. N02 is in category A, because its curves cross.
• N21: large DIF that favours ELL students. It needs review like the others.
• First IRT pass (dif()): ten items flagged: the seven planted ones plus N05, N15 and N16, which have small positive effects. N02’s esd is near 0 but its uesd is about 0.15: non-uniform DIF.
• Clean anchor: with the 20 unflagged items as anchors, the ELL mean moves from -0.68 to -0.55 SD (true value -0.5), and N05, N15 and N16 are no longer flagged. They were artefacts of a contaminated anchor.
• How much it matters: for an ELL and a fluent student of the same ability, the DIF costs the ELL student about 0.7 of a point out of 30. The five word problems cost about 0.8; N21 gives back about 0.2. Most of the 3.5-point gap is impact. Cancellation does not make the items fair.
• Content review: send all seven items to a panel with ELL and numeracy specialists, and revise wording in plain language. Length alone is not the explanation, since N10, N12 and N28 are just as long. Field-test the revised items and check them for DIF again.
• Gender: no item is flagged and the means are almost equal. Report “no evidence of gender DIF” with the sample size; it is not proof of none.

Discussion prompts

• Mantel-Haenszel and the IRT test disagreed about N02. When would you prefer each method?
• The first pass flagged three clean items. What would have happened if the ministry had removed every flagged item without a second pass?
• N21 favours ELL students and partly cancels the other DIF. Why is “it all balances out” not a good defence of a test?
• A plain-language revision of N07 is field-tested and still shows DIF. What would you look at next?

Lab 4

The planted problems in linking (lab_truth("linking") returns them):

Planted problem	What a good analyst does
The 2026 candidates are more able: mean 0.3, SD 1.1 on the Form A scale	Recovers it from the anchors: about 0.36 and 1.12 (mean-sigma) or 0.37 and 1.15 (concurrent)
Form B’s unique items are easier	Finds Form B about 4 points easier at the pass mark
Anchor C07 leaked and is 1.3 easier on Form B	Spots it in the plot of anchor difficulties and by its displacement, and links without it

• Raw scores: Form B candidates score 5.9 points higher on the form but only 1.1 higher on the 12 anchors, so both the group and the form differ.
• Why the anchors shift: each separate calibration puts its own group’s mean at 0, and the 2026 group is more able.
• Drift: C07 is about 1.5 easier on the Form B scale; the other anchors shift by 0.6 or less. After linking, C07’s displacement is about -1.2 and every other anchor is within ±0.4.
• Mean-sigma: with the 11 clean anchors, A is about 1.12 and B about 0.36. With C07, B rises to about 0.45, making the 2026 group look more able than it is.
• Concurrent calibration: Form B group mean 0.37 and SD 1.15, close to the mean-sigma result. Either method is defensible; agreement is evidence for the link.
• Pass mark: 22 on Form A is theta of about 0.10, which corresponds to 26.1 on Form B, so the Form B pass mark is 26 (60% pass, compared with 72% at 22). The remaining rise from 50% is real.
• The cost of drift: with C07 in the anchor, the equated pass mark is 25.0, and 38 more candidates would pass.

Discussion prompts

• Why can’t the council just keep the pass rate at 50% every year?
• An anchor can drift because it leaked, because the curriculum changed, or because it moved position in the test. How would you tell these apart?
• How many anchors would you want for a 40-item test, and where should they sit in difficulty?
• What would change if the two groups had been randomly equivalent (for example, forms spiralled within one administration)?

Lab 5

The planted problems in catbank (lab_truth("catbank") returns them):

Planted problem	What a good analyst does
Only one item is harder than b = 1.5	Sees it in the bank information curve, predicts long and imprecise CATs for strong students, and asks for hard items
Algebra items discriminate better than the others	Notices that an unconstrained CAT is about 70% Algebra and adds content balancing
Five exceptional Algebra items near the middle (M012, M024, M051, M076, M082)	Spots their exposure; sees that randomesque alone cannot fix it, and recommends more items like them and a stronger exposure control

• Where the bank is weak: above theta of about 1.5.
• First item: M012, the most informative item at theta = 0, which every student gets.
• One CAT by hand: the average student finishes in 9 items; the strong student (theta 1.8) uses all 30 and ends with SE 0.35.
• CAT versus the fixed test: about 15 items on average against 40, with about the same overall precision (RMSE 0.32 against 0.30). The CAT is more precise for weak and strong students (SE 0.30 and 0.35 against 0.35 and 0.43), but students above theta 1.5 take about 29 items.
• Exposure: M012 is seen by every student, 16 items by more than 30%, and about half the bank by nobody. Randomesque with k = 5 brings the maximum down to about 75% at almost no cost; the long-term fix is more items of similar quality, plus a probabilistic cap such as Sympson-Hetter.
• Content: about 70% of an unconstrained CAT is Algebra. Cycling through the areas brings Algebra to 27% and adds about 5 items per student, which is worth it.
• Stopping rule: SE 0.30 with at most 30 items (about 21 items with balancing) is a sound choice; SE 0.25 needs about 31 items and strong students hit 40; a fixed length of 20 leaves strong students with SE 0.41.

Discussion prompts

• Why does every student get the same first item, and what could you do about it?
• The college could also use a multistage test (MST) instead of a CAT. What would it gain and lose?
• How would you decide how many hard items to commission, and at what difficulty?
• A student complains that their test was 30 items long and their friend’s only 12. How would you explain it?

Lab 6

The planted problems in openended (lab_truth("openended") returns them):

Planted problem	What a good analyst does
Rater R5 scores 0.6 points too low	Compares each rater with their partner on the same responses, finds R5 about 0.5 below, and recommends recalibration (or a many-facet Rasch adjustment)
Rater R8 is much noisier than the others	Sees R8’s low exact agreement despite an average mean, and recommends retraining or removal, since noise cannot be adjusted away
T6 has a vague rubric	Sees low agreement on one task for every rater and asks for a clearer rubric with anchor papers
T4’s rubric levels 1 and 2 are hard to tell apart	Reads the disordered steps and category curves, collapses 1 and 2, and asks for new descriptors
The engine scores ELL students 0.4 points too low	Looks past the overall statistics to the subgroup SMD and keeps human second reads
ELL students are 0.3 SD lower in ability	Separates this real difference (impact) from the engine’s bias

• Agreement: QWK is 0.82 to 0.87 on T1 to T5 and 0.68 on T6, with 43% exact agreement on T6 against about 60% elsewhere. Adjacent agreement is at least 96% except on T6 (86%). T6’s rubric is the problem, not a rater.
• Raters: R5’s mean score is about 1.47 against about 2.0 for the others, and R5 is 0.53 below their partners on the same responses. R8’s mean is normal but their exact agreement is 46% against about 61%.
• Final scores: about 5% of responses needed a third read.
• Model: the GPCM has the lowest BIC (20454), then the GRM (20487) and the PCM (20496); the likelihood-ratio test also prefers the GPCM over the PCM. Discriminations range from about 0.7 (T2) to 1.5 (T3).
• T4: b1 = -0.11 is above b2 = -1.00, so a score of 1 is never the most likely. After collapsing 1 and 2 the steps are -0.95, 0.43 and 1.34.
• Engine: engine-human QWK is 0.81 to 0.89 and the SMD -0.08 to -0.14, so it passes the overall checks. Within ELL students the SMD is about -0.33 (fluent students about -0.04), well beyond ±0.10. It is not ready to replace a human reader.

Discussion prompts

• The engine’s QWK is higher than the human-human QWK. Why is that comparison unfair to the humans?
• R5’s severity partly cancels out because every response had a second reader. When would it matter more?
• Should students whose responses were read by R8 be rescored? How would you decide?
• Collapsing categories changes the score scale for T4. What has to happen before next year’s administration, and how would you explain the change to teachers?