Chapter 17 Large language models as a colleague

Learning goals. After this chapter you can

  • describe what a large language model (LLM) does and why its output varies and can be wrong;
  • use an LLM through mirtverse to draft analysis code, write items and score responses, and see exactly what is sent to it;
  • explain why every LLM output in testing needs a check, and use mirtverse’s checks for wrong keys, weak distractors and machine-human agreement;
  • make LLM work reproducible by recording and replaying responses, and say what to report about an LLM in a technical manual or paper.

Key terms: large language model, prompt, hallucination, human-in-the-loop, automatic item generation, automated scoring.

17.1 The ideas

A large language model is a neural network trained on a very large amount of text to predict the next piece of text. Given instructions and context, the prompt, it continues them, and that one ability is enough to write items, explain results, score essays against a rubric or draft code. Claude, GPT and the open models that run on a laptop through Ollama all work this way.

For a psychometrician, three properties matter more than how the network works:

  • Its output is a sample. The same prompt can give different answers on different runs and with different versions of the model. An LLM is not a function with a fixed value; it behaves more like a rater.
  • It can be fluently wrong. A model can write an item with no correct option, invent a reference or misread a rubric, in the same confident style as a correct answer. Such errors are called hallucinations, and they are found only by checking.
  • What you send leaves your computer. Unless the model runs locally, the prompt goes to a provider. Test content and response data may be confidential, so what is sent has to be a decision, not an accident.

These properties lead to the way mirtverse uses LLMs, which is also a good rule for any testing program (Hao et al. 2024; Bulut et al. 2024): The model drafts, something else checks, and a person decides. Generated items are answered again by a second, independent model call that does not see the key. Machine scores must agree with human scores, overall and within groups, before they count, following the framework of Williamson, Xi and Breyer (2012). Anything the model is unsure about goes to a person.

mirtverse talks to models through the ellmer package. mv_ask_setup() chooses a provider once: Claude or OpenAI with your own API key, or Ollama for a free model that runs on your own computer. This chapter instead runs in replay mode: mv_llm_replay() answers every request from a file of recorded responses that comes with mirtverse, so the examples need no key, no internet connection and give the same results for every reader. Those responses were written by hand to show what a model returns, including a few deliberate mistakes; they were not produced by a live model.

17.2 Example 1: From a sentence to a tested script

mv_ask() turns a plain-language request into an analysis plan and the plan into an R script built from tested templates. The model never writes code and never sees the responses: It sees the request and a summary of the columns. You can see that payload before anything is sent with dry_run = TRUE:

# mirt's Science data: 392 respondents, 4 attitude items scored 1 to 4.
resp <- Science
request <- "Fit a graded response model to these attitude items, check the fit and explain the results."
mv_ask(request, data = resp, dry_run = TRUE)
#> This would be sent to recorded responses:
#> 
#> Request: Fit a graded response model to these attitude items, check the fit and explain the results.
#> 
#> Data: 392 rows, 4 columns.
#> Columns (name | type | distinct values | min | max | share missing | guessed role | categories):
#> - Comfort | integer | 4 | 1 | 4 | 0.00 | item | 
#> - Work | integer | 4 | 1 | 4 | 0.00 | item | 
#> - Future | integer | 4 | 1 | 4 | 0.00 | item | 
#> - Benefit | integer | 4 | 1 | 4 | 0.00 | item |

The payload holds the request, the number of rows and, for each column, its type, range, share missing and the role mirtverse guesses for it. With anonymize = TRUE the column names are replaced by codes as well. Now the real call, answered from the recording:

s <- mv_ask(request, data = Science, ask = FALSE)
s$parser
#> [1] "recorded responses"
s$plan$model
#> [1] "graded"
s$plan$assumptions
#> [1] "All four columns (Comfort, Work, Future, Benefit) are items scored 1 to 4."
#> [2] "No grouping column was mentioned, so no DIF analysis is included."         
#> [3] "Used the 4 columns that look like item responses as items."

The model chose the graded response model, kept the default steps, and stated two assumptions it made. Read them: An assumption is where a model fills a gap in your request, and a wrong one gives a wrong analysis that runs without error. The script itself is printed with s, and run with run_script(s).

17.3 Example 2: Writing items, and catching the mistakes

The assessment pipeline writes items for the cells of a blueprint. Each multiple-choice item comes with a key and a rationale for every option, and is then checked by a second call that answers it blind.

# A two-cell blueprint for a short Grade 7 numeracy quiz.
cells <- data.frame(content = c("Ratios and rates", "Measurement"),
                    cognitive = c("Apply", "Apply"), n = c(2, 2))
p <- mv_pipeline("Grade 7 numeracy", verbose = FALSE)
p <- set_blueprint(p, cells, construct = "Grade 7 numeracy", use = "classroom diagnostic",
                   population = "Grade 7 students")
p <- suppressWarnings(generate_items(p, n = 2))
p$items[, c("id", "stem", "key", "verify_answer", "key_check")]
#> # A tibble: 4 × 5
#>   id    stem                                       key   verify_answer key_check
#>   <chr> <chr>                                      <chr> <chr>         <chr>    
#> 1 G001  A recipe uses 3 cups of flour for every 2… A     A             agree    
#> 2 G002  A cyclist rides 45 km in 3 hours at a ste… C     C             agree    
#> 3 G003  A garden is 12 m long and 7 m wide. How m… A     B             disagree 
#> 4 G004  A water bottle holds 2.5 litres. How many… B     B             agree

Two of the four items have problems. The writer keyed the garden item (G003) as 84 m, which is the garden’s area, but the question asks for the fence around it, the perimeter of 38 m. The checking call answered B and the item is flagged. In the millilitre item (G004), one distractor is a length, not a volume, so no test taker would choose it:

f <- flags(p, stage = "generation")
f[, c("check", "target", "level", "value")]
#> # A tibble: 4 × 4
#>   check      target                   level  value                              
#>   <chr>      <chr>                    <chr>  <chr>                              
#> 1 key_check  G003                     block  key A, verifier chose B; Fencing a…
#> 2 distractor G004                     review D (implausible)                    
#> 3 coverage   Ratios and rates / Apply review 2 items for a target of 2          
#> 4 coverage   Measurement / Apply      review 2 items for a target of 2
p$items$verify_note[3]
#> [1] "Fencing around the garden is the perimeter, 2 x (12 + 7) = 38 m (option B); 84 m is the area, so the key appears to be wrong."

A block flag stops the pipeline at the next stage until a person fixes the item or acknowledges the flag. The coverage flags are a reminder that items are lost in review and field testing, so a pool needs more items than the blueprint asks for. Notice what the checks do not do: They do not show that the items measure Grade 7 numeracy, that they are fair, or that their difficulty suits the test. Content review and a field test still do that. Items written one by one also have no family: Each needs its own field test. Template-based generation from item models (Chapter 15) trades variety for items that can be calibrated as a family.

17.4 Example 3: Is a recording reproducible?

Replay makes a session reproducible in a strong sense: The same calls on the same data give the same output, on any computer. It also makes the limits of a recording visible. Change the request by one word and the recording has no answer for it:

tryCatch(
  suppressWarnings(generate_items(set_blueprint(mv_pipeline(verbose = FALSE), c(Algebra = 2)),
                                  n = 1)),
  error = function(e) conditionMessage(e))
#> [1] "This request is not in the recording examples.json. A recording answers only the exact calls it was made from. Use a live model (mv_llm_replay(FALSE), then mv_ask_setup()), or record the call with mv_llm_record()."

For your own work, mv_llm_record("session.json") saves every request and response while you use a live model, and mv_llm_replay("session.json") plays them back. Keep the file with your analysis, as you would keep the data and the code. When you report work that used an LLM, say which model and version you used and when, give the prompts (mirtverse’s are in its source code and in the recording), say what was sent and what was not, and describe how the outputs were checked.

Experimental: Language models as respondents. Some researchers ask a language model to answer items as a described person would (“a Grade 7 student who struggles with mathematics”) and treat the answers as a “silicon sample” (Argyle et al. 2023). If that worked, items could be tried out before a field test. simulate_llm_respondents() lets you try it, and compare_synthetic() checks the result against real data. Here twelve personas answer one item from each of the twelve families in itemtext, whose field test gives the real proportions correct (p_field). The recorded answers were written to show the problems that studies of simulated respondents report:

items <- itemtext[endsWith(itemtext$id, "-01"), ]
personas <- c(
  "A Grade 7 student who struggles with mathematics.",
  "A Grade 7 student who struggles with mathematics and is still learning English.",
  "A Grade 7 student who finds fractions and decimals hard but tries every question.",
  "A Grade 7 student who often rushes and makes careless mistakes.",
  "A Grade 7 student who is about average in mathematics.",
  "A Grade 7 student who is about average in mathematics and is still learning English.",
  "A Grade 7 student with average marks who is anxious on tests.",
  "A Grade 7 student who is about average in mathematics and likes word problems.",
  "A Grade 7 student who is strong in mathematics.",
  "A Grade 7 student who is strong in mathematics and is still learning English.",
  "A Grade 7 student in an enriched mathematics program.",
  "A Grade 7 student who is strong in mathematics but reads slowly.")
syn <- simulate_llm_respondents(items, personas)
compare_synthetic(syn, setNames(items$p_field, items$id))
#> <mv_synthetic_comparison> 12 simulated respondents, 12 items (EXPERIMENTAL)
#>    item p_real p_synthetic difference
#>  N01-01   0.54        1.00       0.46
#>  N02-01   0.34        1.00       0.66
#>  N03-01   0.62        1.00       0.38
#>  N04-01   0.41        0.42       0.01
#>  N05-01   0.62        1.00       0.38
#>  N06-01   0.39        0.92       0.53
#>  N07-01   0.22        0.67       0.45
#>  N08-01   0.28        0.17      -0.11
#>  N09-01   0.32        0.92       0.60
#>  N10-01   0.62        1.00       0.38
#>  N11-01   0.36        0.17      -0.19
#>  N12-01   0.58        1.00       0.42
#> 
#> * The simulated respondents find the items easier than real test
#>   takers: Mean proportion correct 0.77 against 0.44.
#> * They order the items by difficulty only loosely like real test takers
#>   do (Spearman correlation 0.68), so they cannot stand in for a field
#>   test of difficulty.
#> * Their answers vary little: 5 distinct response patterns among 12
#>   simulated respondents. Respondents described alike answer alike.
#> * Simulated respondents are experimental: Use them to find unclear
#>   items or to plan a study, not to estimate item parameters or to
#>   replace a field test.

Even “struggling” personas get most items right, so the items look far easier than they are. The short items, such as adding fractions (N06-01) and comparing decimals (N09-01), look easy to the model whatever their real difficulty, and the long word problems look hard. Personas described alike answer alike, and the ones described as “still learning English” miss every long word problem whatever their mathematics: A stereotype, not a measurement. Simulated respondents can help find unclear wording or plan a study. They cannot estimate item parameters, check fairness or replace a field test, and mirtverse labels them experimental for that reason.

17.5 Where do synthetic respondents land?

compare_synthetic() compares summaries, such as proportions correct. Summaries can look acceptable while most of the real population is missing from the synthetic sample. The coverage strip shows where synthetic respondents land. synthetic_coverage() cuts the real score distribution into ten slices of equal size (deciles) and counts how many synthetic respondents fall into each, and gg_coverage() draws one row per attempt, so you can see whether a pattern repeats.

mirtverse includes the data of a study of synthetic personality data: hexaco_real holds the answers of 17,874 real people from Canada, Great Britain and the United States to the 240 IPIP-HEXACO items (Ashton et al. 2007), and hexaco_llm holds 59,611 synthetic respondents from four open-weight language models, generated for the same countries in ten attempts each. To keep the example small, we use the 40 Honesty-Humility items, Canada only, and the first four attempts. Answers are stored as given, so the reverse-keyed items are reversed first:

h <- hexaco_items$item[hexaco_items$trait == "H"]       # 40 items
rev_items <- intersect(h, hexaco_items$item[hexaco_items$reverse])
keyed <- function(d) {
  d <- d[h]
  d[rev_items] <- 8 - d[rev_items]
  d
}
real <- keyed(hexaco_real[hexaco_real$country == "Canada", ])
ca <- hexaco_llm[hexaco_llm$country == "Canada" & hexaco_llm$attempt <= 4, ]
# One list per model, with one data frame per attempt
synthetic <- lapply(split(ca, ca$model),
                    function(m) lapply(split(m, m$attempt), keyed))
cov <- synthetic_coverage(synthetic, real, cor = "pearson")
gg_coverage(cov)
Coverage strip for Honesty-Humility in Canada: Four language models, four attempts each.

Figure 17.1: Coverage strip for Honesty-Humility in Canada: Four language models, four attempts each.

Columns are deciles of the real Honesty-Humility score, from the lowest 10% on the left to the highest 10% on the right. Blue cells hold too many synthetic respondents, red cells too few, and pale cells are within what a real sample of the same size would show by chance. The two numbers on the right summarize each model:

  • Overlap is the share of synthetic respondents who land in the same deciles as real respondents. 100% means the same mix as the real population; a sample that puts everyone into one decile scores about 10%.
  • Structure is how closely the correlations between items match the real ones (100% means identical).

Every model puts far too many synthetic Canadians in the top decile and too few in most other deciles: The synthetic people are more honest and modest than real people, and more alike. The rows within each model look the same, so this is how the models behave, not one unlucky run. Note also that the model with the best Structure has the worst Overlap. A model can know which items go together and still put its respondents in the wrong place, which is why checks based only on correlations, such as reliability or factor structure, are not enough. The article Checking synthetic respondents shows the other options of the strip: Groups such as countries, single items and their answer categories, and the Best Blend, a mix of models chosen on some attempts and checked on others.

17.6 Your turn

  1. Run mv_ask(request, data = Science, dry_run = TRUE, anonymize = TRUE). What is still sent? Is anything in it confidential for your own data?
  2. List the recordings that come with mirtverse with mv_llm_recordings(). Which mirtverse functions made them?
  3. In Example 2, fix the miskeyed item with p$items$key[3] <- "B" and acknowledge the flag with acknowledge(). Why does mirtverse ask for a person to acknowledge a flag rather than clear it automatically?
  4. Your program wants an LLM to score 20,000 short answers next spring. Name three pieces of evidence you would ask for before the machine scores count, and one group of test takers you would check separately.
Solutions
mv_ask(request, data = Science, dry_run = TRUE, anonymize = TRUE)
#> This would be sent to recorded responses:
#> 
#> Request: Fit a graded response model to these attitude items, check the fit and explain the results.
#> 
#> Data: 392 rows, 4 columns.
#> Columns (name | type | distinct values | min | max | share missing | guessed role | categories):
#> - V1 | integer | 4 | 1 | 4 | 0.00 | item | 
#> - V2 | integer | 4 | 1 | 4 | 0.00 | item | 
#> - V3 | integer | 4 | 1 | 4 | 0.00 | item | 
#> - V4 | integer | 4 | 1 | 4 | 0.00 | item |
mv_llm_recordings()[, c("function_name", "prompt_start")]
#> # A tibble: 26 × 2
#>   function_name  prompt_start                                                  
#>   <chr>          <chr>                                                         
#> 1 mv_ask         "Request: Fit a graded response model to these attitude items"
#> 2 generate_items "Construct: Grade 7 numeracy Test takers: Grade 7 students Sc"
#> 3 generate_items "Construct: Grade 7 numeracy Test takers: Grade 7 students Sc"
#> 4 verify_items   "A recipe uses 3 cups of flour for every 2 cups of milk. How "
#> 5 verify_items   "A cyclist rides 45 km in 3 hours at a steady speed. How far "
#> 6 verify_items   "A garden is 12 m long and 7 m wide. How many metres of fenci"
#> # ℹ 20 more rows
  1. The request (with column names coded), the number of rows and the summary of each column (type, distinct values, range, share missing, guessed role) are still sent. Category labels are left out. Individual responses are never sent. For a confidential test, the request itself or the column names may give away content, which is why anonymize = TRUE exists; a local model through Ollama sends nothing at all.
  2. mv_ask(), generate_items() (twice, one call per blueprint cell), verify_items() (once per item, called by generate_items()) and score_open() (once per batch of 10 responses, from the example in system.file("replay", "examples.R", package = "mirtverse")) and simulate_llm_respondents() (once per persona, from the box on simulated respondents).
  3. A check can only say that something looks wrong; deciding whether it is, and what to do, is a professional judgement. Recording who acknowledged which flag, and why, is part of the evidence for the test.
  4. For example: Agreement between machine and human scores on a sample of double-scored responses, compared with agreement between two humans (Williamson et al. 2012); the standardized mean difference between machine and human scores; and the same statistics within groups, such as English language learners, whose answers may be shorter or phrased differently at the same level of understanding. score_open() reports all of these.

In the AI Hub

The AI Hub shows the AI features of mirtverse in one place. Lab 7, run_lab("auto-scoring"), trains and checks an automated scoring engine; Lab 8, run_lab("item-text"), predicts item difficulty from the text; Lab 9, run_lab("aig"), generates items from item models and calibrates their families; Lab 10, run_lab("ai-audit"), audits an AI-scored test before its scores are released; and the mv_ask() guide explains what mv_ask() sends and how it was benchmarked.

Further reading

Hao et al. (2024) describe what large language models change in testing, from item development to score reporting. Bulut et al. (2024) discuss the opportunities and the ethical questions, and Williamson, Xi and Breyer (2012) remain the reference for evaluating automated scores.

References

Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples.” Political Analysis 31 (3): 337–51. https://doi.org/10.1017/pan.2023.2.
Ashton, Michael C., Kibeom Lee, and Lewis R. Goldberg. 2007. “The IPIP–HEXACO Scales: An Alternative, Public-Domain Measure of the Personality Constructs in the HEXACO Model.” Personality and Individual Differences 42 (8): 1515–26. https://doi.org/10.1016/j.paid.2006.10.027.
Bulut, Okan, Maggie Beiting-Parrish, Jodi M. Casabianca, et al. 2024. “The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges.” Chinese/English Journal of Educational Measurement and Evaluation, ahead of print. https://doi.org/10.59863/MIQL7785.
Hao, Jiangang, Alina A. von Davier, Victoria Yaneva, Susan Lottridge, Matthias von Davier, and Deborah J. Harris. 2024. “Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI.” Educational Measurement: Issues and Practice 43 (2): 16–29. https://doi.org/10.1111/emip.12602.
Williamson, David M., Xiaoming Xi, and F. Jay Breyer. 2012. “A Framework for Evaluation and Use of Automated Scoring.” Educational Measurement: Issues and Practice 31 (1): 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x.