Skip to contents

mirtverse comes with six teaching labs, a course map and a glossary. They are written around the package’s own datasets. The instructor toolkit is for the next step: Teaching with your data, your examples and your course’s order of topics, without writing a learnr tutorial from scratch.

With the toolkit you can:

Every function takes file paths, returns what it wrote, and never asks a question, so you can also drive the toolkit from a script or an AI assistant.

You need mirtverse and learnr:

1. Start a course folder

new_course("EDPY507", title = "EDPY 507: Item response theory",
           instructor = "Your Name")

This creates:

Path What it holds
course.json The course title and instructor.
README.md A short how-to for whoever opens the folder next (a TA, or you next year).
labs/ One folder per interactive lab, each with an .Rmd file and a data/ folder.
chapters/ Notes chapters (R Markdown), one file each.
data/ Your raw data files, kept for reference.

The rest of this page assumes your R session is in the course folder (setwd("EDPY507"), or open it as an RStudio project). Every function also takes a course argument if you prefer full paths.

course_materials() lists what is in the folder at any time, including labs and chapters you add by hand:

course_materials()
#> # A tibble: 3 x 6
#>   name     type    title                                 path              source   data
#>   <chr>    <chr>   <chr>                                 <chr>             <chr>    <chr>
#> 1 midterm  lab     An IRT analysis of the midterm        ./labs/midterm    toolkit  the midterm
#> 2 w9       lab     Lab 3: Is the test fair? Detecting DIF ./labs/w9        built-in <NA>
#> 3 09-dif   chapter Differential item functioning ...     ./chapters/09-dif.Rmd book <NA>

2. Build a lab on your own data

Give new_lab() a name and your responses: A data frame, or a file (.csv, .xlsx, .sav, .dta, .sas7bdat or .rds).

new_lab("midterm", data = "data/midterm-2026.csv",
        group = "language",
        data_title = "the EDPY 507 midterm",
        story = paste("The department wants to know whether the midterm worked:",
                      "which items were weak, which model to score it with,",
                      "and whether it was fair to students writing in French."))

What the data need:

  • Scored items. Items scored 0/1, or in ordered categories 0, 1, 2 and so on (a lab on polytomous items compares the partial credit and graded response models instead of the Rasch model and the 2PL). For unscored multiple-choice responses, give the answer key in key and the items are scored for you.
  • Item columns. By default every column with whole-number scores from 0 to 9 is treated as an item. ID columns and the group column are left out. Name the items yourself with items = c("Q1", "Q2", ...) or items = 3:42 when the guess is wrong.
  • An optional group with exactly two values (a column name, or a vector with one value per row) turns on the DIF section.
  • At least about 200 people for stable estimates; new_lab() warns below that. Missing responses are fine.
  • Optional data for the AI sections (see Teaching AI in measurement on your data below): item_info, one row per item with its text or its item family, and scores, machine and human scores of written answers.

The data are copied into the lab’s data/ folder, so the lab works on any computer without your original file.

What students get

By default the lab has every section that suits the data, in this order. Choose and order them yourself with sections =.

Section What students do
intro Read your story, see the size of the data and look at the responses.
classical Compute item means; read item-total correlations; find the weakest item.
dimensionality Compare one- and two-factor models with anova() and BIC.
models Compare the Rasch model with the 2PL (or the graded model); try select_model().
item-fit Run itemfit() and gg_itemfit(); count the misfitting items.
information Plot test information and the standard error; say where the test is precise.
scoring Estimate EAP scores and compare them with sum scores.
dif Run likelihood-ratio DIF tests with dif() (needs a group).
reflection Write a recommendation (free text, good for a short assignment).
wrap-up A summary and pointers to reports and the other labs.
item-features Measure the item text with item_features(), find near-duplicates, and predict difficulty with predict_difficulty() (needs item text).
aig Check with calibrate_families() whether the items of each family are equally difficult (needs item families).
ai-scoring Compare machine and human scores with agreement_report(), then run a full ai_audit() (needs scores).

Nothing in these sections is written for one dataset. The numbers in the text, the checks on students’ code and the right answers to the quiz questions (“Which item has the lowest item-total correlation?”, “Which model does BIC prefer?”, “How many items misfit?”) are all computed from the data when the lab is built. That is what lets you reuse the same lab next term with new data.

Teaching AI in measurement on your data

The last three sections teach the topics of the AI Hub on your own material. They need more than responses, so give new_lab() one or two more tables:

  • item_info, with one row per item and a column item holding the item names used in the responses. A stem column (and, for multiple-choice items, option columns A to E and a key with the letter of the keyed option) turns on item-features. A family column, naming the item model or family each item comes from, turns on aig.
  • scores, with one row per written response the engine scored: machine and human1 (NA where no person read the response), and optionally human2, group, item and period (an administration or engine version). It turns on ai-scoring.
# 60 generated items: their text, their item models (as `family`) and the field test
items <- transform(aigfield_items, family = model)
new_lab("item-factory", data = aigfield, item_info = items,
        data_title = "the generated field-test items",
        sections = c("intro", "models", "item-features", "aig", "wrap-up"))

# an essay-scoring engine, with a validity sample read by people
scores <- read.csv("data/engine-scores.csv")   # machine, human1, human2, group, item
new_lab("our-engine", data = "data/responses.csv", scores = scores,
        sections = c("intro", "models", "ai-scoring", "reflection", "wrap-up"))

item_info can hold extra columns; rename yours to these names first, as the first example does with the item model column. The answers, as in the other sections, are computed from your data, so the questions ask, for example, how many families are not isomorphic in your items and whether your engine can score on its own.

For longer, story-driven AI labs with planted problems, copy Labs 7 to 10 (copy_lab("ai-audit") and so on), and see the AI module of the course map for how to fit them into a term.

3. Add your own content

Ready-made sections

add_lab_section("labs/midterm", "dif")
add_lab_section("labs/midterm", "information", after = "models")

New sections go before the wrap-up section unless you say where with after. lab_outline() shows each section’s id, which is what after and remove_lab_section() use:

lab_outline("labs/midterm")
remove_lab_section("labs/midterm", "dimensionality")

Your own coding exercises

add_lab_exercise() adds a section with an exercise box, an optional hint, a solution and a check. Inside every toolkit lab, the scored responses are resp, the group (if any) is group, and models holds the two item types the lab compares. With use_fit = TRUE the exercise also starts from fit, a one-factor model of the data.

add_lab_exercise("labs/midterm",
  title = "The hardest items",
  prompt = c("Which five items had the lowest mean score?",
             "Save their names as `hardest`."),
  starter = "hardest <- ",
  hint = "# sort(colMeans(resp, na.rm = TRUE)) puts the hardest items first",
  solution = "hardest <- names(sort(colMeans(resp, na.rm = TRUE)))[1:5]",
  check = c(
    "expected <- names(sort(colMeans(resp, na.rm = TRUE)))[1:5]",
    "lab_need(exists('hardest'), 'Save the item names as `hardest`.')",
    "lab_need(setequal(hardest, expected), 'Not quite: sort the item means from low to high.')",
    "'Right. Look at these items before the next exam.'"))

The check runs after the student’s code, in the same environment, so it can see every object the student made. lab_need(condition, message) fails the exercise with your message when the condition is not TRUE; .result holds the value of the student’s last line; and the last value of the check, if it is a string, is shown when every condition passes. Compute the expected answer from resp rather than typing it in, and the exercise will keep working when the data change.

Your own questions

add_lab_question("labs/midterm",
  question = "An item has a negative item-total correlation. What do you check first?",
  answers = c("The answer key", "The item's position on the test", "The sample size"),
  correct = 1,
  messages = c("Right: a miskeyed item is the most common cause.",
               "Position rarely reverses a correlation; check the key first.",
               "A small sample adds noise but rarely flips the sign; check the key first."))

Give more than one correct answer and students get a check-all-that-apply question instead.

Editing by hand

The lab is an ordinary learnr tutorial (labs/midterm/midterm.Rmd), so you can open it in RStudio and change anything: Reword the text, add images, move sections. The learnr documentation (https://pkgs.rstudio.com/learnr/) covers exercises and questions in detail. Two rules keep the toolkit functions working:

  • keep the <!-- mv-section: ... --> comment above each section’s ## heading, and
  • keep chunk labels unique (check_material() tells you if they are not).

4. Swap in other data

swap_lab_data("labs/midterm", data = "data/midterm-2027.csv",
              data_title = "the 2027 midterm")
check_material("labs/midterm")

The text, checks and questions are recomputed from the new data, and any exercises you added keep working as long as their checks compute their answers from resp. Settings you leave out are kept (for example the group column, the story, and the item_info and scores of the AI sections).

The built-in labs are different: Each tells the story of its own dataset, with problems planted on purpose (a miskeyed item, a drifted anchor, a severe rater), and the feedback talks about them. To teach the same topic on your data, build a toolkit lab with the matching sections. To keep the built-in story but change the wording, the order or the questions, copy it:

copy_lab("fairness", as = "week9-fairness")
add_lab_question("labs/week9-fairness", ...)

5. Notes chapters

The mirtverse book has one chapter per week of the course map. Copy a chapter into your course to edit it:

book_chapters()
copy_chapter("dif")    # downloads book/09-dif.Rmd into chapters/

Every example in a book chapter starts with a chunk labelled data-<chapter>-<n> that loads the data and defines resp. Replace it to run the example on your data:

swap_chapter_data("chapters/09-dif.Rmd", chunk = "data-dif-1",
                  code = c("midterm <- read.csv('../data/midterm-2026.csv')",
                           "resp <- midterm[, 3:42]",
                           "group <- midterm$language"))

Leave out chunk to replace every data chunk at once. new_chapter() starts an empty chapter with the book’s headings (learning goals, key terms, two examples, exercises, the matching lab and further reading).

6. Check, preview and share

res <- check_material("labs/midterm")
res[res$status != "pass", ]

check_material() works on a copy of the lab, so nothing in your folder changes. It checks the chunk labels, loads the data, runs every chunk that runs when the lab is built (including the quiz questions), and runs each exercise’s solution followed by its check, so it catches a solution that its own check rejects. Add render = TRUE to build the whole tutorial too. Run it on the course folder to check everything at once: check_material(".").

Then try the lab as a student would:

preview_lab("labs/midterm")

To give the lab to students, either

  • share the lab folder: Students open it with learnr::run_tutorial("path/to/midterm") (they need mirtverse and learnr); or
  • host it, so students need nothing but a browser: rsconnect::deployApp("labs/midterm") deploys it to shinyapps.io or Posit Connect. learnr’s event recorder can log students’ answers there; the teaching labs page has more tips.

A checklist for a new term

  1. new_course() once, and keep the folder under version control or in a synced folder.
  2. For each week, copy_lab() a built-in lab or new_lab() on your data.
  3. Add or remove sections, exercises and questions.
  4. copy_chapter() the matching book chapters and swap_chapter_data() where you use your own data.
  5. check_material(".") until nothing fails, then preview_lab() each lab.
  6. Next term: swap_lab_data() with the new data, and check again.