Certification exam, Form A: Technical report

Psychometrics team (fictional example) | November 03, 2026 | mirt 1.47 | mirtverse 0.2.0

Summary

Overview

  • A 2PL model with 1 dimension was fitted to 1200 respondents and 31 items (EM estimation).
  • The estimation converged normally.

Model fit

  • Overall model fit is close: M2 = 561.36, df = 434, RMSEA = 0.016, CFI = 0.987, SRMSR = 0.030.
  • The chi-square test is significant, but with large samples this is common; the RMSEA indicates the misfit is small in practical terms.

Reliability

  • Marginal reliability is 0.83 (good).

Targeting

  • The average item location is -0.18 and the average person score is -0.00, so the test is well targeted for this group.
  • Measurement is most precise around theta = -0.52 (information = 6.04, SE = 0.41).

Items

  • Low discrimination: Q12 (a = 0.42), Q30 (a = 0.43), Q32 (a = 0.47). These items barely separate high and low scorers; review the key and wording.
  • Extreme location: Q32 (b = 3.05). These items are far too easy or too hard for most respondents.
  • Low item-total correlation: Q12 (r = 0.16), Q21 (r = 0.19), Q30 (r = 0.19), Q32 (r = 0.17). These items relate weakly to the rest of the test.
  • Misfit: Q04 (S-X2 p = .004), Q12 (S-X2 p = .002), Q17 (S-X2 p = .002). The model does not describe responses to these items well (alpha = 0.01); inspect them with gg_empirical() or item_card().

Local dependence

  • 1 item pair shows possible local dependence (Q3 exceeds the average of -0.03 by more than 0.2).
  • Strongest: Q21 & Q22 (Q3 = 0.27).
  • Check whether these items share a stimulus, wording, or content; consider a testlet or bifactor model.

Person fit

  • 1.8% of respondents have unusual response patterns (Zh < -1.96), which is within what chance alone would produce.

Model

Statistic Value
nobs 1200
nitems 31
nfact 1
itemtype 2PL
npar 62
logLik -20593.4
AIC 41310.7
BIC 41626.3
SABIC 41429.4
converged yes
method EM
fit_stat M2
M2 561.4
df 434
p.value 0.000
RMSEA 0.016
SRMSR 0.030
CFI 0.987
TLI 0.986
reliability 0.831

Items

Status uses the item health rules: red for misfit or low discrimination, amber for other flags, green otherwise.

item status itemtype N mean item_total_r S_X2 df RMSEA p.value flag
Q01 green 2PL 1200 0.365 0.319 14.433 22 0 0.885
Q02 green 2PL 1200 0.567 0.375 25.562 21 0.013 0.224
Q03 green 2PL 1200 0.312 0.377 22.891 21 0.009 0.350
Q04 red 2PL 1200 0.338 0.311 43.356 22 0.028 0.004 misfit
Q05 green 2PL 1200 0.629 0.360 20.135 20 0.002 0.450
Q06 green 2PL 1200 0.266 0.301 34.593 21 0.023 0.031
Q07 green 2PL 1200 0.747 0.316 23.277 20 0.012 0.275
Q08 green 2PL 1200 0.347 0.296 16.305 22 0 0.800
Q09 green 2PL 1200 0.795 0.209 28.219 22 0.015 0.169
Q10 green 2PL 1200 0.499 0.300 31.586 22 0.019 0.085
Q11 green 2PL 1200 0.917 0.281 21.980 16 0.018 0.144
Q12 red 2PL 1200 0.263 0.157 46.575 22 0.031 0.002 low discrimination; low item-total correlation; misfit
Q13 green 2PL 1200 0.665 0.361 30.598 20 0.021 0.061
Q15 green 2PL 1200 0.759 0.289 26.694 21 0.015 0.181
Q16 green 2PL 1200 0.369 0.342 30.285 22 0.018 0.112
Q17 red 2PL 1200 0.921 0.233 38.053 17 0.032 0.002 misfit
Q18 green 2PL 1200 0.398 0.484 32.216 20 0.023 0.041
Q19 green 2PL 1200 0.484 0.336 15.923 21 0 0.774
Q20 green 2PL 1200 0.690 0.436 24.100 18 0.017 0.152
Q21 amber 2PL 1200 0.769 0.191 23.694 23 0.005 0.421 low item-total correlation
Q22 green 2PL 1200 0.815 0.244 16.518 21 0 0.740
Q23 green 2PL 1200 0.341 0.281 16.358 22 0 0.798
Q24 green 2PL 1200 0.632 0.393 17.984 20 0 0.588
Q25 green 2PL 1200 0.390 0.303 22.183 22 0.003 0.449
Q26 green 2PL 1200 0.427 0.482 24.095 19 0.015 0.193
Q27 green 2PL 1200 0.752 0.292 16.813 21 0 0.722
Q28 green 2PL 1200 0.549 0.331 17.983 21 0 0.650
Q29 green 2PL 1200 0.318 0.373 26.799 21 0.015 0.178
Q30 red 2PL 1200 0.532 0.186 35.340 23 0.021 0.048 low discrimination; low item-total correlation
Q31 green 2PL 1200 0.767 0.410 11.500 17 0 0.829
Q32 red 2PL 1200 0.203 0.167 20.112 23 0 0.635 low discrimination; extreme location; low item-total correlation

Item parameters

Parameterization: IRT (a, b, g, u).

item a b
Q01 0.833 0.750
Q02 1.039 -0.335
Q03 1.036 0.909
Q04 0.770 0.978
Q05 1.059 -0.630
Q06 0.813 1.413
Q07 0.998 -1.301
Q08 0.761 0.926
Q09 0.650 -2.266
Q10 0.749 -0.008
Q11 1.573 -2.046
Q12 0.416 2.568
Q13 1.085 -0.796
Q15 0.901 -1.485
Q16 0.918 0.671
Q17 1.273 -2.373
Q18 1.492 0.357
Q19 0.883 0.067
Q20 1.576 -0.748
Q21 0.542 -2.359
Q22 0.790 -2.103
Q23 0.709 1.023
Q24 1.209 -0.597
Q25 0.793 0.628
Q26 1.519 0.240
Q27 0.914 -1.419
Q28 0.886 -0.277
Q29 1.014 0.896
Q30 0.425 -0.324
Q31 1.652 -1.059
Q32 0.471 3.047

Graphics

Test information

Wright map

Item characteristic curves

Item fit

Local dependence

Scores

Sum score conversion table

Expected a posteriori (EAP) ability for each total score, with its standard error. Observed and expected counts show how well the model reproduces the score distribution.

Sum.Scores F1 SE_F1 observed expected
0 -3.073 0.557 0 0.136
1 -2.824 0.532 0 0.670
2 -2.586 0.509 1 1.868
3 -2.362 0.489 1 3.927
4 -2.151 0.470 2 6.956
5 -1.952 0.455 4 10.980
6 -1.763 0.441 11 15.943
7 -1.583 0.430 19 21.725
8 -1.410 0.421 25 28.154
9 -1.243 0.414 41 35.027
10 -1.081 0.408 33 42.125
11 -0.923 0.404 61 49.223
12 -0.767 0.401 70 56.084
13 -0.614 0.400 90 62.465
14 -0.462 0.399 94 68.117
15 -0.310 0.400 69 72.792
16 -0.159 0.402 85 76.263
17 -0.006 0.405 82 78.328
18 0.149 0.410 76 78.831
19 0.306 0.416 68 77.660
20 0.466 0.423 59 74.756
21 0.631 0.432 64 70.121
22 0.801 0.442 45 63.826
23 0.976 0.454 40 56.040
24 1.159 0.468 46 47.062
25 1.350 0.484 21 37.350
26 1.550 0.503 30 27.532
27 1.760 0.523 21 18.369
28 1.981 0.545 19 10.654
29 2.212 0.569 17 5.021
30 2.451 0.593 5 1.693
31 2.692 0.615 1 0.301

Item health cards

Blueprint

Construct: Professional knowledge for certification. Test takers: not stated. Score use: pass/fail certification. Forms have 12 items (multiple choice); target reliability 0.85.

content cognitive n
Ethics NA 4
Practice NA 5
Law NA 3

Field-test design

The field test uses 2 form(s) of 18 items, each taken by 600 test takers, with 4 anchor items on every form, for calibration with the 2PL model.

In 2 simulated field tests the slopes were recovered with an RMSE of 0.22 and the intercepts with an RMSE of 0.12.

form role item
1 anchor 4
2 anchor 4
1 unique 14
2 unique 14

Groups

Ability estimates (EAP) by group, with the average standard error. Groups with fewer than 10 respondents are left out.

group n mean_theta sd_theta mean_se
English 750 0.118 0.913 0.411
French 450 -0.198 0.878 0.402

Cut scores

1 cut score(s) on the summed-score scale divide test takers into Fail, Pass. Decision quality was estimated by the method of Lee (2010) for the calibrated test, over 1,200 abilities: classification accuracy 0.861, consistency 0.809 (kappa 0.61).

Cuts

cut label theta raw sem
1 Pass 0.157 18 0.421

Decision quality by cut

cut label accuracy consistency kappa false_positive false_negative
1 Pass 0.861 0.809 0.613 0.095 0.044

Expected share in each category

category expected true
Fail 0.555 0.606
Pass 0.445 0.394

Scoring

Test takers were scored by EAP: mean ability -0.00 (SD 0.91), median standard error 0.39.

Test security

Security checks run: preknowledge, answer copying. Security flags are evidence for an investigation, not proof of misconduct; individual test takers are not named in this report.

No test taker showed evidence.

Pipeline decisions

Every stage ran its checks and raised flags for review. Flags only warned (strict = FALSE). 9 flag(s) were raised; 1 are still open and 8 were acknowledged, overridden or fixed.

Stages

stage status summary
0 Blueprint done 3 cells, 12 items per form
1 Item generation not run
2 Item review not run
3 Field-test design done 2 forms of 18 items, 4 anchors, 1,200 test takers
4 Data intake done 1,200 respondents, 32 items, scored with key
5 Calibration done 2PL, converged, reliability 0.83
6 Linking and bank done 31 items added to the bank
7 Forms not run
8 Cut scores done 1 cut, accuracy 0.86, consistency 0.81
9 Scoring done EAP scores for 1,200 test takers, reliability 0.83
10 Test security done 2 checks; 0 test takers with evidence
11 Reporting done written to certification-technical-report.html
12 Bank monitoring not run

Flags

id level stage check target value rule status note advice
2 block intake possible_miskey Q14 p = 0.18, item-total r = -0.23 p < 0.1 or item-total r < 0 fixed Item dropped: Key confirmed wrong by the item writer Check the answer key and the item text; fix the key or drop the item.
4 block calibration dif_large Q09 MH D-DIF = -2.01, French vs English (favours English) ETS category C overridden Translation reviewed; French version revised Have a fairness panel review the item; drop or revise it unless the difference is construct-relevant.
1 review field_test recovery design RMSE of slopes = 0.22 RMSE <= 0.2 acknowledged Sample fixed by the testing window Parameters would be estimated imprecisely; increase the sample per form or reduce the number of forms.
3 review calibration local_dependence Q21 & Q22 Q3 = 0.27 |Q3| > 0.2 acknowledged Q21 and Q22 share a case; keep them apart on forms The items may share a stimulus or cue each other; review them together or treat them as a testlet.
5 review calibration dif_moderate Q27 MH D-DIF = -1.01, French vs English (favours English) ETS category B acknowledged Sent to the bias panel Review the item for content that may favour one group.
6 review calibration low_reliability test marginal reliability = 0.83 reliability < 0.85 acknowledged Operational forms will be longer than the field test Scores are less precise than planned; add items or report wider score bands.
8 review cuts cut_precision cut 1 (Pass) SEM = 0.42 at theta = 0.16 SEM <= 0.35 acknowledged Add items near the cut to the operational forms The test has little information at this cut; assemble forms targeted at it (assemble_forms(theta = cut)).
9 review scoring score_reliability scores empirical reliability = 0.83 reliability >= 0.85 acknowledged Field-test scores are not reported to candidates Scores are less precise than the target; report score bands and consider a longer test.
7 info calibration person_misfit 22 respondents 1.8% of respondents Zh < -1.96 open About 2.5% are expected by chance; many more may point to guessing, cheating or careless responding. See p$persons.

Decision log

time action detail
2026-11-03 05:21:32 set blueprint 3 cells; use: pass/fail certification
2026-11-03 05:21:34 design field test 2 forms of 18 items, 4 anchors, 1,200 test takers
2026-11-03 05:21:34 acknowledge acknowledged #1 recovery (design): Sample fixed by the testing window
2026-11-03 05:21:34 add responses 1200 respondents, 32 items
2026-11-03 05:21:34 drop items Q14: Key confirmed wrong by the item writer
2026-11-03 05:21:37 calibrate 2PL
2026-11-03 05:21:37 acknowledge acknowledged #3 local_dependence (Q21 & Q22): Q21 and Q22 share a case; keep them apart on forms
2026-11-03 05:21:37 override overridden #4 dif_large (Q09): Translation reviewed; French version revised
2026-11-03 05:21:37 acknowledge acknowledged #5 dif_moderate (Q27): Sent to the bias panel
2026-11-03 05:21:37 acknowledge acknowledged #6 low_reliability (test): Operational forms will be longer than the field test
2026-11-03 05:21:37 bank items 31 items added to Certification bank
2026-11-03 05:21:37 set cuts raw cuts at 18
2026-11-03 05:21:37 score persons EAP
2026-11-03 05:21:37 acknowledge acknowledged #8 cut_precision (cut 1 (Pass)): Add items near the cut to the operational forms
2026-11-03 05:21:37 acknowledge acknowledged #9 score_reliability (scores): Field-test scores are not reported to candidates
2026-11-03 05:21:39 check security preknowledge, answer_copying
2026-11-03 05:21:40 report pkgdown/assets/case-study/certification-technical-report.html

Respondents flagged

Counts only; the respondents are listed in p$persons in R.

check respondents
person_misfit 22

Rules in effect

Preset: standards; 1 rule(s) changed by the user.

Show all rules
rule value changed
miskey_p 0.1 default
miskey_r 0 default
missing_item 0.2 default
missing_person 0.5 default
straight_min_items 5 default
rapid_rt 0.1 default
rapid_person 0.1 default
min_category_n 10 default
rmsea 0.06 default
cfi 0.95 default
q3 0.2 default
fit_alpha 0.01 default
fit_adjust BH default
a_min 0.3 default
a_max 4 default
b_max 4 default
c_max 0.35 default
dif_alpha 0.05 default
dif_b 1 default
dif_c 1.5 default
dif_poly_b 0.17 default
dif_poly_c 0.25 default
reliability 0.85 changed
person_misfit -1.96 default
min_anchors 5 default
min_anchor_share 0.2 default
anchor_displacement 0.3 default
link_slope_min 0.8 default
link_slope_max 1.2 default
parallel_tol 0.1 default
cat_sem 0.35 default
cat_bias 0.1 default
max_exposure 0.25 default
unused_share 0.2 default
bank_depth 10 default
mst_routing 0.15 default
cut_accuracy 0.85 default
cut_consistency 0.8 default
cut_sem 0.35 default
panel_sd 0.5 default
duplicate_sim 0.9 default
coverage_ratio 3 default
min_reviewers 2 default
review_kappa 0.4 default
n_rasch 200 default
n_2pl 500 default
n_3pl 1000 default
recovery_rmse 0.2 default
n_validation 200 default
qwk_min 0.7 default
qwk_drop 0.1 default
score_smd 0.15 default
subgroup_smd 0.1 default
min_confidence 0.5 default
min_words 3 default
max_se 1 default
preknowledge_alpha 0.001 default
leak_p 0.15 default
leak_rt 0.25 default
drift_a 0.3 default
drift_b 0.3 default
copy_alpha 0.001 default
rt_resid -3 default
rt_share 0.2 default
site_alpha 0.001 default
gain_sem 3 default
min_group_n 10 default
exposure_windows 2 default
cell_depth 2 default
reliability_drop 0.05 default
unused_days 365 default

Reproducibility

Run this code to reproduce the analysis:

library(mirtverse)
p <- mv_pipeline(title = "Certification exam: Form A field test")
p <- set_blueprint(p, cells = c(Ethics = 4, Practice = 5, Law = 3), construct = "Professional knowledge for certification", 
    use = "pass/fail certification", reliability = 0.85)
p <- design_field_test(p, n_forms = 2, n_per_form = 600, items = 32, reps = 2, seed = 1)
p <- acknowledge(p, check = "recovery", note = "Sample fixed by the testing window")
p <- add_responses(p, data = certexam, items = 3:34, key = certexam_key, group = "language", 
    id = "id")
p <- drop_items(p, items = "Q14", note = "Key confirmed wrong by the item writer")
p <- calibrate(p, model = "2PL")
p <- acknowledge(p, check = "local_dependence", note = "Q21 and Q22 share a case; keep them apart on forms")
p <- override(p, check = "dif_large", note = "Translation reviewed; French version revised")
p <- acknowledge(p, check = "dif_moderate", note = "Sent to the bias panel")
p <- acknowledge(p, check = "low_reliability", note = "Operational forms will be longer than the field test")
p <- bank_items(p, name = "Certification bank")
p <- set_cuts(p, cuts = 18, scale = "raw", labels = c("Fail", "Pass"))
p <- score_persons(p)
p <- acknowledge(p, check = "cut_precision", note = "Add items near the cut to the operational forms")
p <- acknowledge(p, check = "score_reliability", note = "Field-test scores are not reported to candidates")
p <- check_security(p, leaked = c("Q01", "Q02", "Q03", "Q04", "Q05"))
p <- pipeline_report(p, file = "pkgdown/assets/case-study/certification-technical-report.html", 
    title = "Certification exam, Form A: Technical report", author = "Psychometrics team (fictional example)", 
    open = FALSE)
Session information
R version 4.5.3 (2026-03-11)
Platform: x86_64-conda-linux-gnu
Running under: Ubuntu 24.04.5 LTS

Matrix products: default
BLAS/LAPACK: /tmp/mm/renv/lib/libopenblasp-r0.3.34.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
[1] mirtverse_0.2.0 mirt_1.47       lattice_0.23-1 

loaded via a namespace (and not attached):
 [1] gtable_0.3.6          ggplot2_4.0.3         vctrs_0.7.3          
 [4] tools_4.5.3           generics_0.1.4        parallel_4.5.3       
 [7] tibble_3.3.1          proxy_0.4-29          cluster_2.1.8.3      
[10] pkgconfig_2.0.3       R.oo_1.27.1           Matrix_1.7-6         
[13] RColorBrewer_1.1-3    S7_0.2.2              RcppParallel_5.1.11-2
[16] lifecycle_1.0.5       compiler_4.5.3        farver_2.1.2         
[19] splines2_0.5.4        GPArotation_2026.8-2  textshaping_1.0.5    
[22] brio_1.1.5            codetools_0.2-20      permute_0.9-10       
[25] dcurver_0.9.3         htmltools_0.5.9       class_7.3-24         
[28] pillar_1.11.1         MASS_7.3-66           R.utils_2.13.0       
[31] vegan_2.7-5           sessioninfo_1.2.4     nlme_3.1-170         
[34] parallelly_1.50.0     Deriv_4.3.5           qs2_0.2.1            
[37] tidyselect_1.2.1      digest_0.6.39         future_1.82.0        
[40] dplyr_1.2.1           listenv_1.1.0         labeling_0.4.3       
[43] splines_4.5.3         fastmap_1.2.0         grid_4.5.3           
[46] cli_3.6.6             beepr_2.0             magrittr_2.0.5       
[49] patchwork_1.3.2       base64enc_0.1-6       future.apply_1.21.0  
[52] e1071_1.7-17          withr_3.0.3           clipr_0.8.1          
[55] SimDesign_2.26        scales_1.4.0          globals_0.20.0       
[58] audio_0.1-12          otel_0.2.0            gridExtra_2.3.1      
[61] progressr_1.1.0       ragg_1.5.2            R.methodsS3_1.8.2    
[64] stringfish_0.19.0     pbapply_1.7-4         testthat_3.3.2       
[67] viridisLite_0.4.3     mgcv_1.9-4            rlang_1.3.0          
[70] Rcpp_1.1.2            glue_1.8.1            R6_2.6.1             
[73] systemfonts_1.3.2