Compare simulated respondents with real ones (experimental)
Source:R/ai-synthetic.R
compare_synthetic.RdExperimental. Checks whether the respondents simulated with
simulate_llm_respondents() answer like real test takers. With a vector
of real proportions correct, it compares the items' difficulty. With real
responses, it also compares the spread of total scores and how well each
item separates high and low scorers (item-total correlations). For the
simulated answers alone it counts the distinct response patterns: Real
respondents rarely give identical answers, while language models playing
similar personas often do.
Arguments
- synthetic
an
mv_syntheticobject.- real
the real data: A data frame or matrix of 0/1 responses with the items as columns (named as in
synthetic), or a named numeric vector of proportions correct.- tolerance
the difference in mean proportion correct, and in the correlation of item difficulties below 1, that the notes treat as meaningful.
Value
An object of class mv_synthetic_comparison: A list with items
(a tibble with item, p_real, p_synthetic and difference, plus
rit_real and rit_synthetic with real responses), summary (a one-row
tibble) and notes (what the comparison shows, in sentences). It has
print() and autoplot() methods.
Examples
# a small made-up example: synthetic answers that are too easy
syn <- structure(list(
responses = data.frame(I1 = c(1, 1, 1, 1), I2 = c(1, 1, 1, 0), I3 = c(1, 1, 0, 0)),
provider = "example", model = ""), class = "mv_synthetic")
compare_synthetic(syn, c(I1 = 0.7, I2 = 0.5, I3 = 0.3))
#> <mv_synthetic_comparison> 4 simulated respondents, 3 items (EXPERIMENTAL)
#> item p_real p_synthetic difference
#> I1 0.7 1.00 0.30
#> I2 0.5 0.75 0.25
#> I3 0.3 0.50 0.20
#>
#> * The simulated respondents find the items easier than real test
#> takers: Mean proportion correct 0.75 against 0.50.
#> * They order the items by difficulty much like real test takers
#> (Spearman correlation 1.00); this still says nothing about any single
#> new item.
#> * Their answers vary little: 3 distinct response patterns among 4
#> simulated respondents. Respondents described alike answer alike.
#> * Simulated respondents are experimental: Use them to find unclear
#> items or to plan a study, not to estimate item parameters or to
#> replace a field test.