Fits a simple, transparent automated scoring engine: A ridge regression
of human scores on the words of the responses ("bag of words"). It runs
in base R with no internet connection and no API key, so it suits
teaching and small programs, and it plugs straight into score_open()
through its predict() method.
Usage
train_scorer(
data,
response = "response",
score = "human1",
by = NULL,
ngrams = 1:2,
length = FALSE,
min_docs = 3,
lambda = NULL,
folds = 5,
seed = 1
)
# S3 method for class 'mv_scorer'
tidy(x, ...)Arguments
- data
a data frame with one row per human-scored response.
- response, score
names of the columns that hold the response text and the human score used for training.
- by
optional name of a column (such as the prompt or item) to train a separate scorer for each of its values.
- ngrams
1for single words,1:2to add two-word phrases.- length
logical; add the log number of words as a feature.
- min_docs
smallest number of training responses a word must appear in to be used.
- lambda
the ridge penalty;
NULLchooses it by cross-validation.- folds
number of cross-validation folds.
- seed
random seed for the fold split, or
NULL.- x
a scorer from
train_scorer().- ...
not used.
Value
An object of class mv_scorer: A list with one model per value
of by (models), the settings, and cv, a tibble of out-of-fold
predictions for the training responses (row, by, human,
predicted, machine). Use predict() to score new responses,
tidy() for the word weights, and agreement_report() to check it.
tidy() returns a tibble with one row per feature and model:
by, term and weight, the change in the predicted score for one
more use of the word (on the log(1 + count) scale), sorted from the
largest positive weight.
Details
Each response is turned into counts of its words and, with ngrams = 1:2,
of its two-word phrases, kept when they appear in at least min_docs
training responses. The counts enter as log(1 + count). With
length = TRUE the log of the number of words is added as a feature,
as many commercial engines do. The ridge penalty lambda is chosen by
folds-fold cross-validation unless you give it, and the
cross-validated predictions are kept, so the scorer's agreement with
people can be judged on responses it was not trained on (see
agreement_report()).
A ridge regression treats scoring as prediction: It learns which words go with high human scores in the training responses, whatever the reason. Words that signal a key idea help, but so can words that merely go with longer or more fluent answers. That is why the scorer's agreement must be checked overall and within groups before its scores are used.
See also
score_open() to use the scorer in an assessment pipeline,
agreement_report() to check it; the
AI Hub.
Examples
e1 <- essays[essays$prompt == "E1", ]
s <- train_scorer(e1, response = "response", score = "human1")
s
#> <mv_scorer> ridge regression on words and two-word phrases
#> 400 responses, 662 features, lambda = 10, cross-validated RMSE = 0.56, QWK = 0.89
#> Check it with agreement_report(); use it with predict() or score_open().
head(tidy(s))
#> # A tibble: 6 × 3
#> by term weight
#> <chr> <chr> <dbl>
#> 1 all less 0.425
#> 2 all with 0.347
#> 3 all is less 0.314
#> 4 all than 0.281
#> 5 all on 0.249
#> 6 all are 0.244
predict(s, c("Ice is less dense than water, so it floats.", "I don't know."))
#> [1] 1 0