Skip to contents

Machine learning papers often describe ideas a psychometrician already knows under other names, and sometimes use a familiar word for something different. This page lines the two vocabularies up. Each row gives the machine learning or AI term, what it means in plain words, the closest idea in measurement, and where they differ, which is usually where the misunderstandings start.

The analogies are approximate on purpose. They give you a foothold, not an identity. Terms with a full entry in the glossary link there. The AI Hub shows how these ideas fit into the measurement workflow.

Data and models

Machine learning term What it means Closest psychometric idea Where they differ
Model A function that maps inputs to a prediction A measurement model, such as the 2PL A measurement model claims to describe how responses arise from a latent trait; a prediction model only has to predict well
Feature A measured input to a model A covariate, or an item property in an explanatory IRT model ML models may use thousands of features with no theory behind them
Label, target The outcome a model learns to predict The criterion, or a human score In scoring, the “label” is a human rating that contains its own error
Supervised learning Learning to predict a known label from features Regression on a criterion IRT is closer to unsupervised learning: The trait is never observed
Unsupervised learning Finding structure in data without labels Factor analysis, latent class analysis ML clustering rarely tests whether the structure fits or replicates
Latent representation Hidden variables a model builds to summarize inputs The latent trait θ Latent dimensions in a neural network have no intended meaning and are not identified
Classification Predicting a category Mastery decisions, latent class membership, pass/fail at a cut score ML reports accuracy against labels; psychometrics also asks how consistent the decision would be on a retest
Logistic regression Predicting a 0/1 outcome with a logit link The 1PL and 2PL item response functions In IRT the predictor (θ) is latent and estimated jointly with the item parameters
Neural network Layers of weighted sums passed through non-linear functions A very large, flexible multidimensional model Its parameters are not interpretable one by one
Decision tree, random forest, boosting Models that split cases by feature values and combine many such trees Recursive partitioning for DIF, such as Rasch trees Ensembles predict well but hide which feature does what

Training and evaluation

Machine learning term What it means Closest psychometric idea Where they differ
Training Estimating a model’s parameters from data Calibration Training usually minimizes prediction error, not a likelihood of a measurement model
Loss function The quantity training minimizes The negative log-likelihood Many ML losses are not likelihoods, so likelihood-ratio tests and information criteria do not apply
Training, validation and test sets Separate data for fitting, tuning and the final check Calibration sample and cross-validation sample ML treats the held-out test set as the only honest evidence; IRT often relies on fit to the calibration data
Overfitting Fitting noise, so new data are predicted worse Capitalizing on chance in item selection or model modification The cure in ML is held-out data; in IRT it is often a simpler model or a priori constraints
Cross-validation Rotating which part of the data is held out Cross-validation of a regression, double cross-validation in test construction Fold-to-fold variation shows how unstable the estimate is
Regularization (lasso, ridge) Shrinking parameters toward zero to reduce overfitting Priors on item parameters, as in Bayesian or MAP estimation The amount of shrinkage is tuned for prediction, not chosen on substantive grounds
Hyperparameter A setting chosen before training, such as tree depth Estimation settings: Number of quadrature points, prior variances Hyperparameters are tuned on validation data and can change results a lot
Bias-variance trade-off Simpler models err systematically; complex ones vary a lot Choosing between 1PL, 2PL and 3PL with AIC or BIC Same idea; ML usually settles it with held-out error instead of information criteria
Accuracy, precision, recall, F1 Ways to count correct and incorrect classifications Classification accuracy, sensitivity and specificity at a cut score ML “precision” is a share of positive predictions, not measurement precision
AUC How well a score ranks positives above negatives The probability a master outscores a non-master; a discrimination index AUC ignores calibration, the match between predicted and observed rates
Calibration (ML sense) Predicted probabilities match observed rates Item fit: Model-implied versus observed proportions Same word, different meaning: In IRT “calibration” is estimating item parameters
Data leakage Information in the features that would not be available in use Item exposure or contaminated anchor items Leakage makes a model look accurate in testing and fail in use
Distribution shift New data differ from training data Parameter drift, population invariance ML models rarely state the population they are valid for
Explainability (SHAP, feature importance) Methods that describe what drove a prediction Item parameters, which are interpretable by design Explanations of a black box are approximations of it, not the model itself

Text and language models

Machine learning term What it means Closest psychometric idea Where they differ
Natural language processing (NLP) Methods that turn text into numbers or text Content coding, readability formulas NLP features are computed automatically and at scale
Token A word or word piece a language model reads None close Token counts, not word counts, set the cost and length limits of a language model
Bag of words Representing a text by counts of its words, ignoring order A checklist rubric that counts key terms Ignores meaning and order, so it rewards keywords
Embedding A vector that places texts with similar meaning close together A factor score for a text Embedding dimensions have no meaning, and different models give different embeddings
Large language model A model trained on huge text corpora to predict the next token None close; sometimes used as an item writer, rater or simulated respondent Its output is a sample, not a fixed estimate
Prompt The instructions and context given to a language model Item-writing guidelines, rater training materials Small wording changes can change the output, so the prompt is part of the method
Temperature How random a language model’s output is None close At temperature 0 output is more repeatable but still not guaranteed identical
Zero-shot and few-shot Asking a model to do a task with no examples, or a few in the prompt Rater training with anchor papers The anchors chosen change the scores, as they do for human raters
Fine-tuning Further training a pretrained model on your own labelled data Training raters on local anchor sets Needs many human-scored responses and must be repeated when the model changes
Hallucination Fluent but false output Item-writing flaws: A wrong key, two keys, an implausible distractor Errors look confident, so they are found only by checking
Retrieval-augmented generation Giving the model source documents to answer from Writing items from a specified content source Reduces, but does not remove, invented content
Agent A language model that calls tools and takes several steps on its own An automated test assembly or scoring pipeline Each step can fail, so checks are needed between steps

Scoring and agreement

Machine learning term What it means Closest psychometric idea Where they differ
Automated scoring A model that scores constructed responses A rater The engine is consistent with itself but can be consistently wrong for a subgroup
Ground truth The correct label The true score In scoring, human scores stand in for the truth but contain rater error
Quadratic weighted kappa Chance-corrected agreement on an ordered scale Inter-rater reliability Depends on the score distribution, so compare machine-human with human-human QWK on the same responses
PRMSE Share of true-score variance a machine score explains Reliability of the machine score with respect to the human true score Needs double-scored responses to separate human error from machine error
Human-in-the-loop People review or override AI output Read-behind and adjudication by a third rater Routing rules decide which responses humans see, and they need their own checks

Fairness and governance

Machine learning term What it means Closest psychometric idea Where they differ
Algorithmic bias A model works worse or differently for a group Measurement bias, DIF ML fairness often compares outcomes; psychometrics compares people of equal ability
Demographic parity Equal rates of a positive outcome across groups Impact (group differences in scores) Equal outcomes are not required when groups differ in the trait; DIF conditions on the trait for that reason
Equalized odds Equal error rates across groups Equal classification accuracy across groups; differential prediction Closer to measurement invariance, but still conditions on an observed label, not the latent trait
Model card, datasheet A short document of a model’s data, uses and limits A technical manual, validity evidence Rarely includes the evidence a validity argument would ask for
Responsible AI Principles for safe, fair and transparent AI use The Standards for Educational and Psychological Testing The Standards ask for evidence for a specific use; responsible AI principles are often general

Where to go next

  • The AI Hub gives a learning path through these ideas.
  • The reading list has an annotated set of papers and books for each topic.
  • Lab 6 puts an automated scoring engine next to human raters on the same responses, and Lab 7 has students train an engine with train_scorer() and check it for fairness.