Comparison of methods for quick estimation of psychometric thresholds

Abstract

Threshold estimation plays a crucial role in modern cognitive science. Numerous psychophysical methods exist to quickly estimate key psychometric function parameters, like thresholds. Using Monte Carlo simulations of 60 to 120 trials, we compare stimulus selection methods in their ability to estimate thresholds. We evaluate model performance using measures including parameter bias, interquartile range, half-width of the limits of agreement and probability of missing the detection of participants responding randomly (miss rate). Psychometric functions are classically assumed to be monotonic, but they can be non-monotonic, with performance peaking and decreasing. Therefore, we investigate methods estimating threshold parameters for non-monotonic participants. We also assess methods’ ability to accurately identify ground-truth threshold parameters when participants respond randomly, or have parameters outside the normal range. For monotonic participants, Bayesian methods outperform the Method Of Constant Stimuli (MOCS) by an order of magnitude. For non-monotonic participants, when methods assume monotonicity, even using 120 trials results in catastrophic inaccuracy with median biases over 4,000%. When assuming non-monotonicity instead, methods result in psychometric properties comparable to those obtained for monotonic participants. However, MOCS, Psi and psi-grid inaccurately identify participants responding randomly, showing miss rates over 21%. Two methods in the Psi family outperform other methods, reaching results comparable to the monotonic case. In all tested scenarios, the best methods are psi-marginal and psi-marg-grid, a new method that combines psi-marginal and psi-grid. When these two methods assume non-monotonicity, they impose little performance cost when evaluating truly monotonic functions (example of cost: miss rate increased from 0 to 4% for identifying participants responding randomly with 60 trials) and no cost with 120 trials. Among these methods, we therefore recommend psi-marg-grid and psi-marginal for efficient threshold estimation regardless of potential monotonicity.