Test-retest reliability for common tasks in vision science


Research in perception and attention has typically sought to evaluate cognitive mechanisms according to the average response to a manipulation. Recently, there has been a shift toward appreciating the value of individual differences and the insight gained by exploring the impacts of between-participant variation on human cognition. However, a recent study suggests that many robust, well-established cognitive control tasks suffer from surprisingly low levels of test-retest reliability (Hedge et al., 2018b). We tested a large sample of undergraduate students (n = 160) in two sessions (separated by 1–3 weeks) on four commonly used tasks in vision science. We implemented measures that spanned a range of perceptual and attentional processes, including motion coherence (MoCo), useful field of view (UFOV), multiple-object tracking (MOT), and visual working memory (VWM). Intraclass correlations ranged from good to poor suggesting that some task measures are more suitable for assessing individual differences than others. VWM capacity (ICC = 0.77), MoCo threshold (ICC = 0.60), UFOV middle accuracy (ICC = 0.60) and UFOV outer accuracy (ICC = 0.74) showed good-to-excellent reliability. Other measures, namely the maximum number of items tracked in MOT (ICC = 0.41) and UFOV number accuracy (ICC = 0.48), showed moderate reliability; the MOT threshold (ICC = 0.36) and UFOV inner accuracy (ICC = 0.30) showed poor reliability. In this paper, we present these results alongside a summary of reliabilities estimated previously for other vision science tasks. We then offer useful recommendations for evaluating test-retest reliability when considering a task for use in evaluating individual differences.

Publication DOI: https://doi.org/10.31234/osf.io/7fm2z
Divisions: College of Health & Life Sciences > Aston Institute of Health & Neurodevelopment (AIHN)
College of Health & Life Sciences
College of Health & Life Sciences > School of Psychology
Additional Information: Copyright 2022, The Authors. This work is licensed under a Creative Commons Attribution 4.0 International License. Funding Information: This work was partially supported by funding from the Psychological Sciences Research Group at the University of the West of England
Uncontrolled Keywords: Attention,Individual differences,Perception,Visual cognition,Ophthalmology,Sensory Systems
Publication ISSN: 1534-7362
Last Modified: 12 Jun 2024 16:44
Date Deposited: 11 Aug 2022 09:51
Full Text Link:
Related URLs: https://jov.arv ... ticleid=2783521 (Publisher URL)
https://osf.io/gtusw/ (Related URL)
PURE Output Type: Article
Published Date: 2022-07-29
Accepted Date: 2022-05-24
Authors: Clark, Kait
Birch-Hurst, Kayley
Pennington, Charlotte R. (ORCID Profile 0000-0002-5259-642X)
Petrie, Austin C. P.
Lee, Joshua T.
Hedge, Craig (ORCID Profile 0000-0001-6145-3319)



Version: Draft Version

Access Restriction: Restricted to Repository staff only

License: Creative Commons Attribution


Version: Published Version

License: Creative Commons Attribution

| Preview

Export / Share Citation


Additional statistics for this record