Grant Number:	5R01CA172463-04 Interpret this number
Primary Investigator:	Nelson, Kerrie
Organization:	Boston University Medical Campus
Project Title:	Model Agreement in Cancer Diagnostic Tests
Fiscal Year:	2017

DESCRIPTION (provided by applicant): Many cancer diagnostic tests involve the classification of a patient by a medical expert using an ordered categorical scale. Such tests involve elements of subjectivity and estimation on the part of the expert due to the necessity to interpret imperfect diagnostic test results, leading to discrepancies between experts' classifications, often severely so, even in common diagnostic procedures such as mammography and in the classification of breast density, an important predictor of breast cancer. This has motivated many large-scale studies to be conducted to examine levels of agreement between experts in common diagnostic settings and to investigate if factors such as rater experience affect the consistency of ratings made by different experts. However, limited statistical methods currently exist to assess agreement in large-scale studies such as these. Our overall goals are two-fold: (1) to develop novel and flexible statistical methods and agreement measures for assessing reliability in large-scale studies involving two or more medical experts when using one or more diagnostic tests with ordered categorical scales, and (2) to use these methods to assess reliability in recently conducted large-scale breast cancer and breast density studies and to examine the impact of factors such as rater experience and the patient's prior history that can play important roles in reliability in these population- based settings. Due to widespread use of screening mammography in the community, conclusions drawn from our analyses of large-scale agreement studies in diagnostic testing will have significant and far-reaching implications for breast cancer screening and diagnosis in the community. The proposed methods in our application provide a novel and comprehensive approach to examine agreement in large-scale studies and focus on assessing and comparing agreement between experts when they classify subjects according to ordered categorical classification scales in diagnostic tests. Methods developed will be made freely available and easily implemented using standard statistical software. Our analyses of large-scale cancer agreement studies using our proposed methods will provide new insights into the screening interpretative performance of radiologists.




Persistent inter-observer variability of breast density assessment using BI-RADS® 5th edition guidelines.
Authors: Portnow L.H. , Georgian-Smith D. , Haider I. , Barrios M. , Bay C.P. , Nelson K.P. , Raza S. .
Source: Clinical Imaging, 2022 Mar; 83, p. 21-27.
EPub date: 2021-12-10 00:00:00.0.
PMID: 34952487
Related Citations

Measuring rater bias in diagnostic tests with ordinal ratings.
Authors: Kim C. , Lin X. , Nelson K.P. .
Source: Statistics In Medicine, 2021-05-09 00:00:00.0; , .
EPub date: 2021-05-09 00:00:00.0.
PMID: 33969509
Related Citations

Methods of assessing categorical agreement between correlated screening tests in clinical studies.
Authors: Zhou T.J. , Raza S. , Nelson K.P. .
Source: Journal Of Applied Statistics, 2021; 48(10), p. 1861-1881.
EPub date: 2020-06-09 00:00:00.0.
PMID: 34305250
Related Citations

Measuring intrarater association between correlated ordinal ratings.
Authors: Nelson K.P. , Zhou T.J. , Edwards D. .
Source: Biometrical Journal. Biometrische Zeitschrift, 2020-06-11 00:00:00.0; , .
EPub date: 2020-06-11 00:00:00.0.
PMID: 32529683
Related Citations

Bayesian hierarchical latent class models for estimating diagnostic accuracy.
Authors: Wang C. , Lin X. , Nelson K.P. .
Source: Statistical Methods In Medical Research, 2019-05-30 00:00:00.0; , p. 962280219852649.
EPub date: 2019-05-30 00:00:00.0.
PMID: 31146651
Related Citations

A paired kappa to compare binary ratings across two medical tests.
Authors: Nelson K.P. , Edwards D. .
Source: Statistics In Medicine, 2019-05-17 00:00:00.0; , .
EPub date: 2019-05-17 00:00:00.0.
PMID: 31099902
Related Citations

Evaluating the effects of rater and subject factors on measures of association.
Authors: Nelson K.P. , Mitani A.A. , Edwards D. .
Source: Biometrical Journal. Biometrische Zeitschrift, 2018 05; 60(3), p. 639-656.
EPub date: 2018-01-19 00:00:00.0.
PMID: 29349801
Related Citations

Modeling rater diagnostic skills in binary classification processes.
Authors: Lin X. , Chen H. , Edwards D. , Nelson K.P. .
Source: Statistics In Medicine, 2017-11-02 00:00:00.0; , .
EPub date: 2017-11-02 00:00:00.0.
PMID: 29094378
Related Citations

Summary measures of agreement and association between many raters' ordinal classifications.
Authors: Mitani A.A. , Freer P.E. , Nelson K.P. .
Source: Annals Of Epidemiology, 2017 Oct; 27(10), p. 677-685.e4.
EPub date: 2017-09-22 00:00:00.0.
PMID: 29029991
Related Citations

Assessing the influence of rater and subject characteristics on measures of agreement for ordinal ratings.
Authors: Nelson K.P. , Mitani A.A. , Edwards D. .
Source: Statistics In Medicine, 2017-06-13 00:00:00.0; , .
EPub date: 2017-06-13 00:00:00.0.
PMID: 28612356
Related Citations

A measure of association for ordered categorical data in population-based studies.
Authors: Nelson K.P. , Edwards D. .
Source: Statistical Methods In Medical Research, 2016-05-16 00:00:00.0; , .
EPub date: 2016-05-16 00:00:00.0.
PMID: 27184590
Related Citations

Measures of agreement between many raters for ordinal classifications.
Authors: Nelson K.P. , Edwards D. .
Source: Statistics In Medicine, 2015-10-15 00:00:00.0; 34(23), p. 3116-32.
EPub date: 2015-10-15 00:00:00.0.
PMID: 26095449
Related Citations