RECEIVER OPERATING CHARACTERISTIC (ROC) LITERATURE RESEARCH

Receiver Operating Characteristic (ROC) Literature Research

PI: Kelly H. Zou, Ph.D.

Grant Acknowledgement: This research has been supported by NIH R01-LM007861 "Improved Tumor Resection in Image-Guided Neurosurgery."

A. BACKGROUND (EARLY PAPERS AND TEXTBOOKS).

J.P. Egan. Signal Detection Theory and ROC Analysis, Academic Press,New York, 1975.
D.M. Green and J.A. Swets. Signal Detection Theory and Psychophysics. Robert E. Krieger Publishing Co., Huntington,New York, 1974.
D. Jalihal and L.W. Nolte. Signal detection theory and reconstruction algorithms-performance for images in noise. IEEE Transactions on Biomedical Engineering, 41:501-4, 1994.
L.B. Lusted. Signal detectability and medical decision-making. Science, 171:1217-1219, 1971.
N.A. Macmillan and C.D. Creelman. Detection Theory: A User's Guide, Cambridge University Press, Cambridge, 1991.
B.J. McNeil and S.J. Adelstein. Determining the value of diagnostic and screening tests. Journal of Nuclear Medicine, 17:439-448, 1976.
B.J. McNeil, E.Keeler, and S.J. Adelstein. Primer on certain elements of medical decision making. New England Journal of Medicine, 293:211-215, 1975.
W.W. Peterson, T.G. Birdsall, and W.C. Fox. The theory of signal detectibility. Transactions of the IRE Professional Group in Information Theory, PGIT, 2-4:171-212, 1954.
J.A. Swets and R.M. Pickett. Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Academic Press,New York, 1982.
J.A. Swets. Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers. Lawrence Erlbaum Associates, Publishers, Mahwah, New Jersy, 1995.
M. Treisman and A. Faulkner. The effect of signal probability on the slope of the receiver operating characteristic given by the rating procedure British J. Math. Statist. Psych, 37:199,215,1984.

B. OVERVIEWS / REVIEWS.

C.B. Begg. Statistical methods in medical diagnosis. CRC Critical Reviews in Medical Informatics, 1:1-22, 1986.
C.B. Begg. Advances in statistical methodology for diagnostic medicine in the 1980's. Statistics in Medicine, 10:1887-1895, 1991.
G. Champbell. Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Statistics in Medicine, 13 :499-508,1994.
R.M. Centor. Signal detectability: The use of ROC curves and their analyses. Medical Decision Making, 11:102-6, 1991.
L.S. Erdreich and E.T. Lee. Use of relative operating characteristic analysis in epidemiology: A method for dealing with subjective judgement. American Journal of Epidemiology, 114:649-62, 1981.
D.J. Goodenough, K.Rossmann, and L.B. Lusted. Radiographic applications of receiver operating characteristic (ROC) curves. Radiology, 110:89-95, 1974.
J.A. Hanley. Receiver operating characteristic (ROC) methodology: The state of the art. Critical Reviews in Diagnostic Imaging, 29:307-35, 1989.
A.R. Henderson. Assessing test accuracy and its clinical consequences: a primer for receiver operating characteristic curve analysis. [Review] Annals of Clinical Biochemistry,30:521-39, 1993.
J.K. Hsiao, J.J. Bartko, and W.Z. Potter. Diagnosing diagnoses: Receiver operating characteristic methods and psychiatry. Archives of General Psychiatry, 46:664-7, 1989.
C.E. Metz. Basic principles of ROC analysis. Seminars In Nuclear Medicine, 8:283-98, 1978.
C.E. Metz. ROC methodology in radiologic imaging. Investigative Radiology, 21:720-733, 1986.
C.E. Phelps. The methodologic foundations of studies of the appropriateness of medical care [see comments]. New England Journal of Medicine, 329(17):1241-5, 1993.
Y.T. van der Schouw, A.L. Verbeek, and S.H.Ruijs. Guidelines for the assessment of new diagnostic tests. Invest Radiol, 30:334-40,1995.
M. Schulzer. Diagnostic tests: a statistical review. [Review] Muscle and Nerve, 17(7):815-9, 1994.
J.A. Swets. ROC analysis applied to the evaluation of medical imaging techniques. Investigative Radiology, 14:109-21, 1979.
J.A. Swets. Measuring the accuracy of diagnostic systems. Science, 240:1285-93, 1988.
D.A. Turner. An intuitive approach to receiver operating characteristic curve analysis. Journal of Nuclear Medicine, 19:213-20, 1978.

C. DESIGN OF ROC STUDIES / BIAS.

C.B. Begg and R.A. Greenes. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics, 39:207-215, 1983.
C.B. Begg. Biases in the assessment of diagnostic tests. Statistics in Medicine, 6:411-423, 1987.
C.B. Begg. Experimental design of medical imaging trials: Issues and options. Investigative Radiology, 24:934-936, 1989.
C.B. Begg and B.J. McNeil. Assessment of radiologic tests: Control of bias and other design considerations. Radiology, 167:565-9, 1988.
K.S. Berbaum, D.D. Dorfman, and E.A. Franken, Jr. Measuring observer performance by ROC analysis: Indications and complications. Investigative Radiology, 24:228-33, 1989.
G.A. Diamond. ROC steady: A receiver operating characteristic curve that is invariant relative to selection bias. Medical Decision Making, 7:238-43, 1987.
G.A. Diamond. ROCky III. Medical Decision Making, 7:247-9, 1987.
G.A. Diamond. Can the discriminant accuracy of a test be determined in the face of selection bias? Medical Decision Making,11:48-56, 1991.
G.A. Diamond. Scotched on the ROCs. Medical Decision Making, 11:198-200, 1991.
G.A. Diamond. What is the effect of sampling error on ROC analysis in the face of verification bias? Medical Decision Making, 12:155-6, 1992.
C.A. Gatsonis and X.H. Zhou. Group sequential designs for comparative ROC studies in diagnostic medicine, Technical report, Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts, 1993.
W.F. Good, D.Gur, W.H. Straub, and J.H. Feist. Comparing imaging systems by ROC studies: Detection versus interpretation. Investigative Radiology, 24:932-3, 1989.
R.Gray, C.B. Begg, and R.A. Greenes. Construction of receiver operating characteristic curves when disease verification is subject to selection bias. Medical Decision Making, 4:151-64, 1984.
R.A. Greenes and C.B. Begg. Assessment of diagnostic technologies: Methodology for unbiased estimation from samples of selectively-verified patients. Investigative Radiology, 20:751-756, 1985.
D.Gur, J.L. King, H.E. Rockette, C.A. Britton, F.L. Thaete, and R.J. Hoy. Practical issues of experimental ROC analysis: Selection of controls. Investigative Radiology, 25:583-6, 1990.
D.Gur, H.E. Rockette, W.F. Good, B.S. Slasky, L.A. Cooperstein, W.H. Straub, N.A. Obuchowski, and C.Metz. Effect of observer instruction on ROC study of chest images. Investigative Radiology, 25:230-4, 1990.
J.A. Hanley and C.B. Begg. Response to ROC steady. Medical Decision Making, 7:244-6, 1987.
J.A. Hanley. Verification bias and the one-parameter logistic ROC curve-some clarifications. [Comment on Diamond 1991]. Medical Decision Making, 11:203-7, 1991.
M.G. Hunink and C.B. Begg. Diamond's correction method-a real gem or just cubic zirconium? [Comment on Diamond 1991]. Medical Decision Making, 11:201-3, 1991.
J.L. King, C.A. Britton, G.D., H.E. Rockette, and P.L. Davis. On the validity of the continuous and discrete confidence rating scales in receiver operating characteristic studies. Investigative Radiology,28:962-963, 1993.
C.E. Metz. Some practical issues of experimental design and data analysis in radiological ROC studies. Investigative Radiology, 24:234-45, 1989.
C.E. Metz and J.H. Shen. Gains in accuracy from replicated readings of diagnostic images: Prediction and assessment in terms of ROC analysis. Medical Decision Making, 12:60-75, 1992.
N.A. Obuchowski. Computing sample size for receiver operating characteristic studies. Investigative Radiology, 29:238-243, 1994.
D.F. Ransohoff and A.R. Feinstein. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New England Journal of Medicine, 299:926-930, 1978.
H.E. Rockette, D.Gur, L.A. Cooperstein, N.A. Obuchowski, J.L. King, C.R. Fuhrman, E.K. Tabor, and C.E. Metz. Effect of two rating formats in multi-disease ROC study of chest images. Investigative Radiology, 25:225-9, 1990.
H.E. Rockette, D.Gur, and C.E. Metz. The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Investigative Radiology, 27:169-172, 1992.

D. CURVE-FITTING.

I.G. Abrahamson and H.Levitt. Statistical analysis of data from experiments in human signal detection. [Maximum likelihood ROC curve - general location-scale family]. Journal of Mathematical Psychology, 6:391-417, 1969.
G. Campbell and M.V. Ratnaparkhi. An application of Lomax distributions in receiver operating characteristic (ROC) curve analysis. Comm.Statist. A--Theory Methods :1681-1697,1993.
D.D. Dorfman and E.Alf, Jr. Maximum likelihood estimation of parameters of signal detection theory-a direct solution.[Binormal ROC curve-dichotomous diagnostic test]. Psychometrika, 33:117-124, 1968.
D.D. Dorfman and K.S. Berbum. Degeneracy and discrete receiver operating characteristic rating data. Academic Radiology,2:907-915,1995.
D.D. Dorfman and E.Alf, Jr. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data. [Binormal ROC curve-ordinal data]. Journal of Mathematical Psychology, 6:487-496, 1969.
D.R. Grey and B.J.T. Morgan. Some aspects of ROC curve-fitting: Normal and logistic models. [Corrections and improvements to Dorfman-Alf; minimum chi-square bilogistic]. Journal of Mathematical Psychology, 9:128-139, 1972.
J.A. Hanley. The robustness of the ``binormal'' assumptions used in fitting ROC curves. Medical Decision Making, 8:197-203, 1988.
J.A. Hanley. The use of the 'binormal' model for parametric roc analysis of quantitative diagnostic tests. Statistics in Medicine, 15 : 1575-1585, 1996.
H.L. Kundel, D.D. Dorfman and K.S. Berbaum. Degeneracy and discrete receiver operating characteristic rating data, 2 :907-15,1995.
C.E. Metz and X. Pan. "Proper" binormal ROC curves: theory and maximum-likelihood estimation, unpublished manuscript, University of Chicago.
D. Mossman. Resampling techniques in the analysis of non-binormal ROC data. Med Decis Making , 15:358-66,1995.
J.C. Ogilvie and C.D. Creelman. Maximum-likelihood estimation of receiver operating characteristic curve parameters. Journal of Mathematical Psychology, 5:377-391, 1968.
E. Somoza. Eccentric Diagnostic Tests. Medical Decision Making, 16:15-23, 1996.
J.A. Swets. Form of empirical ROCs in discrimination and diagnostic tasks: Implications for theory and measurement of performance. Psychological Bulletin, 99:181-198, 1986.
A.N. Tosteson and C.B. Begg. A general regression methodology for ROC curve estimation. Medical Decision Making, 8:204-15, 1988.
A.N. Tosteson, J. Wittenberg, and C.B. Begg. ROC curve regression analysis: the use of ordinal regression models for diagnostic test assessment. Environmental Health Perspectives, 8:73-8, 1994.
X.H. Zhou. Testing an underlying assumption on a ROC curve based on rating data. Med Decis Making, 15:276-82,1995.

E. INDICES OF DIAGNOSTIC ACCURACY.

R.M. Centor and J.S. Schwartz. An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve. Medical Decision Making, 5:149-56, 1985.
C. Cox. Location-scale cumulative odds models for ordinal data: a generalized non-linear model approach. Statistics in Medicine,14:1191-203, 1995.
J.A. Hanley and B.J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143:29-36, 1982.
R.D. Hays. ROC: Estimation of the area under a receiver operating characteristic curve. Appl. Psych. Meas, 14 :208,1990.
J.Hilden. The area under the ROC curve and its competitors. Medical Decision Making, 11:95-101, 1991.
D.Katz and B.Foxman. How well do prediction equations predict? Using receiver operating characteristic curves and accuracy curves to compare validity and generalizability. Epidemiology,4(4):319-26, 1993.
D.K. McClish. Analyzing a portion of the ROC curve. Medical Decision Making, 9:190-5, 1989.
T.O. Nelson. ROC curves and measures of discrimination accuracy: A reply to Swets. [Comment on Swets 1986a]. Psychological Bulletin, 100:128-132, 1986.
J.J. Riera-Diaz. The estimation of event related potentials affected by random shifts and scalings. International Journal of Bio-Medical Computing,38:109-20,1995.
A.J. Simpson and M.J. Fitter. What is the best index of detectability? [Recommends the area under the binormal curve]. Psychological Bulletin, 80:481-488, 1973.
J.A. Swets. Indices of discrimination or diagnostic accuracy: Their ROCs and implied models. Psychological Bulletin, 99:100-117, 1986.

F. STATISTICAL INFERENCE.

A.Agresti. A survey of models for repeated ordered categorical response data. Statistics in Medicine, 8:1209-1224, 1989.
D.Bamber. The area above the ordinal dominance graph and the area below the receiver operating graph. Journal of Mathematical Psychology, 12:387-415, 1975.
C.A. Beam. Random-effects models in the receiver operating characteristic curve-based assessment of the effectiveness of diagnostic imaging technology: concepts, approaches, and issues. Academic Radiology, 2: supple: 4-13, 1995.
C.A. Beam and H.S. Wieand. A statistical method for the comparison of a discrete diagnostic test with several continuous diagnostic tests. Biometrics, 47:907-919, 1991.
D.A. Block. Comparing two diagnostic tests against the same "gold standard" in the same sample. Biometrics, 53: 73-85, 1997.
C. Cox. Location-scale cumulative odds models for ordinal data: a generalized non-linear model approach. Statistics in Medicine, 15 :1191-203, 1995.
E.R. DeLong, W.B. Vernon, and R.R. Bollinger. Sensitivity and specificity of a monitoring test. Biometrics, 41:947-958, 1985.
E.R. DeLong, D.M. DeLong, and D.L. Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44:837-45, 1988.
D.D. Dorfman, K.S. Berbaum, and C.E. Metz. Receiver operating characteristic rating analysis: Generalization to the population of readers and patients with the jackknife method. Investigative Radiology, 27:723-31, 1992.
B. Emir, S.Wieand, J.Q. Su, and S. Cha. Analysis of repeated markers used to predict progression of cancer. Statistics in Medicine, 17: 2563-78, 1998.
B. Emir, S. Wieand, S.-H. Jung, and Z. Ying. Comparison of diagnostic markers with repeated measurements: A non-parametric ROC curve approach. Staistics in Medicine, 19: 511-23, 2000.
C.A. Gatsonis. Random-effects models for diagnostic accuracy data. Academic Radiology, 2:suppl:14-21, 1995.
J.A. Hanley. Alternative approaches to receiver operating characteristic analyses. Radiology, 168:568-70, 1988.
J.A. Hanley and B.J. McNeil. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148:839-43, 1983.
R.A. Hilgers. Distribution-free confidence bounds for ROC curves. Methods of Information In Medicine, 30:96-101, 1991.
F. Hsieh and B.W. Turnbull. Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Annals of Statistics, 24: 25-40, 1996.
K. Jensen, H.-H. Muller and H. Schafer. Regional confidence bands for ROC curves. Statistics in Medicine, 19: 493-509, 2000.
V. Kairisto and A. Poola. Software for illustrative presentation of basic clinical characteristics of laboratory tests--GraphROC for Windows. Scandinavian Journal of Clinical & Laboratory Investigation, 222(suppl):43-60, 1995.
K. Kim. A bivariate cumulative probit regression model for ordered categorical data. Statistics in Medicine, 14 :1341-1352, 1995.
G.Ma and W.J. Hall. Confidence bands for receiver operating characteristic curves. Medical Decision Making, 13:191-197, 1993.
D.K. McClish. Comparing the areas under more than two independent ROC curves. Medical Decision Making, 7:149-55, 1987.
D.K. McClish. Determining a range of false-positive rates for which ROC curves differ. Medical Decision Making, 10:283-7, 1990.
B.J. McNeil and J.A. Hanley. Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Medical Decision Making, 4:137-50, 1984.
A. Moise, B. Clement, M. Raissis and P. Manopoulos. A test for crossing receiver operating characteristic (ROC) curves. Comm. Statist. A -- Theory Methods, 17:1985-2004,1988.
C.E. Metz. Statistical analysis of ROC data in evaluating diagnostic performance. In D.E. Herbert and R.H. Myers, editors, Multiple regression analysis: Applications in the health sciences, pages 365-384,American Institute of Physics, New York,1986.
C.E. Metz. Quantification of failure to demonstrate statistical significance: The usefulness of confidence intervals. Investigative Radiology, 28:59-63, 1993.
C.E. Metz and H.B. Kronman. Statistical significance tests for binormal ROC curves. Journal of Mathematical Psychology, 22:218-243, 1980.
C.E. Metz, J.H. Shen, and B.A. Herman. New methods for estimating a binormal ROC curve from continuously-distributed test results. Unfinished manuscript; presented at the 1990 Joint Statistical Meetings in Anaheim, California.
C.E. Metz, P.-L. Wang, and H.B. Kronman. A new approach for testing the significance of differences between ROC curves measured from correlated data. In F.Deconinck, editor, Information Processing in Medical Imaging: Proceedings of the Eighth Conference, 432-445.Martinus Nijhoff Publishers, The Hague, 1984.
A.Moise, B.Clement, and M.Raissis. A test for crossing receiver operating characteristic (ROC) curves. Communications in Statistics-Theory and Methods, 17:1985-2003, 1988.
P.A. Murtaugh. ROC curves with multiple marker measurements. Biometrics, 51 :1514-22, 1995.
N.A. Obuchowski. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Academic Radiology, 2: supple: 22-29,1995.
N.A. Obuchowski Nonparametric analysis of clustered ROC curve data. Unpublished manuscript, 1996 ENAR meeting.
H.E. Rockette, N.A. Obuchowski, and D.Gur. Nonparametric estimation of degenerate ROC data sets used for comparison of imaging systems. Investigative Radiology, 25:835-7, 1990.
H.E. Rockette, N.Obuchowski, C.E. Metz, and D.Gur. Statistical issues in ROC curve analysis. Proceedings of the SPIE, 1234:111-119, 1990.
D.E. Shapiro. The interpretation of diagnostic tests. Statistical Methods in Medical Research, 8: 113-134, 1999.
E.K. Shultz. Multivariate receiver-operating characteristic curve analysis: prostate cancer screening as an example.[Review] Clinical Chemistry, 41:1248-55, 1995.
P.J. Smith, T.J. Thompson, M.M. Engelgau and W.H. Herman. A generalized linear model for analysing receiver operating characteristic curves. Statistics in Medicine, 15 :323-33, 1996.
E. Svensson, and S. Holm. Separation of systematic and random differences in ordinal rating scales. Statistics in Medicine,13:2437-53, 1994.
M. Swaving, H. Van Houwelingen, F.P. Ottes, T. Steerneman. Statistical comparison of ROC curves from multiple readers. Medical Decision Making, 16 : 143-153, 1996.
M.L. Thompson and W.Zucchini. On the statistical analysis of ROC curves. Statistics In Medicine, 8:1277-90, 1989.
A.L. Toledano and C.A. Gatsonis. Regression analysis of correlated receiver operating characteristic data. Academic Radiology,2:supple:30--36, 1995.
A.Y. Toledano and C.A. Gatsonis. Ordinal regression methodology for roc curves derived from correlated data. Statistics in Medicine,15:1807-26,1996.
S. Wieand, M.H. Gail, B.R. James, and K.L. James. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika , 76:585-592, 1989.
X.H. Zhou. Testing an underlying assumption on a ROC curve based on rating data. Medical Decision Making,15:276-282,1995.
X.H. Zhou Testing an underlying assumption on a ROC curve based on rating data. Medical Decision Making, 15 :276-82, 1995.
X.H. Zhou and C.A. Gatsonis. A smple method for comparing correlated roc curves using incomplete data. Statistics in Medicine, 15 :1687-1693, 1996.
K.H. Zou, C. M Tempany, J. R. Fielding, and S. G. Silverman. Original smooth receiver operating characteristic curve estimation from continous data: Statistical methods for analyzing the predictive value of spiral CT of ureteral stones. Academic Radiology, 5: 680-687, 1998.
K.H. Zou, W.J. Hall and D.E.Shapiro. Smooth nonparametric receiver operating characteristic curves for continuous diagnostic tests. Statistics in Medicine, 16: 2143-2156, 1997.
K.H. Zou and W. J. Hall. Two transformation models for estimating an ROC curve derived from continous data. Journal of Applied Statistics, 26: 621-631, 2000.
K.H. Zou. Comparison of correlated ROC curves derived from repeated diagnostic test data. Academic Radiology, 8: 225-233, 2001.

G. IMPERFECT GOLD STANDARD.

T.A. Alonzo and M.S. Pepe. Using a combination of reference tests to assess the accuracy of a new diagnostic test. Statistics in Medicine, 18: 2987-3003, 1999.
G.Baker, S. Evaluating a new test using a reference test with estimated sensitivity and specificity. Communication in Statistics, Part A-Theory and Methods, 20:2739-2752, 1991.
C.B. Begg and C.E. Metz. Consensus diagnoses and ``gold standards''. [Comment on Henkelman, Kay and Bronskill 1990]. Medical Decision Making, 10:29-30, 1990.
G. Campbell and J.M. Deleo. Fundamentals of fuzzy receiver operating characteristic (ROC) functions. Comput. Sci. Statist.: Proc. 21st Symp. Interface (Kenneth Berk and Linda Malone, eds.), Amer. Statist. Assoc. (Alexandria, VA) :543-548,1989.
P.Deneef. Evaluating rapid test for streptococcal pharyngitis: The apparent accuracy of a diagnostic test when there are errors in the standard of comparison. Medical Decision Making, 7:92-96, 1987.
S.V. Faraone and M.T. Tsuang. Measuring diagnostic accuracy in the absence of a "gold standard".[Review] American Journal of Psychiatry, 151(5):650-7, 1994.
R.A. Greenberg and J.F. Jekel. Some problems in the determination of the false positive and false negative rates of tuberculin tests. American Review of Respiratory Disease, 100:645-650, 1969.
J.J. Gart and A.A. Buck. Comparison of a screening test and a reference test in epidemiological studies: II. a probabilistic model for the comparison of diagnostic tests. American Journal of Epidemiology, 83:593-602, 1966.
D.A. Grayson. Statistical diagnosis and the influence of diagnostic error. Biometrics, 43:975-984, 1987.
W.J. Hall & D.E. Shapiro A receiver operating characteristic curve for subjective probability of disease assessments in the absence of a gold standard (abstract only). Medical Decision Making, 14:430, 1994.
R.M. Henkelman, I.Kay, and M.J. Bronskill. Receiver operator characteristic (ROC) analysis without truth. Medical Decision Making, 10:24-9, 1990.
S.L. Hui and S.D. Walter. Estimating the error rates of diagnostic tests. Biometrics, 36:167-171, 1980.
S.L. Hui and X.H. Zhou. Evaluation of diagnostic tests without gold standards. Statistical methods in medical research, 7: 354-370, 1998.
H.C. Kraemer. The robustness of common measures of 2X2 association to bias due to misclassifications. American Statistician, 39:286-290, 1985.
N.J.D. Nagelkerke, V.Fidler, and M.Buwalda. Instrumental variables in the evaluation of diagnostic test procedures when the true disease state is unknown. Statistics in Medicine, 7:739-744, 1988.
C.E. Phelps and A.Hutson. Estimating diagnostic test accuracy using a `fuzzy' gold standard. Medical Decision Making, 15:44-57, 1995.
R.M. Poses, R.D. Cebul, and R.M. Centor. Evaluating physicians' probabilistic judgements. Medical Decision Making, 8:233-240, 1988.
R.M. Poses, C.B. Bekes, R.L. Winkler, W.E. Scott, and F.J. Copare. Are two (inexperienced) heads better than one (experienced) head? Archives of Internal Medicine, 150:1874-1878, 1990.
M.Staquet, M.Rozencweig, Y.J. Lee, and F.M. Muggia. Methodology for the assessment of new dichotomous diagnostic tests. Journal of Chronic Disease, 34:599-610, 1981.
V. Torrance-Rynard and S. Walter. Effects of dependent errors in the assessment of diagnostic test performance. Statistics in Medicine, 16: 2157-2175, 1997.
J.S. Uebersax. Statistical modeling of expert ratings on medical treatment appropriateness. Journal of the American Statistical Association, 88:421-427, 1993.
J.S. Uebersax and W.M. Grove. Latent class analysis of diagnostic agreement. Statistics in Medicine, 9:559-572, 1990.
J.S. Uebersax and W.M. Grove. A latent trait finite mixture model for the analysis of rating agreement. Biometrics, 49:823-835, 1993.
P.M. Vacek. The effect of conditional dependence on the evaluation of diagnostic tests. Biometrics, 41:959-968, 1985.
P.N. Valenstein. Evaluating diagnostic tests with imperfect standards. American Journal of Clinical Pathology, 93:252-258, 1990.
S.D. Walter and L.M. Irwig. Estimation of test error rates, disease prevalence, and relative risk from misclassified data. Journal of Clinical Epidemiology, 41:923-38, 1988.
X.H. Zhou and R. Higgs. Assessing the relative accuracies of two screening tests in the presence of verification bias. Statistics in Medicine, 19: 1697-1705, 2000.
X.H. Zhou and R. Higgs. COMPROC and CHECKNORM: computer programs for comparing accuracies of diagnostic tests using ROC curves in the presence of verification bias. Computer Methods and Programs in Biomedicine, 57: 179-186,1998.
X.H. Zhou and C.A. Rodenberg. Estimating a ROC curve in the presence of verification bias. Communication in Statistics, Ther., 27: 635-657.
X.H. Zhou. Maximum likelihood estimators of sensitivity and specificity corrected for verification bias. Commun. Statist. Theor. Meth, 22: 3177-3198, 1993.
X.H. Zhou. Effect of verification bias on positive and negative predictive values. Statistics in Medicine, 13: 1737-1745, 1994.
X.H. Zhou. Nonparametric ML estimate of a ROC area corrected for verification bias. Biometrics, 52: 310-316, 1996.
X.H. Zhou. Comparing the correlated areas under the ROC curves of two diagnostic tests in the presence of verification bias. Biometrics, 54, 453-470, 1998.
X.H. Zhou. Comparing accuracies of two screening tests in the presence of verication bias. JRSS, Ser C, 47: 135-147, 1998.
X.H. Zhou. Correction for verificatin bias in studies of a diagnostic test's accuracy. Statistics Methods in Medical Research, 7: 337-353, 1998.

H. META-ANALYSIS.

C.. Berkey, J.J. Anderson and D.C. Hoaglin. Multiple-outcome meta-analysis of clinical trials. Statistics in Medicine, 15 :537-557, 1996.
C.S. Berkey, D.C. Hoaglin, F. Mosteller and G.A. Colditz. A random-effects regression model for meta-analysis. Statistics in Medicine, 14 :195-411, 1995.
W.J. Hall. Efficiency of weighted averages, Technical Report 92/02, Department of Biostatistics, University of Rochester, Rochester, New York, Feb. 1992.
M.Helfand. Meta-analysis in deriving summary estimates of test performance. Medical Decision Making, 13:182-3, 1993.
L.Irwig, A.N.A. Tosteson, C.Gatsonis, J.Lau, G.Colditz, T.Chalmers, and F.Mosteller. Guidelines for meta-analyses evaluating diagnostic tests. Annals of Internal Medicine, 120:667-676, 1994.
J.W. P.F. Kardaun and O.J. W.F. Kardaun. Comparative diagnostic performance of three radiological procedures for the detection of lumbar disk herniation. Methods of Information in Medicine, 29:12-22, 1990.
J.W. P.F. Kardaun, J.Schipper, and R.Braakman. CT, myelography, and phlebography in the detection of lumbar disk herniation: An analysis of the literature. American Journal of Neuroradiology, 10:1111-1122, 1989.
D.L. Kent, D.R. Haynor, E.B. Larson, and R.A. Deyo. Diagnosis of lumbar spinal stenosis in adults: A meta-analysis of the accuracy of CT, MR, and myelography. American Journal of Roentgenology, 158:1135-44, 1992.
B.Littenberg, A.I. Mushlin, and the Diagnostic Technology AssessmentConsortium. Technetium bone scanning in the diagnosis of osteomyelitis: A meta-analysis of test performance. Journal of General Internal Medicine, 7:158-163, 1992.
B.Littenberg and L.E. Moses. Estimating diagnostic accuracy from multiple conflicting reports: A new meta-analytic method. Medical Decision Making, 13:313-321, 1993.
D.K. McClish. Combining and comparing area estimates across studies or strata. Medical Decision Making, 12:274-9, 1992.
A.S. Midgette, T.A. Stukel, and B.Littenberg. A meta-analytic method for summarizing diagnostic test performances. Medical Decision Making, 13:253-257, 1993.
L.E. Moses, D.E. Shapiro, and B.Littenberg. Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytic approaches and some additional considerations. Statistics in Medicine, 12:1293-1316, 1993.
H.E. Rockett, W.L. Campbell, C.A. Britton, J.M. Holbert, J.L. King, D. Gur. Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies. Academic Radiology, 6: 723-729.
C.M. Rutter and C.Gatsonis. Regression methods for meta-analysis of diagnostic test data. Academic Radiology,2:supple:48-56,1995.
D.E. Shapiro. Meta-analysis in ROC space: Combining independent studies of a diagnostic test, Technical Report 421, [Ph.D. Dissertation]. Department of Statistics, Stanford University, March 1993.
D.E. Shapiro. Issues in combining independent estimates of sensitivity and specificity of a diagnostic test. Academic Radiology,2:Suppl:37-47,1995.
J.A. Swets. Sensitivities and specificities of diagnostic tests. Journal of the American Medical Association, 248:548-549, 1982.
X.H. Zhou Empirical Bayes combination of estimated areas under ROC curves using estimating equations. Medical Decision Making, 16 :24-28, 1996.
X.H. Zhou Methods for combining rates from several studies. Statistics in Medicine, 18: 557-566, 1999.

I. CONTINUOUS MEASUREMENTS.

G.Campbell and J.M. Deleo. Fundamentals of fuzzy receiver operating characteristic (ROC) functions. Computer Science and Statistics: Proceedings of the 21st Symposium on the Inference, pages 543-548, 1989.
G.Campbell, D.Levy, A.Lausier, M.R. Horton, and J.J. Bailey. Nonparametric comparison of entire ROC curves for computerized ECG left ventricular hypertrophy algorithms using data from the Framingham Heart Study. Journal of Electrocardiology, 22 Suppl:152-7, 1989.
G.Campbell, D.Levy, and J.J. Bailey. Bootstrap comparison of fuzzy ROC curves for ECG-LVH algorithms using data from the Framingham Heart Study. Journal of Electrocardiology, 23 Suppl:132-7, 1990.
G.Campbell. Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Statistics in Medicine, 13:499-508, 1994.
M.J. Goddard and I.Hinberg. Receiver operator characteristic (ROC) curves and non-normal data: An empirical study. Statistics In Medicine, 9:325-37, 1990.
S.W. Greenhouse and N.Mantel. The evaluation of diagnostic tests. Biometrics, 6:399-412, 1950.
M.Jaraiedi and G.D. Herrin. A non-parametric method for fitting ROC curves for evaluation of industrial inspection performance. Applied Stochastic Models and Data Analysis, 4:79-88, 1988.
V. Kairisto. Optimal bin widths for frequency histograms and ROC curves [letter]. Clinical Chemistry, 41:766-7, 1995.
V. Kairisto. Software for illustrative presentation of basic clinical characteristics of laboratory tests--GraphROC for Windows. Scandinavian Journal of Clinical & Laboratory Investigation, 222(suppl):43-60,1995.
K.Linnet. Comparison of quantitative diagnostic tests: Type I error, power, and sample size. Statistics in Medicine, 6:147-158, 1987.
K.Linnet. A review on the methodology for assessing diagnostic tests. Clinical Chemistry, 34:1379-1386, 1988.
K.Linnet. Assessing diagnostic tests by a strictly proper scoring rule. Statistics in Medicine, 8:609-618, 1989.
E.A. Robertson, M.H. Zweig, and A.C. VanSteirtghem. Evaluating the clinical efficacy of laboratory tests. American Journal of Clinical Pathology, 79:78-86, 1983.
S.Sukhatme and C.A. Beam. Stratification in nonparametric ROC studies. Biometrics, 50(1):149-63, 1994.
Y.T. van der Schouw, H.Straatman, and A.L. Verbeek. ROC curves and the areas under them for dichotomized tests: empirical findings for logistically and normally distributed diagnostic test results. Medical Decision Making, 14(4):374-81, 1994.
M.H. Zweig. Evaluation of the clinical accuracy of laboratory tests. Archives of Pathology Laboratory Medicine, 112:383-6, 1988.
M.H. Zweig and G.Campbell. Receiver operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39:561-577, 1993.
H. Zou, D.E. Shapiro and W.J. Hall. Smooth nonparametric receiver operating characteristic curves for continuous diagnostic tests (abstract only). Medical Decision Making, 14:431, 1994.

J. GENERALIZATIONS (MULTIPLE DISEASES, LOCALIZATION).

D.P. Chakraborty, E.S. Breatnach, M.V. Yester, B.Soto, and G.T. Barnes. Digital and conventional chest imaging: A modified ROC study of observer performance using simulated nodules. Radiology, 158:35-9, 1986.
D.P. Chakraborty. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Medical Physics, 16:561-8, 1989.
D.P. Chakraborty and L.H. Winter. Free-response methodology: Alternate analysis and a new observer-performance experiment. Radiology, 174:873-81, 1990.
P. DeNeef and D.L. Kent. Using treatment-tradeoff preferences to select diagnostic strategies: Linking the ROC curve to threshold analysis. Medical Decision Making, 13 :126-132,1993.
S. Dreiseitl, L. Ohno-Machado, M. Binder. Comparing three-class diagnostic tests by three-way ROC analysis. Medical Decision Making, 20: 323-331, 2000.
O.Gefeller and H.Brenner. How to correct for chance agreement in the estimation of sensitivity and specificity of diagnostic tests. Methods of Information in Medicine, 33:180-186, 1994.
C.Herrmann, E.Buhr, D.Hoeschen and S.Y. Fan SYnewblock. Comparison of ROC and AFC methods in a visual detection task. Medical Physics, 20(3):805-11, 1993.
K. Hnatkova, J.D. Poloniecki, A.J. Camm, and M.Malik Computation of multifactorial receiver operator and predictive accuracy characteristics. Computer Methods and Programs in Biomedicine, 42(3):147-56, 1994.
W. Leisenring , M.S. Pepe G. Longton. A marginal regression modelling framework for evaluating medical diagnostic tests. 1996 ENAR meeting. Unpublished manuscript.
P.A. Murtaugh ROC curves with multiple marker measurements. Biometrics, 51 (4):1514-22,1995.
B. Reiser, D. Faraggi. Confidence intervals for the generalized ROC criterion. Biometrics, 53: 644-652, 2000.
H.E. Rockett, J.L. King, J.L. Medina, H.B. Eisen, M.L. Brown and D. Gur. Imaging systems evaluation: effect of subtle cases on the design and analysis of receiver operating characteristic studies. American Journal of Roentgenology, 165 :679-83,1995.
E.K. Shultz. Multivariate receiver-operating characteristic curve analysis: prostate cancer screening as an example. [Review] Clinical Chemistry,41:1248-55, 1995.
S.J. Starr, C.E. Metz, L.B. Lusted, and D.J. Goodenough. Visual detection and localization of radiographic images. Radiology, 116:533-8, 1975.
S.J. Starr, C.E. Metz, and L.B. Lusted. Comments on the generalization of receiver operating characteristic analysis to detection and localization tasks. Physics In Medicine and Biology, 22:376-81, 1977.
W.R. Steinbach and K.Richter. Multiple classification and receiver operating characteristic (ROC) analysis. Medical Decision Making, 7:234-7, 1987.
A. Taube. To judge the judges--kappa, ROC or what?. Arch Anat Cytol Pathol,43:227-33,1995.
L.L. Wheeless, R.D. Robinson, O.P. Lapets, C.Cox, A.Rubio, M.Weintraub, and L.J. Benjamin. Classification of red blood cells as normal, sickle, or other abnormal, using a single image analysis feature. Cytometry, 17(2):159-66, 1994.

K. MEDICAL DECISION MAKING (OPTIMAL CUTOFF, UTILITY).

R.Bairagi and C.M. Suchindran. An estimator of the cutoff point maximizing sum of sensitivity and specificity. Sankhya, Series B, Indian Journal of Statistics, 51:263-269, 1989.
G.R. Bergus. When is a test positive? The use of decision analysis to optimize test interpretation. Family Medicine, 25(10):656-60, 1993.
D.Borowiak and J.F. Reed, III. Utility of combining two diagnostic tests. Computer Methods and Programs in Biomedicine, 35:171-175, 1991.
F.A. Connell and T.D. Koepsell. Measures of gain in certainty from a diagnostic test. American Journal of Epidemiology, 121:744-53, 1985.
P.DeNeef and D.L. Kent. Using treatment-tradeoff preferences to select diagnostic strategies: Linking the ROC curve to threshold analysis. Medical Decision Making, 13:126-32, 1993.
W.L. England. An exponential model used for optimal threshold selection on ROC curves. Medical Decision Making, 8:120-31, 1988.
A.Fajardo-Gutierrez, L.Yamamoto-Kimura, L.Yanez-Velasco, J.Garduno-Espinosa, and M.C. Martinez-Garcia. The utility of joint sensitivity and specificity curves applied to a diagnostic test. Salud Publica de Mexico, 36(3):311-7, 1994.
D.G. Fryback and J.R. Thornbury. The efficacy of diagnostic imaging. Medical Decision Making, 11:95-101, 1991.
P.F. Griner, R.J. Mayewski, A.I. Mushlin, and P.Greenland. Selection and interpretation of diagnostic tests and procedures: Principles and applications. Annals of Internal Medicine, 94:553-592, 1981.
W.J. Hall and C.E. Phelps. A clinically meaningful measure of the difference between diagnostic technologies. Medical Decision Making, 7:286, 1987.
H.C. Kraemer. Evaluating Medical Tests: Objective and Quantitative Guidelines. Sage Publications, Newbury Park, California, 1992.
C.E. Metz, D.J. Goodenough, and K.Rossmann. Evaluation of receiver operating characteristic curve data in terms of information theory, with applications in radiography. Radiology, 109:297-303, 1973.
D.Mossman and E.Somoza. Maximizing diagnostic information from the dexamethasone suppression test: An approach to criterion selection using receiver operating characteristic analysis. Archives of General Psychiatry, 46:653-60, 1989.
D.Mossman and E.Somoza. Assessing improvements in the dexamethasone suppression test using receiver operating characteristics analysis. Biological Psychiatry, 25:159-73, 1989.
D.A. Noe. Selecting a diagnostic study's cutoff value by using its receiver operating characteristic curve. Clinical Chemistry, 29:571-2, 1983.
J.C. Peirce and R.G. Cornell. Integrating stratum-specific likelihood ratios with the analysis of ROC curves. Medical Decision Making, 13:141-51, 1993.
C.E. Phelps and A.I. Mushlin. Focusing technology assessment using medical decision theory. Medical Decision Making, 8:279-289, 1988.
F.Sainfort. Evaluation of medical technologies: A generalized ROC analysis. Medical Decision Making, 11:208-20, 1991.
H.Schafer. Constructing a cut-off point for a quantitative diagnostic test. Statistics in Medicine, 8:1381-1391, 1989.
E.Somoza Classifying binormal diagnostic tests using separation-asymmetry diagrams with constant-performance curves. Medical Decision Making, 14:157-68, 1994.
E.Somoza Classification of diagnostic tests. International Journal of Bio-Medical Computing, 37 :41-55, 1994.
E.Somoza and D.Mossman. ``Biological markers'' and psychiatric diagnosis: Risk-benefit balancing using ROC analysis. Biological Psychiatry, 29:811-26, 1991.
E.Somoza and D.Mossman. Comparing and optimizing diagnostic tests: An information-theoretical approach. Medical Decision Making, 12:179-88, 1992.
E.Somoza and D.Mossman. Comparing diagnostic tests using information theory: The INFO-ROC technique. Journal of Neuropsychiatry and Clinical Neurosciences, 4:214-9, 1992.
E.Somoza, L.Soutullo-Esperon, and D.Mossman. Evaluation and optimization of diagnostic tests using receiver operating characteristic analysis and information theory. International Journal of Bio-Medical Computing, 24:153-89, 1989.
J.Vermont, J.L. Bosson, P.Franccois, C.Robert, A.Rueff, and J.Demongeot. Strategies for graphical threshold determination. Computer Methods and Programs In Biomedicine, 35:141-50, 1991.
C.D. Ward. The differential positive rate, a derivative of receiver operating characteristic curves useful in comparing tests and determining decision levels. Clinical Chemistry, 32:1428-9, 1986.

L. LOGISTIC REGRESSION AND CLASSIFICATION TREES .

R.F. Raubertas, L.E. Rodewald, S.G. Humiston, and P.G. Szilagyi. ROC curves for classification trees. Medical Decision Making, 14:169-174, 1994.

M. MAMMOGRAPHY.

J.K. Gohagan, E.L. Spitznagel, M.M. McCrate, and T.B. Frank. ROC analysis of mammography and palpation for breast screening. Investigative Radiology, 19:587-92, 1984.
J.E. Goin. ROC curve estimation and hypothesis testing: Applications to breast cancer detection. Pattern Recognition, 15:263-269, 1982.
X.M. Tu, J. Kowalski, G. Jia. Bayesian analysis of prevalence with covariates using simulation-based techniques: Applications to HIV screening. Statistics in Medicine, 18: 3059-3073, 1999.
G. Tusch. Evaluation of partial classification algorithms using ROC curves. Medinfo, 8 :904-8,1995.

N. SAMPLE SIZE CONSIDERATIONS.

N.A. Obuchowski and D.K. McClish. Sample size determination for diagnostic accuracy studies involving binormal ROC curve parameters. Statistics in Medicine, 16: 1529-1542, 1997.
N.A. Obuchowski. Sample size tables for receiver operating characteristic studies. AJR, 175: 603-108, 2000.

O. MEASUREMENT ERROR MODELS.

D. Faraggi. THe effect of random measurement error on receiver operating characteristic (ROC) curves. Statistics in Medicine, 19: 61-70, 2000.
M. Coffin, S. Sukhatme. Receiver opearating characteristic studies and measurement errors. Biometrics, 53: 823-837, 1997.

P. UNCLASSIFIED.

T.A. Alonzo, M.S. Pepe. Distribution-free ROC analysis using binary regression techniques. Biostatistics, 3(3):421-432, 2002.
W.S. Andrus and K.T. Bird. Editorial: Radiology and the receiver operating characteristic (ROC) curve. Chest, 67:378-9, 1975.
J.Ashton and M.L. Moeschberger. A SAS macro for estimating the error rates of two diagnostic tests, neither being a gold standard. Proceedings of the SAS Users Group International Conference, 995-996, 1988.
S.R. Aylward. Continuous mixture modeling via goodness-of-fit ridges. Pattern Recognition, 35(9):1821-1833, 2002.
S.V. Beiden, M.A. Maloof, R.F. Wagner. A general model for finite-sample effects in training and testing of competing classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12):1561-1569, 2003.
S.V. Beiden, R.F. Wagner, G. Campbell. Components-of-variance models and multiple-bootstrap experiments: An alternative method for random-effects, receiver operating characteristic analysis. Academic Radiology, 7(5):341-349, 2000.
S.V. Beiden, R.F. Wagner, G. Campbell et al. Components-of-variance models for random-effects ROC analysis: The case of unequal variance structures across modalities. Academic Radiology, 8(7):605-615, 2001.
S.V. Beiden, R.F. Wagner, G. Campbell et al. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Academic Radiology, 8(7):616-622, 2001.
B.M. Bennett. On tests for equality of predictive values for t diagnostic procedures. Statistics in Medicine, 4:535-539, 1985.
K. Bernstein, K. Matthews. Correlation of system performance parameters to ROC analysis of PET/CT images. Medical Physics, 31(6):1819-1819, 2004.
J.Brismar. Understanding receiver-operating-characteristic curves: A graphic approach. American Journal of Roentgenology, 157:1119-21, 1991.
W.S. Browner and T.B. Newman. Are all significant p values created equal? the analogy between diagnostic tests and clinical research. Journal of the American Medical Association, 257:2459-2463, 1987.
D.P. Chakraborty, K.S. Berbaum Observer studies involving detection and localization: Modeling, analysis, and validation. Medical Physics, 31(8):2313-2330, 2004.
H.P. Chan, B. Sahiner, R.F. Wagner, et al. Classifier design for computer-aided diagnosis: Effects of finite sample size on the mean performance of classical and neural network classifiers. Medical Physics, 26(12):2654-2668, 1999.
Y.Charpak, C.Blery, and C.Chastang. Designing a study for evaluating a protocol for the selective performance of preoperative tests. Statistics in Medicine, 6:813-822, 1987.
J. Chen J, K.M. Conigrave, P. Macaskill, et al. Combining carbohydrate-deficient transferrin and gamma-glutamyltransferase to increase diagnostic accuracy for problem drinking. Alcohol and Alcoholism, 38(6):574-582, 2003.
M.S. Chester. Human visual perception and ROC methodology in medical imaging. Physics In Medicine and Biology, 37:1433-76, 1992.
T.E. Cohn, D.G. Green, and W.P. Tanner, Jr. Receiver operating characteristic analysis: Application to the study of quantum fluctuation effects in optic nerve of rana pipiens. Journal of General Physiology, 66:583-616, 1975.
R. Dahyot, P. Charbonnier, F. Heits A Bayesian approach to object detection using probabilistic appearance-based models. Pattern Analysis and Applications, 7(3):317-332, 2004.
R.Detrano. Accuracy curves: An alternative graphical representation of probability data. Journal of Clinical Epidemiology, 42:983-6, 1989.
S. Dreiseitl, L. Ohno-Machado, M. Binder. Comparing three-class diagnostic tests by three-way ROC analysis. Medical Decision Making, 20(3):323-331, 2000.
D.C. Edwards, C.E. Metz, R.W. Nishikawa. The hypervolume under the ROC hypersurface of "near-guessing" and "near-perfect" observers in N-class classification tasks. IEEE Transactions of Medical Imaging, 24(3):293-299, 2005.
D.C. Edwards, C.E. Metz, M.A. Kupinski. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Transactions on Medical Imaging, 23(7):891-895, 2004.
D. Faraggi, B. Reiser, E.F. Schisterman. ROC curve analysis for biomarkers based on pooled assessments. Statistics in Medicine, 22(15):2515-2527, 2003.
T. Fawcett, P.A. Flach A response to Webb and Ting's On the application of ROC analysis to predict classification performance under varying class distributions. Machine Learning, 58(1):33-38, 2005.
A.R. Feinstein. On the sensitivity, specificity, and discrimination of diagnostic tests. Clinical Pharmacology and Therapeutics, 17:104-118, 1975.
C. Ferri, J. Hernandez-Orallo, M.A. Salido. Volume under the ROC surface for multi-class problems. Lecture Notes in Artificial Intelligence, 2837:108-120, 2003.
J.Fuentes and M.Martinez. An alternative to receiver-operating characteristic curves. Clinical Chemistry, 29:1445, 1983.
J. Furnkranz, P.A. Flach. ROC 'n' rule learning - Towards a better understanding of covering algorithms. Machine Learning, 58(1):39-77, 2005.
M.H. Gail and S.B. Green. A generalization of the one-sided two-sample Kolmogorov-Smirnov statistic for evaluating diagnostic tests. Biometrics, 32:561-570, 1976.
O. Gefeller and H. Brenner. How to correct for chance agreement in the estimation of sensitivity and specificiety of diagnostic tests. Meth. Inform. in Med, :180-6,1994.
A.J. Girling. Rank statistics expressible as integrals under P-P-plots and receiver operating characteristic curves. Journal of the Royal Statistical Society Series B-Statistical Methodology, 62:367-382, 2000.
P.Glasziou and J.Hilden. Decision tables and logic in decision analysis. Medical Decision Making, 6:154-60, 1986.
G. Gorog. An Excel program for calculating and plotting receiver-operator characteristic (ROC) curves, histograms and descriptive statistics. Computers in Biology and Medicine, 24(3):167-9, 1994.
M. Greiner, D. Pfeiffer, R.D. Smith. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Preventive Veterinary Medicine, 45(1-2):23-41, 2000.
D.H. Gustafson, F. Sainfort, M. Eichler, et al. Developing and testing a model to predict outcomes of organizational change. Health Services Research, 38(2):751-776, 2003.
K.O. Hajian-Tilaki, J.A. Hanley. Comparison of three methods for estimating the standard error of the area under the curve in ROC analysis of quantitative data. Academic Radiology, 9(11):1278-1285, 2002.
A.P. Hallstrom and G.B. Trobaugh. Specificity, sensitivity, and prevalence in the design of randomized trials: A univariate analysis. Controlled Clinical Trials, 6:128-135, 1985.
T. Hansen, H. Neumann. A biologically motivated scheme for robust junction detection. Lecture Notes in Computer Science, 2525:16-26, 2002
M.B. Harrington. Some methodological questions concerning receiver operating characteristic (ROC) analysis as a method for assessing image quality in radiology. Journal of Digital Imaging, 3:211-8, 1990.
R.D. Hays. ROC: Estimation of the area under a receiver operating characteristic curve. Applied Psychological Measurement Inc., 14:208-208, 1990.
J.W. Hoppin, M.A. Kupinski, G.A. Kastis, et al. Objective comparison of quantitative imaging modalities without the use of a gold standard. IEEE Transactions on Medical Imaging, 21(5):441-449, 2002.
J.M. Irvine. Assessing target search performance: the free-response operator characteristic model. Optical Engineering, 43(12):2926-2934,2004.
H. Ishwaran, C.A. Gatsonis. A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. Canadian Journal of Statistics-Revue Canadienne de Statistique, 28(4):731-750, 2000.
R.H. Jones and M.W. McClatchey. Beyond sensitivity, specificity and statistical independence. Statistics in Medicine, 7:1289-1295, 1988.
V. Kairisto and A. Poola. Software for illustrative presentation of basic clinical characteristics of laboratory tests--GraphROC for Windows. Scandinavian Journal of Clinical & Laboratory Investigation, 222 (supple):43-60,1995.
L.Kuncheva and K.Andreeva. DREAM: a shell-like software system for medical data analysis and decision support. Computer Methods and Programs in Biomedicine, 40(2):73-81, 1993.
M.A. Kupinski,D.C. Edwards, M.L. Giger, et al. Ideal observer approximation using Bayesian classification neural networks. IEEE Transactions on Medical Imaging, 20(9):886-899, 2001.
P.A. Lachenbruch. Multiple reading procedures: The performance of diagnostic tests. Statistics in Medicine, 7:549-557, 1988.
A.R. Laird, B.P. Rogers, M.E. Meyerand. Comparison of Fourier and wavelet resampling methods. Magnetic Resonance in Medicine, 51(2):418-422, 2004.
R.A. Lew and P.S. Levy. Estimation of prevalence on the basis of screening tests. Statistics in Medicine, 8:1225-1230, 1989.
R.B. Lirio, I.C. Donderiz, and M.C. PerezAbalo. Maximum likelihood estimation of signal detection model parameters for the assessment of two-stage diagnostic strategies. International Journal of Bio-Medical Computing, 31:117-26, 1992.
B. Liu, C.E. Metz, Y.L. Jiang. An ROC comparison of four methods of combining information from multiple images of the same patient. Medical Physics, 31(9):2552-2563, 2004.
A.Y. Liu, E.F. Schisterman, E. Teoh. Sample size and power calculation in comparing diagnostic accuracy of biomarkers with pooled assessments. Journal of Applied Statistics 31(1):49-59, 2004.
A.Y. Liu, E.F. Schisterman, Y. Zhu. On linear combinations of biomarkers to improve diagnostic accuracy. Statistics in Medicine, 24(1):37-47, 2005.
Y. Lu, D.N. Heller, S.J. Zhao. Receiver operating characteristic (ROC) analysis for diagnostic examinations with uninterpretable cases. Statistics in Medicine, 21(13):1849-1865, 2002.
K. Lui. A discussion on the conventional estimator of sensitivity and specificity in multiple tests. Statistics in Medicine, 8:1231-1240, 1989.
P. Macaskill. Empirical Bayes estimates generated in a hierarchical summary ROC analysis agreed closely with those of a full Bayesian analysis. Journal of Clinical Epidemiology, 57(9):925-932, 2004.
F.A. Mann, C.F. Hildebolt, and A.J. Wilson. Statistical analysis with receiver operating characteristic curves. Radiology, 184:37-8, 1992.
R.J. Marshall. The predictive value of simple rules for combining two diagnostic tests. Biometrics, 45:1213-1222, 1989.
C. Metz. Models in medicine IV. ROC analysis: Methodology and practical applications. Medical Physics, 30(6):1442-1442, 2003.
D.K. McClish and D.Quade. Improving estimates of prevalence by repeated testing. Biometrics, 41:81-89, 1985.
M.J. Monsour, A.T. Evans, and L.L. Kupper. Confidence intervals for post-test probability. Statistics in Medicine, 10:443-456, 1991.
A.I. Mushlin, A.S. Detsky, C.E. Phelps, P.W. O'Connor D.K. Kido, W.Kucharczyk, D.W. Giang, C.Mooney, C.M. Tansey and W.J. Hall. The accuracy of magnetic resonance imaging in patients with suspected multiple sclerosis. The Rochester-Toronto Magnetic Resonance Imaging Study Group. JAMA, 269(24):3146-51, 1993.
D.B. Mumford. ROC analysis. British Journal of Psychiatry, 157:301, 1990.
C.T. Nakas, C.T. Yannoutsos. Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine, 23(22):3437-3449, 2004.
N.A. Obuchowski. How many observers are needed in clinical studies of medical imaging? American Journal of Roentgenology, 182(4):867-869, 2004.
N.A. Obuchowski. Fundamentals of clinical research for radiologists - ROC analysis American Journal of Roentgenology, 184(2):364-372, 2005.
W.Palmas, T.A. Denton, A.P. Morise, and G.A. Diamond. Afterimages: integration of diagnostic information through Bayesian enhancement of scintigraphic images American Heart Journal, 128(2):281-7, 1994 Aug.
J. Paetz. Finding optimal decision scores by evolutionary strategies. Artificial Intelligence in Medicine, 32(2):85-95, 2004.
S.H. Park, J.M. Goo, C.H. Jo. Receiver operating characteristic (ROC) curve: Practical review for radiologists. Korean Journal of Radiology, 5(1):11-18, 2004.
G. Parker, D. Hadzi-Pavlovic, L. Both, et al. Measuring disordered personality functioning: to love and to work reprised. Acta Psychiatrica Scandinavica, 110(3):230-239, 2004.
J.L. Pater and A.R. Willan. Clinical trials as diagnostic tests. Controlled Clinical Trials, 5:107-113, 1984.
P.E. Politser. Explanations of statistical concepts: Can they penetrate the haze of Bayes? Methods of Information in Medicine, 23:99-108, 1984.
C. Perlich, F. Provost, J.S. Simonoff. Tree induction vs. logistic regression: A learning-curve analysis. Journal of Machine Learning Research, 4(2):211-255, 2004.
G. Regehr, J. Colliver. On the equivalence of classic ROC analysis and the loss-function model to set cut points in sequential testing. Academic Medicine, 78(4):361-364, 2003.
M.Rodriguez, I.Moussa, J.Froning, M.Kochumian, and V.F. Froelicher. Improved exercise test accuracy using discriminant function analysis and "recovery ST slope". Journal of Electrocardiology, 26(3):207-18, 1993.
H.E. Rockett, J.L. KIng, J.L. Medina, H.B. Eisen, M.L. Brown, D. Gur. Imaging systems evaluation: effect of subtle cases on the design and analysis of receiver operating characteristic studies. American Journal of Roentgenology,165:679-83, 1995.
B. Sahiner, H.P. Chan, N. Petrick, et al. Feature selection and classifier performance in computer-aided diagnosis: The effect of finite sample size. Medical Physics, 27(7):1509-1522, 2000.
M.S. Scher, B.L. Jones, D.A. Steppe, et al. Functional brain maturation in neonates as measured by EEG-sleep analyses. Clinical Neurophysiology, 114(5):875-882, 2003.
E.F. Schisterman, D. Faraggi, B.Reiser. Adjusting the generalized ROC curve for covariates. Statistics in Medicine, 23(21):3319-3331, 2004.
P.M. Shankar, V.A. Dumane, T. George, et al. Classification of breast masses in ultrasonic B scans using Nakagami and K distributions. Physics in Medicine and Biology, 48(14):2229-2240, 2003.
P.M. Shankar, V.A. Dumane, C.W. Piccoli, et al. Computer-aided classification of breast masses in ultrasonic B-scans using a multiparameter approach. IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control 50(8):1002-1009, 2003.
H.C. Sox, Jr. Probability theory in the use of diagnostic tests. Annals of Internal Medicine, 104:60-66, 1986.
C. Stephan, S. Wesseling, T. Schink, et al. Comparison of eight computer programs for receiver-operating characteristic analysis. Clinical Chemistry, 49(3):433-439, 2003.
R.G. Swensson, J.L. King, D. Gur. A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation. Medical Physics, 28(8):1597-1609, 2001.
Y. Takahashi, K. Murase, H. Higashino, et al. Receiver operating characteristic (ROC) analysis of images reconstructed with iterative expectation maximization algorithms. Annals of Nuclear Medicine, 15(6):521-525, 2001.
A.Taube. Sensitivity, specificity and predictive values: A graphical approach. Statistics in Medicine, 5:585-591, 1986.
L.A. Thibodeau. Evaluating diagnostic tests. Biometrics, 37:801-804, 1981.
L.A. Thibodeau. Sensitivity and specificity. Encyclopedia of Statistical Sciences, 8:370-372, 1988.
M.L. Thompson. Assessing the diagnostic accuracy of a sequence of tests. Biostatistics, 4(3):341-351, 2003.
M.Treisman and A.Faulkner. The effect of signal probability on the slope of the receiver operating characteristic given by the rating procedure. British Journal of Mathematical and Statistical Psychology, 37199-215, 1984.
P.M. Vacek. The effect of correlation between two diagnostic tests on maximum likelihood estimates of test error rates and disease prevalences. In ASA Proceedings of the Statistical Computing Section, 271-276, 1984.
P.M. Vacek, M.Kresnow, and R.M. Potter. The sampling distributions of estimators for the error rates of diagnostic tests. ASA Proceedings of the Statistical Computing Section, 438-440, 1985.
R. Wagner. Multiple-reader, multiple-case (MRMC) ROC analysis in diagnostic imaging, computer-aided diagnosis, and statistical pattern recognition. Medical Physics, 29(6):1331-1331, 2002.
R.F. Wagner, S.V. Beiden, G. Campbell, et al. Assessment of medical imaging and computer-assist systems: Lessons from recent experience. Academic Radiology, 9(11):1264-1277, 2002.
R.F. Wagner, S.V. Beiden, C.E. Metz. Continuous versus categorical data for ROC analysis: Some quantitative considerations. Academic Radiology, 8(4):328-334, 2001.
J.J. Wang, Z. Wang, G.K. Aguirre, et al. To smooth or not to smooth? - ROC analysis of perfusion fMRI data. Magnetic Resonance Imaging, 23(1):75-81, 2005.
G.I. Webb, K.M. Ting. On the application of ROC analysis to predict classification performance under varying class distributions. Machine Learning, 58(1):25-32, 2005.
M. Weiner, R. Centor. Web-based, interactive receiver operating characteristics (ROC) analysis. Journal of the American Medical Informatics Association, Suppl. S:1058-1058, 2001.
T. Wigren, S. Remle, B. Wahlberg. Analysis of a low-complexity change detection scheme. International Journal of Adaptive Control and SIgnal Processing, 14(5):481-503, 2000.
A.A. White and J.R. Landis. A general categorical data methodology for evaluating medical diagnostic tests. Communication in Statistics, Part A-Theory and Methods, 117:567-605, 1982.
T.Yanagawa and B.C. Gladen. Estimating disease rates from a diagnostic test. American Journal of Epidemiology, 119:1015-1023, 1984.
T.Yanagawa and F.Kasagi. Estimating prevalence and incidence of disease from a diagnostic test. Statistical Theory and Data Analysis, 783-801, 1985.
K.H. Zou, S.K. Warfield, J.R. Fielding, et al. Statistical validation based on parametric receiver operating characteristic analysis of continuous classification data. Academic Radiology, 10(12):1359-1368, 2003.