Simultaneous truth and performance level estimation (STAPLE): An algorithm for the validation of image segmentation

IEEE Transactions on Medical Imaging

Volumn 23, Issue 7, 2004, Pages 903-921

  Warfield, Simon K ^a,b,c   Zou, Kelly H ^a   Wells, William M ^a  

^a BRIGHAM AND WOMEN S HOSPITAL   (United States)

^b BOSTON CHILDREN S HOSPITAL   (United States)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Accuracy; Classifier fusion; Expectation maximization; Gold standard; Ground truth; Markov random field; Precision; Segmentation; Sensitivity; Specificity; STAPLE; Validation

Indexed keywords

ACCURACY; CLASSIFIER FUSION; EXPECTATION-MAXIMIZATION; GOLD STANDARD; GROUND TRUTH; SEGMENTATION PROTOCOL; VALIDATION;

ALGORITHMS; CONSTRAINT THEORY; DATA REDUCTION; IMAGING TECHNIQUES; MARKOV PROCESSES; MEDICAL IMAGING; PARAMETER ESTIMATION; SENSITIVITY ANALYSIS;

IMAGE SEGMENTATION;

ALGORITHM; ARTICLE; BRAIN; COMPARATIVE STUDY; COMPUTER ASSISTED DIAGNOSIS; DECISION SUPPORT SYSTEM; HISTOLOGY; HUMAN; IMAGE ENHANCEMENT; IMAGE QUALITY; METHODOLOGY; NEWBORN; NUCLEAR MAGNETIC RESONANCE IMAGING; OBSERVER VARIATION; PROBABILITY; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; STATISTICAL MODEL; VALIDATION STUDY;

ALGORITHMS; BRAIN; DECISION MAKING, COMPUTER-ASSISTED; HUMANS; IMAGE ENHANCEMENT; IMAGE INTERPRETATION, COMPUTER-ASSISTED; INFANT, NEWBORN; MAGNETIC RESONANCE IMAGING; MARKOV CHAINS; MODELS, STATISTICAL; OBSERVER VARIATION; PHANTOMS, IMAGING; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY;

EID: 1942438249     PISSN: 02780062     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMI.2004.828354     Document Type: Article

References (62)

1. D. Nicoll and W. Detmer, Basic Principles of Diagnostic Test Use and Interpretation. New York: MCGraw-Hill, 2001, ch. 1, pp. 1-16.
2. M. Styner, C. Brechbühler, G. Székely, and G. Gerig, "Parametric estimate of intensity inhomogeneities applied to MRI." IEEE Trans. Med. Imag., vol. 19, no. 3, pp. 153-165, Mar. 2000.
3. D. Collins, A. Zijdenbos, V. Kollokian, J. Sled, N. Kabani, C. Holmes, and A. Evans, "Design and construction of a realistic digital brain phantom," IEEE Trans. Med. Imag., vol. 17. no. 3, pp. 463-468, June 1998.
4. V. Spitzer, M. J. Ackerman, A. L. Scherzinger, and D. Whitlock, "The visible human male: A technical report." J. Amer. Med. Inform. Assoc., vol. 3, no. 2, pp. 118-130, 1996.
5. T. S. Yoo, M. J. Ackerman, and M. Vannier, "Toward a common validation methodology for segmentation and registration algorithms," in Proc. 3rd Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2000). A. M. DiGioia and S. Delp, Eds., 2000, pp. 422-431.
6. D. V. Iosifescu, M. E. Shenton, S. K. Warfield, R. Kikinis, J. Dengler, F. A. Jolesz, and R. W. MCCarley, "An automated registration algorithm for measuring MRI subcortical brain structures." NeuroImage, vol. 6, pp. 12-25, 1997.
7. S. K. Warfield, R. V. Mulkern, C. S. Winalski, F. A. Jolesz, and R. Kikinis, "An image processing strategy for the quantification and visualization of exercise induced muscle MRI signal enhancement," J. Magn. Reson. Imag., vol. 11. no. 5, pp. 525-531, May 2000.
8. J. M. Bland and D. G. Altman, "Applying the right statistics: Analyses of measurement studies," Ultrasound Obstet. Gynecol., vol. 22, pp. 85-93, 2003.
9. D. N. Levin, X. Hu, K. K. Tan, and S. Galhotra, "Surface of the brain: Three-dimensional MR images created with volume rendering." Radiology, vol. 171, pp. 277-280, 1989.
10. M. Everingham, H. Muller, and B. Thomas, "Evaluating image segmentation algorithms using the pareto front," in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2002, vol. 2353, ECCV, pp. 34-48.
11. L. R. Dice, "Measures of the amount of ecologie association between species," Ecology, vol. 26, no. 3, pp. 297-302, 1945.
12. P. Jaccard, "The distribution of flora in the alpine zone," New Phytologist, vol. 11, pp. 37-50, 1912.
13. R. E. Hogan, K. E. Mark, L. Wang, S. Joshi, M. I. Miller, and R. D. Bucholz, "Mesial temporal sclerosis and temporal lobe epilepsy: MR imaging deformation-based segmentatino of the hippocampus in five patients," Radiology, vol. 216, pp. 291-297, 2000.
14. F. Bello and A. C. F. Colchester, "Measuring global and local spatial correspondence using information theory," in Proc. 1st Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 98), 1998, pp. 964-973.
15. K. H. Zou, W. M. Wells, R. Kikinis, and S. K. Warfield, "Three validation metrics for automated probabilistic image segmentation in brain tumors," Statist. Med., vol. 23. pp. 1259-1282, 2003.
16. G. Gerig, M. Jomier, and M. Chakos, "Valmet: A new validation tool for assessing and improving 3D object segmentation," in Proc. 4th Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2001), M. A. Viergever, Ed., Utrecht, The Netherlands, 2001. pp. 516-523.
17. A. P. Zijdenbos, B. M. Dawant, R. A. Margolin, and A. C. Palmer, "Morphometric analysis of white matter lesions in MR images: Method and validation," IEEE Trans. Med. Imag., vol. 13, pp. 716-724, Dec. 1994.
18. S. Warfield, J. Dengler, J. Zaers, C. R. Guttmann, W. M. Wells III, G. J. Ettinger, J. Hiller, and R. Kikinis, "Automatic identification of grey matter structures from MRI to improve the segmentation of white matter lesions." J. Image Guid. Surg., vol. 1, no. 6, pp. 326-338, 1995.
19. R. Kikinis, M. E. Shenton, G. Gerig, J. Martin, M. Anderson, D. Metcalf, C. R. G. Guttmann, R. W. MCCarley, W. E. Lorenson, H. Cline, and F. Jolesz, "Routine quantitative analysis of brain and cerebrospinal fluid spaces with MR imaging," J. Magn. Reson. Imag., vol. 2, pp. 619-629, 1992.
20. L. P. Cordella, P. Foggia, C. Sansone, F. Tortorella, and M. Vento, "Reliability parameters to improve combination strategies in multi-expert systems," Pattern Anal. Applicat., vol. 2, pp. 205-214, 1999.
21. T. K. Ho, J. H. Hull, and S. N. Srihari, "Decision combination in multiple classifier systems," IEEE Trans. Pattern Anal. Machine Intell., vol. 16, pp. 66-75, Jan. 1994.
22. J. Kittler, M. Hatef, R. P. W. Duin, and J. Matas, "On combining classifiers," IEEE Trans. Pattern Anal. Machine Intell., vol. 20, pp. 226-239, Mar. 1998.
23. D. Windridge and J. Kittler, "A morphologically optimal strategy for classifier combination: Multiple expert fusion as a tomographic process," IEEE Trans. Pattern Anal. Machine Intell., vol. 25, pp. 343-353, Mar. 2003.
24. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, "Adaptive mixtures of local experts," Neural Computation, vol. 3, pp. 79-87, 1991.
25. M. I. Jordan and R. A. Jacobs, "Hierarchical Mixtures of Experts and the EM Algorithm,", Tech. Rep. AIM-1440, 1993.
26. P. Melville, R. J. Mooney, and R. Nagarajan, "Content-boosted collaborative filtering for improved recommendations," in Proc. 18th Nat. Conf. Artificial Intelligence, vol. AAAI-2002, 2002, pp. 187-192.
27. L. Irwig, P. Macaskill, P. Glasziou, and M. Fahey, "Meta-analytic methods for diagnostic test accuracy," J. Clin. Epidemiol., vol. 48. no. 1, pp. 119-130, 1995.
28. S.-L. Normand, "Tutorial in biostatistics meta-analysis: Formulating, evaluating, combining, and reporting," Statist. Med., vol. 18, pp. 321-359, 1999.
29. T. A. Alonzo and M. S. Pepe, "Using a combination of reference tests to assess the accuracy of a new diagnostic test," Statist. Med., vol. 18, pp. 2987-3003, 1999.
30. S. V. B. SV, G. Campbell, K. L. Meier, and R. F. Wagner, "On the problem of ROC analysis without truth: The EM algorithm and the information matrix," Proc. SPIE, vol. 3981, pp. 126-134, 2000.
31. A. Dempster, N. Laird, and D. Rubin, "Maximum-likelihood from incomplete data via the EM algorithm," J. Roy. Statist. Soc. Ser. B., vol. 39, pp. 34-37, 1977.
32. G. J. MCLachlan and T. Krishnan, The EM Algorithm and Extensions. New York: Wiley-Interscience, 1996.
33. S. K. Warfield, K. H. Zou, M. R. Kaus, and W. M. Wells, "Simultaneous validation of image segmentation and assessment of expert quality," in Proc. Int. Symp. Biomedical Imaging: Macro to Nano, J. Fessler and M. Vannier, Eds., Washington, DC, 2002, pp. 1494:1-1494:4.
34. S. K. Warfield, K. H. Zou, and W. M. Wells, "Validation of image segmentation and expert quality with an expectation-maximization algorithm," in Proc. 5th Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2002),. Tokyo, Japan, 2002, pp. 298-306.
35. J. Rexilius, "Physics-Based Nonrigid Registration for Medical Image Analysis," masters thesis, Med. Univ. Luebeck, Luebeck, Germany, 2001.
36. K. Friston, J. Ashburner, J. Poline, C. Frith, J. Heather, and R. Frackowiak. "Spatial registration and normalization of images," Human Brain Mapping, vol. 2, pp. 165-189, 1995.
37. D. L. Collins, "3D Model-based segmentation of individual brain structures from magnetic resonance imaging data." Ph.D. dissertation, MCGill Univ., Montreal, Canada, 1994.
38. E. Ising, "Beitrag zur Theorie des Ferromagnetismus," Zeitschrift f. Physik., vol. 31. pp. 253-258, 1925.
39. T. Kapur, W. Grimson, R. Kikinis, and W. W. III, "Enhanced spatial priors for segmentation of magnetic resonance imagery," in Proc. 1st Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 98), 1998, pp. 457-468.
40. K. Van Leemput, F. Maes, D. Vanermeulen, and P. Suetens, "Automated model-based bias field correction of MR images of the brain," IEEE Trans. Med. Imag., vol. 18, pp. 885-895, Oct. 1999.
41. Y. Zhang, M. Brady, and S. Smith, "Segmentation of brain mr images through a hidden Markov random field model and the expectation-maximization algorithm." IEEE Trans. Med. Imag., vol. 20, pp. 45-57, Jan 2001.
42. D. M. Greig, B. T. Porteous, and A. H. Seheult, "Exact maximum a posteriori estimation for binary images." J. Roy. Statist. Soc. B, vol. 51, no. 2, pp. 271-279, 1989.
43. H. Ishikawa, "Exact optimization for Markov random fields with convex priors," IEEE Trans. Pattern Anal. Machine Intell., vol. 25, pp. 1333-1336. Oct. 2003.
44. I. M. Elfadel, "From fields to networks," Ph.D. dissertation, Massachussetts Inst. Technol., Cambridge, MA, 1993.
45. J. Besag. "On the statistical analysis of dirty pictures," J. Roy. Statist. Soc. B, vol. 48, no. 3. pp. 259-302, 1986.
46. _, "Spatial interaction and the statistical analysis of lattice systems," J. Roy. Statist. Soc. Ser. B., vol. 36, pp. 192-235, 1974.
47. T. Kapur, "Model-based three-dimensional medical image segmentation." Ph.D. dissertation, Massachussetts Inst. Technol., Cambridge, MA, 1999.
48. J. Zhang and G. G. Hanauer, "The application of mean field theory to image motion estimation," IEEE Trans. Imag. Processing, pp. 19-33, Jan. 1995.
49. D. Geieger and F. Girosi, "Parallel and deterministic algorithms from MRF's: Surface reconstruction," IEEE Pattern Anal. Machine Intell., vol. 13, pp. 401-412, May 1991.
50. J. Edmonds and R. M. Karp, "Theoretical improvements in algorithmic efficiency for network flow problems," J. Assoc. Comput. Mach., vol. 19, pp. 248-264, 1972.
51. J. Rexilius, S. K. Warfield, C. R. G. Guttmann, X. Wei, R. Benson, L. Wolfson, M. Shenton, H. Handels, and R. Kikinis, "A novel nonrigid registration algorithm and applications," in Proc. 4th Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2001), W. J. Niessen and M. A. Viergever, Eds., Utrecht, The Netherlands, 2001, pp. 923-931.
52. A. Bharatha, M. Hirose, N. Hata, S. K. Warfield, M. Ferrant, K. H. Zou, E. Suarez-Santana, J. Ruiz-Alzola, A. D'Amico, R. A. Cormack, R. Kikinis, F. A. Jolesz, and C. M. Tempany, "Evaluation of three-dimensional finite element-based deformable registration of pre- and intra-operative prostate imaging," Med. Phys., vol. 28, pp. 2551-2560, Dec. 2001.
53. M. Kaus, S. K. Warfield, A. Nabavi, P. M. Black, F. A. Jolesz, and R. Kikinis, "Automated segmentation of MRI of brain tumors," Radiology, vol. 218, pp. 586-591, Feb. 2001.
54. S. K. Warfield, M. Kaus, F. A. Jolesz, and R. Kikinis, "Adaptive, template moderated, spatially varying statistical classification," Med. Image Anal., vol. 4, no. 1, pp. 43-55, Mar. 2000.
55. _, "Adaptive template moderated spatially varying statistical classification," in Proc. 1st Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 98), W. M. Wells, A. Colchester, and S. Delp. Eds., MA, 1998, pp. 231-238.
56. P. Hüppi, S. Warfield, R. Kikinis, P. D. Barnes, G. P. Zientara, F. A. Jolesz, M. K. Tsuji, and J. J. Volpe, "Quantitative magnetic resonance imaging of brain development in premature and mature newborns," Ann. Neural., vol. 43, no. 2, pp. 224-235, Feb. 1998.
57. T. E. Inder, P. S. Huppi, S. K. Warfield, R. Kikinis, G. P. Zientara, P. D. Barnes, F. A. Jolesz, and J. J. Volpe, "Periventricular white matter injury in the premature infant is followed by reduced cerebral cortical gray matter volume at term," Ann. Neurol., vol. 46, no. 5, pp. 755-760, 1999.
58. B. P. Murphy, T. E. Inder, P. S. Huppi, S. Warfield, G. P. Zientara, R. Kikinis, F. A. Jolesz, and J. J. Volpe, "Impaired cerebral cortical gray matter growth after treatment with dexamethasone for neonatal chronic lung disease," Pediatrics, vol. 107, no. 2, pp. 217-221, Feb. 2001.
59. A. Lefohn, J. Cates, and R. Whitaker, "Interactive, CPU-Based Level Sets for 3D Segmentation," in Proc. 6th Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2003), 2003, pp. 564-572.
60. C. Fennema-Notestine, I. B. Ozyurt, G. G. Brown, C. P. Clark, S. Morris, A. Bischoff-Grethe, M. W. Bondi, and T. L. Jernigan, (2003) Bias correction, pulse sequence, and neurodegeneration influence performance of automated skull-stripping methods. Program 863.23. 2003 Abstract Viewer/Itinerary Planner. Washington, CD: Soc. Neurosci., 2003. [Online].
61. T. Rohlfing, D. B. Russakoff, and C. R. Maurer, "Expectation maximization strategies for multi-atlas multi-label segmentation," in Proc. Int. Conf. Information Processing in Medical Imaging, 2003, pp. 210-221.
62. _, "Extraction and application of expert priors to combine multiple segmentations of human brain tissue," in Proc. 6th Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2003), 2003, pp. 578-585.