Performance evaluation of image processing algorithms in CADe

Technology in Cancer Research and Treatment

Volumn 4, Issue 2, 2005, Pages 159-172

  Wirth, Michael A ^a  

^a UNIVERSITY OF GUELPH   (Canada)

Author keywords

Algorithm; Design process; Image databases; Mammogram; Performance evaluation

Indexed keywords

ACCURACY; ALGORITHM; BREAST CANCER; CANCER SCREENING; COMPUTER AIDED DESIGN; CONFERENCE PAPER; DATA BASE; EVALUATION; HISTOGRAM; IMAGE PROCESSING; MAMMOGRAPHY; MATHEMATICAL ANALYSIS; RECEIVER OPERATING CHARACTERISTIC; RELIABILITY; REPRODUCIBILITY; SENSITIVITY ANALYSIS;

EID: 17644415996     PISSN: 15330346     EISSN: None     Source Type: Journal    
DOI: 10.1177/153303460500400205     Document Type: Conference Paper

References (105)

1. Bird, R. E., Wallace, T. W., and Yankaskas, B. C. Analysis of Cancers Missed at Screening Mammography. Radiology 184, 613-617 (1992).
2. Gur, D., Sumkin, J. H., Rockette, H. E., Ganott, M. A., Hakim, C., Hardesty, L. A., Poller, W. R., Shah, M., and Wallace, L. Changes in Breast Cancer Detection and Mammography Recall Rates after the Introduction of a Computer-aided Detection System. Journal of the National Cancer Institute 96, 185-190 (2004).
3. Goergen, S. K., Evans, J., Cohen, G. P., and MacMillan, J. H. Characteristics of Breast Carcinomas Missed by Screening Radiologists. Radiology 204, 131-135 (1997).
4. Nodine, C. F., Kundel, H. L., Lauver, S. C., and Toto, L. C. Nature of Expertise in Searching Mammograms for Breast Cancer. Academic Radiology 3, 1000-1006 (1996).
5. Birdwell, R. L., Ikeda, D. M., O'Shaughnessy, K. F., and Sickles, E. A. Mammographic Characteristics of 115 Missed Cancers Later Detected with Screening Mammography and the Potential Utility of Computer-aided Detection. Radiology 219, 192-202 (2001).
6. Thurfjell, E. L., Lernevall, K. A., and Taube, A. A. S. Benefit of Independent Double Reading in a Population-based Mammography Screening Program. Radiology 191, 241-244 (1994).
7. Anttinen, I., Pamilo, M., Soiva, M., and Roiha, M. Double Reading of Mammography Screening Films: One Radiologist or Two? Radiology 45, 414-421 (1993).
8. Hendee, W. R., Beam, C., and Hendrick, E. Proposition: All Mammograms Should be Double-read. Medical Physics 26, 115-118 (1999).
9. Vyborny, C. J. Can Computers Help Radiologists Read Mammograms? Radiology 191, 315-317 (1994).
10. http://www.confirma.com/products_cadstream.asp.
11. Kerlikowske, K., Carney, P. A., Geller, B., Mandelson, M. T., Taplin, S. H., Malvin, K., Ernster, V., Urban, N., Cutter, G., Rosenberg, R., and Ballard-Barbash, R. Performance of Screening Mammography Among Women With and Without a First-degree Relative with Breast Cancer. Annals of Internal Medicine 133, 855-863 (2000).
12. Davies, E. R. Tradeoffs Between Speed and Accuracy in Two-stage Template Matching. Signal Processing 15, 351-363 (1988).
13. Winsberg, F., Elkin, M., Macy, J., Bordaz, V., and Weymouth, W. Detection of Abnormalities in Mammograms by Means of Optical Scanning and Computer Analysis. Radiology 89, 211-215 (1967).
14. http://www.icadmed.com/.
15. http://www.r2tech.com.
16. Warren Burhenne, L. J., Wood, S. A., D'Orsi, C. J., Feig, S., Kopans, D. B., O'Shaughnessy, K. F., Sickles, E. A., Tabar, L., Vyborny, C. J., and Castellino, R. A. Potential Contribution of Computer-aided Detection to the Sensitivity of Screening Mammography. Radiology 215, 554-562 (2000).
17. Freer, T. W. and Ulissey, M. J. Screening Mammography with Computer-aided Detection: Prospective Study of 12,860 Patients in a Community Breast Center. Radiology 220, 781-786 (2001).
18. Sittek, H., Perlet, C., Helmberger, R., Linsmeler, E., Kessler, M., and Reiser, M. Computer-aided Diagnosis in Routine Mammography. Radiologe 38, 848-852 (1998).
19. Champness, J., Taylor, P., and Given-Wilson, R. Impact of Computer-placed Prompts on Sensitivity and Specificity with Different Groups of Mammographic Film Readers. In: 6th International Workshop on Digital Mammography, pp 276-280. Ed., H.-O. Peitgen. Springer-Verlag, Bremen, Germany (2002).
20. Malich, A., Marx, C., Facius, M., Boehm, T., Fleck, M., and Kaiser, W. A. Tumour Detection Rate of a New Commercially Available Computer-aided Detection System. European Radiolgy 11, 2454-2459 (2001).
21. Vyborny, C. J., Doi, T., O'Shaughnessy, K. F., Romsdahl, H. M., Schneider, A. C., and Stein, A. A. Breast Cancer: Importance of Spiculation in Computer Aided Detection. Radiology 215, 703-707 (2000).
22. Sittek, H., Hermann, K., Perlet, C., Kunzer, I., Kessler, M., and Reiser, M. Computer-aided Diagnosis in Mammography. Radiologe 37, 610-616 (1997).
23. Ciatto, S., Del Turco, M. R., Risso, G., Catarzi, S., Bonardi, R., Viterbo, V., Gnutti, P., Guglielmoni, B., Pinelli, L., Pandiscia, A., Navarra, F., Lauria, A., Palmiero, R., and Indovina, P. L. Comparison of Standard Reading and Computer Aided Detection (CAD) on a National Proficiency Test of Screening Mammography. European Journal of Radiology 45, 135-138 (2003).
24. Campanini, R., Dongiovanni, D., Iampieri, E., Lanconelli, N., Masotti, M., Palermo, G., Riccardi, A., and Roffilli, M. A Novel Featureless Approach to Mass Detection in Digital Mammograms Based on Support Vector Machines. Physics in Medicine and Biology 49, 961-976 (2004).
25. Baker, J. A., Rosen, E. L., Lo, J. Y., Gimenez, E. I., Walsh, R., and Soo, M. S. Computer-aided Detection (CAD) in Screening Mammography: Sensitivity of Commercial CAD Systems for Detecting Architectural Distortion. American Journal of Roentgenology 181, 1083-1088 (2003).
26. Davies, E. R. Algorithms for Inspection: Constraints, Tradeoffs and the Design Process. In: IEE Colloquium on Industrial Inspection, pp 6/1-5. IEE, London, UK (1997).
27. Courtney, P., Thacker, N., A. F., C. Algorithmic Modeling for Performance Evaluation. Machine Vision and Applications 9, 219-228 (1997).
28. Malich, A., Azhari, T., Bohm, T., Fleck, M., and Kaiser, W. A. Reproducibility - An Important Factor Determining the Quality of Computer Aided Detection (CAD) Systems. European Journal of Radiology 36, 170-174 (2000).
29. Zheng, B., Hardesty, L. A., Poller, W. R., Sumkin, J. H., and Golla, S. Mammography with Computer-aided Detection: Reproducibility Assessment - Initial Experience. Radiology 225, 58-62 (2003).
30. Ho, W. T. and Lam, P. W. T. Clinical Performance of Computer-assisted Detection (CAD) System in Detecting Carcinoma in Breasts of Different Densities. Clinical Radiology 58, 133-136 (2003).
31. Helvie, M. A., Hadjiiski, L., Makariou, E., Chan, H.-P., Petrick, N., Sahiner, B., Lo, C-S., B., Freedman, M. T., Adler, D. D., Bailey, J., Blane, C., Hoff, D., Hunt, K., Joynt, L., Klein, K., Paramagul, C., Patterson, S. K., and Roubidoux, M. A. Sensitivity of Noncommercial Computer-aided Detection System for Mammographic Breast Cancer Detection: Pilot Clinical Trial. Radiology 231, 208-214 (2004).
32. Chan, H.-P., Wei, D., Helvie, M. A., Sahiner, B., Adler, D. D., Goodsitt, M. M., and Petrick, N. Computer-aided Classification of Mammographic Masses and Normal Tissue: Linear Discriminant Analysis in Texture Feature Space. Physics in Medicine and Biology 40, 857-876 (1995).
33. Petrick, N., Chan, H.-P., Sahiner, B., and Helvie, M. A. Combined Adaptive Enhancement and Region-growing Segmentation of Breast Masses on Digitized Mammograms. Medical Physics 26, 1642-1654 (1999).
34. Petrick, N., Chan, H.-P., Wei, D., Sahiner, B., Helvie, M. A., and Adler, D. D. Automated Detection of Breast Masses on Mammograms Using Adaptive Contrast Enhancement and Texture Classification. Medical Physics 23, 1685-1696 (1996).
35. Petrick, N., Sahiner, B., Chan, H.-P., Helvie, M. A., Paquerault, S., and Hadjiiski, L. Breast Cancer Detection: Evaluation of a Mass-detection Algorithm for Computer-aided Diagnosis - Experience in 263 Patients. Radiology 218, 217-224 (2002).
36. Pisano, E. D., Cole, E. B., Hemminger, B. M., Yaffe, M. J., Aylward, S. R., Maidment, A. D. A., Johnston, R. E., Williams, M. B., Niklason, L. T., Conant, E. F., Fajardo, L. L., Kopans, D. B., Brown, M. E., and Pizer, S. M. Image Processing Algorithms for Digital Mammography: A Pictorial Essay. Radiographics 20, 1479-1491 (2000).
37. Davies, D. H. and Dance, D. R. Automatic Computer Detection of Clustered Calcifications in Digital Mammograms. Physics in Medicine and Biology 35, 1111-1118 (1990).
38. Chandrasekhar, R. and Attikiouzel, Y. Segmenting the Breast Border and Nipple on Mammograms. Australian Journal of Intelligent Information Processing Systems 6, 24-29 (2000).
39. Chandrasekhar, R. and Attikiouzel, Y. Automatic Breast Border Segmentation by Background Modeling and Subtraction. In: 5th International Workshop on Digital Mammography, pp 560-565. Ed., M. J. Yaffe. Medical Physics Publishing, Toronto, Canada (2000).
40. Chandrasekhar, R. and Attikiouzel, Y. Gross Segmentation of Mammograms Using a Polynomial Model. In: 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 1056-1058. IEEE, Amsterdam, Netherlands (1996).
41. Ojala, T., Näppi, J., and Nevalainen, O. Accurate Segmentation of the Breast Region from Digitized Mammograms. Computerized Medical Imaging and Graphics 25, 47-59 (2001).
42. Ojala, T., Liang, J., Näppi, J., and Nevalainen, O. Interactive Segmentation of the Breast Region from Digitized Mammograms with United Snakes. In. University of Turku, Turku, (1999).
43. McLoughlin, K. J. and Bones, P. J. Location of the Breast-air Boundary for a Digital Mammogram Image. In: Image and Vision Computing '00, Hamilton, New Zealand, (2000).
44. Méndez, A. J., Tahoces, P. G., Lado, M. J., Souto, M., Correa, J. L., and Vidal, J. J. Automatic Detection of Breast Border and Nipple in Digital Mammograms. Computer Methods and Programs in Biomedicine 49, 253-262 (1996).
45. Bick, U., Giger, M. L., Schmidt, R. A., Nishikawa, R. M., Wolverton, D. E., and Doi, K. Automated Segmentation of Digitized Mammograms. Academic Radiology 2, 1-9 (1995).
46. Neyhart, J. T., Ciocco, M. D., Polikar, R., Mandayam, S., and Tseng, M. Dynamic Segmentation of Breast Tissue in Digitized Mammograms. In: 23nd Annual EMBS International Conference, pp 2669-2672. IEEE, Istanbul, Turkey, (2001).
47. Wirth, M. A., Lyon, J., and Nikitenko, D. A Fuzzy Approach to Segmenting the Breast Region in Mammograms. In: North American Fuzzy Information Processing Society Conference, Banff, Canada. In press (2004).
48. Ferrari, R. J., Rangayyan, R. M., Desautels, J. E. L., Borges, R. A., and Frere, A. F. Automatic Identification of the Pectoral Muscle in Mammograms. IEEE Transactions on Medical Imaging 23, 232-245 (2004).
49. Lyon, J. A Density-typed Approach to Enhancement and Quantitative Evaluation of Mammograms. In: Dept. Computing and Information Science. University of Guelph, Guelph, p 194 (2004).
50. Sivaramakrishna, R., Obuchowski, N. A., Chilcote, W. A., Cardenosa, G., and Powell, K. A. Comparing the Peformance of Mammographic Enhancement Algorithms: A Preference Study. American Journal of Roentgenology 175, 45-51 (2000).
51. Puff, D. T., Pisano, E. D., Muller, K. E., Johnston, R. E., Hemminger, B. M., Burbeck, C. A., McLelland, R., and Pizer, S. M. A Method for Determination of Optimal Image Enhancement for the Detection of Mammographic Abnormalities. Journal of Digital Imaging 7, 161-171 (1994).
52. Cromartie, R. and Pizer, S. M. Structure-sensitive Adaptive Contrast Enhancement Methods and Their Evaluation. Image and Vision Computing 11, 460-467 (1993).
53. Zuiderveld, K. Contrast Limited Adaptive Histogram Equalization. In: Graphics Gems IV, pp 474-485. Ed., P. S. Heckbert. AP Professional, Boston, (1994).
54. Braccialarghe, D. and Kaufmann, G. H. Contrast Enhancement of Mammographic Features: A Comparison of Four Methods. Optical Engineering 35, 76-80 (1996).
55. Gordon, R. and Rangayyan, R. M. Feature Enhancement of Film Mammograms Using Fixed and Adaptive Neighborhoods. Applied Optics 23, 560-564 (1984).
56. Tahoces, P. G., Correa, J., Souto, M., Gonzalez, R. C., Gómez, L., and Vidal, J. J. Enhancement of Chest and Breast Radiographs by Automatic Spatial Filtering. IEEE Transactions on Medical Imaging 10, 330-335 (1991).
57. Rehm, K., Seeley, G. W., Dallas, W. J., Ovitt, T. W., and Seeger, J. F. Design and Testing of Artifact-suppressed Adaptive Histogram Equalization: A Contrast-enhancement Technique for Display of Digial Chest Radiographs. Journal of Thoracic Imaging 5, 85-91 (1990).
58. Sosa Urias, J. F. Feature Enhancement of Images Using Maximal Contrast Pixel to Pixel. Applied Optics 30, 4598-4599 (1991).
59. Ji, T.-L., Sundareshan, M. K., and Roehrig, H. Adaptive Image Contrast Enhancement Based on Human Visual Properties. IEEE Transactions on Medical Imaging 13, 573-586 (1994).
60. Pizer, S. M., Ainburn, E. P., Austin, J. D., Cromartie, R., Geselowitz, A., Greer, T., Haar-Romeny, B., Zimmerman, J. B., and Zuiderveld, K. Adaptive Histogram Equalization and Its Variations. Computer Vision, Graphics and Image Processing 39, 355-368 (1987).
61. Morrow, W. M., Paranjape, R. B., Rangayyan, R. M., and Desautels, J. E. L. Region-based Contrast Enhancement of Mammograms. IEEE Transactions on Medical Imaging 11, 392-406 (1992).
62. Laine, A. F., Schuler, S., Fan, J., and Huda, W. Mammographic Feature Enhancement by Multiscale Analysis. IEEE Transactions on Medical Imaging 13, 725-740 (1994).
63. Eskicioglu, A. M. and Fisher, P. S. Image Quality Measures and Their Performance. IEEE Transactions on Communications 43, 2959-2965 (1995).
64. Li, H., Liu, K. J. R., Lo, C.-S., B. Fractal Modeling and Segmentation for the Enhancement of Microcalcifications in Digital Mammograms. IEEE Transactions on Medical Imaging 16, 785-798 (1997).
65. Rueckert, D., Sonoda, L. I., Hayes, C., Hill, D., Leach, M. O., and Hawkes, D. J. Nonrigid Registration Using Free-form Deformations: Application to Breast MR Images. IEEE Transactions on Medical Imaging 18, 712-721 (1999).
66. Denton, E. R. E., Sonoda, L. I., Rueckert, D., Rankin, S. C., Hayes, C., Leach, M. O., Hill, D. L. G., and Hawkes, D. J. Comparison and Evaluation of Rigid, Affine, and Nonrigid Registration of Breast MR Images. Journal of Computer Assisted Tomography 23, 800-805 (1999).
67. Brückner, T., Lucht, R., and Brix, G. Comparison of Rigid and Elastic Matching of Dynamic Magnetic Resonance Mammographic Images by Mutual Information. Medical Physics 27, 2456-2461 (2000).
68. Rueckert, D., Sonoda, L. I., Denton, E., Rankin, S., Hayes, C., Leach, M. O., Hill, D., and Hawkes, D. J. Evaluation of Non-rigid Registration Using Free-form Deformations for Breast MR Images. In: Medical Image Analysis and Understanding, Oxford, UK, (1999).
69. Rueckert, D., Sonoda, L. I., Denton, E., Rankin, S., Hayes, C., Leach, M. O., Hill, D., and Hawkes, D. J. Comparison and Evaluation of Rigid and Non-rigid Registration of Breast MR Images. In: SPIE Medical Imaging: Image Processing, pp 78-88, Ed., K. M. Hanson. SPIE Press, San Diego, CA (1999).
70. Hayton, P. M., Brady, M., Smith, S. M., and Moore, N. A Non-rigid Registration Algorithm for Dynamic Breast MR Images. Artificial Intelligence 114, 125-156 (1999).
71. Wirth, M. A. Characterizing Spatial Deformation in Breast Images. In. University of Guelph, Guelph, (2002).
72. Schnabel, J. A., Tanner, C., Castellano Smith, A. D., Degenhard, A., Hayes, C., Leach, M. O., Hose, R., Hill, D. L. G., and Hawkes, D. J. Validation of Non-rigid Registration of Contrast-enhanced MR Mammography Using Finite Element Methods. In: Workshop on Bildverarbeitung für die Medizin, Informatik Aktuell, pp 143-146. Springer Verlag (2002).
73. Bottema, K. J. and Slavotinek, J. P. Computer Aided Screening Mammography. In: New Approaches in Mecical Image Analysis, pp 177-190, Eds., B. Pham, M. Braun, and A. P. Maeder. SPIE, Ballarat, Australia (1999).
74. Shile, P. E. and Guingrich, J. A. Detecting "Missed" Breast Cancers: A Comparison of CAD Systems. Applied Radiology (supplement) 31, 1-3 (2002).
75. Kupinski, M. and Lobregt, S. A Discrete Dynamic Contour Model. IEEE Transactions on Medical Imaging 17, 510-517 (1998).
76. te Brake, G. M., Stoutjesdijk, M. J., and Karssemeijer, N. A Discrete Dynamic Contour Model for Mass Segmentation in Digital Mammograms. In: SPIE Medical Imaging: Image Processing, pp 911-919, Ed., K. M. Hanson. SPIE Press, San Diego, CA, (1999).
77. Timp, S., Karssemeijer, N., and Hendriks, J. Comparison of Three Different Mass Segmentation Methods. In: 6th International Workshop on Digital Mammography, pp. 218-222, Ed., H.-O. Peitgen. Springer-Verlag, Bremen, Germany (2002).
78. Kallergi, M., Camey, G. M., and Gaviria, J. Evaluating the Performance of Detection Algorithms in Digital Mammography. Medical Physics 26, 267-275 (1999).
79. Ashby, A. E., Hernandez, J. M., Logan, C. M., Mascio, L. N., Frankel, S., and Kegelmeyer, W. P., Jr. USCF/LLNL High Resolution Digital Mammogram Library. In: Engineering in Medicine and Biology Conference, pp 539-540. IEEE (1995).
80. Collins, D., Zijdenbos, A., Kollokian, V., Sled, J., Kabani, N., Holmes, C., and Evans, A. Design and Construction of a Realistic Digital Brain Phantom. IEEE Transactions on Medical Imaging 17, 463-468 (1998).
81. Warfield, S. K., Zou, K. H., and Wells, W. M. I. Simultaneous Truth and Performance Level Estimation (STAPLE): An Algorithm for the Validation of Image Segmentation. IEEE Transactions on Medical Imaging 23, 903-921 (2004).
82. Vieira, L. and Undrill, P. Use of a Breast Model to Test the Performance of Image Registration Methods. In: Medical Image Analysis and Understanding. Oxford, UK (1999).
83. Krishnan, S., Chenevert, T. L., Helvie, M. A., and Londy, F. L. Linear Motion Correction in Three Dimensions Applied to Dynamic Gadolinium Enhanced Breast Imaging. Medical Physics 26, 707-714 (1999).
84. Chalana, V. and Kim, Y. A Methodology for Evaluation of Boundary Detection Algorithms on Medical Images. IEEE Transactions on Medical Imaging 16, 642-652 (1997).
85. Petroski, H. To Engineer is Human. St Martin's Press, New York (1982).
86. Wirth, M. A. and Stapinski, A. Segmentation of the Breast Region in Mammograms Using Snakes. In: 1st Canadian Conference on Computer and Robot Vision, pp 385-392. IEEE Computer Society, London, ON, Canada (2004).
87. Nishikawa, R. H. Mammographic Databases. Breast Disease 10, 137-150 (1998).
88. Suckling, J., Parker, J., Dance, D. R., Astley, S., Hutt, I., Boggis, C. R. M., Ricketts, I., Stamatakis, E., Cerneaz, N., Kok, S.-L., Taylor, P., Betal, D., and Savage, J. The Mammographic Image Analysis Society Digital Mammogram Database. In: International Workshop on Digital Mammography, pp 375-378, Eds., A. G. Gale, S. M. Astley, D. R. Dance, A. Y. Cairns. Elsevier, York, England (1994).
89. Heath, M., Bowyer, K., Kopans, D., Moore, R., and Kegelmeyer, P. J. The Digital Database for Screening Mammography. In: 5th International Workshop on Digital Mammography, pp 212-218. Medical Physics Publishing, Toronto, Canada (2000).
90. Kimme-Smith, C. Clinical Considerations for a Mammography Database. In: Biomedical Image Processing and Biomedical Visualization, pp 546-547, Eds., R. S. Acharya, D. B. Goldgof. SPIE (1993).
91. Mammographic Image Analysis Society, M. Digital Mammogram Database. Published Electronically (1994). http://s20c.smb.man.ac.uk/services/MIAS/ MIAScom.html
92. Alto, H., Rangayyan, R. M., and Desautels, J. E. L. An Indexed Atlas of Digital Mammograms. In: 6th International Workshop on Digital Mammography, pp 309-311, Ed., H.-O. Peitgen. Springer-Verlag, Bremen, Germany (2002).
93. Arimura, H., Katsuragawa, S., Ishida, T., Oda, N., Nakata, H., and Doi, K. Performance Evaluation of an Advanced Method for Automated Identification of View Positions of Chest Radiographs by Use of a Large Database. In: SPIE Medical Imaging: Image Processing, pp 308-315. SPIE, San Diego, USA (2002).
94. Bamford, P., Jackway, P. T., and Lovell, B. Progress in the Robust Automated Segmentation of Real Cell Images. In: New Approaches in Mecical Image Analysis, pp 34-56, Eds., B. Pham, M. Braun, A. P. Maeder. SPIE, Ballarat, Australia (1999).
95. Schiabel, H., Nunes, F. L. S., Escarpinati, M. C., and Benatti, R. H. Performance of a Processing Scheme for Clustered Microcalcifications Detection with Different Images Database. In: Engineering in Medicine and Biology Society Conference, pp 1199-1202. IEEE, Chicago, IL USA (2000).
96. Yarlagadda, N. C., Bowyer, K. W., and Li, R. Baseline Comparison of Microcalcification Detection Algorithms. In: Proceedings 5th International Workshop on Digital Mammography, pp 414-420, Ed., M. J. Yaffe. Medical Physics Publishing, Toronto, Canada (2000).
97. Karssemeijer, N. Adaptive Noise Equalization and Recognition of Microcalcification Clusters in Mammograms. International Journal of Pattern Recognition and Artificial Intelligence 7, 1403-1416 (1993).
98. Wirth, M. A., Wang, R., and Stapinski, A. An Algorithm for Extracting the Skin Surface in MR Breast Images. In: 11th Annual Meeting International Society for Magnetic Resonance in Medicine, p 947. International Society for Magnetic Resonance in Medicine, Toronto, Canada (2003).
99. Gerig, G., Jomier, M., and Chakos, M. Valmet: A New Validation Tool for Assessing and Improving 3D Object Segmentation. In: Medical Image Computing and Computer-Assisted Intervention, pp 516-528. Springer-Verlag, Berlin, (2001).
100. Wang, Z., Bovik, A. C., and Lu, L. Why is Image Quality Assessment So Difficult? In: IEEE International Conference on Acoustics, Speech, and Signal Processing, pp 3313-3316. IEEE (2002).
101. Mezaris, V., Kompatsiaris, I., and Strintzis, M. G. Still Image Objective Segmentation Evaluation Using Ground Truth. In: COST Workshop on Information and Knowledge Management for Integrated Media Communication, pp 9-14. Prague, Czech Republic (2003).
102. West, J. B., Fitzpatrick, J. M., Wang, M. Y., Dawant, B. M., Maurer, J., C. R., Kessler, R. M., Maciunas, R. J., Barillot, C., Lemoine, D., Collignon, A. M., Maes, F., Suetens, P., Vandermeulen, D., van den Eisen, P. A., Hemler, P. F., Napel, S., Sumanaweera, T. S., Harkness, B. A., Hill, D. L., Studholme, C., Malandain, G., Pennec, X., Noz, M. E., Maguire, J., G. Q., Pollack, M., Pelizzari, C. A., Robb, R. A., and Hanson, D. P., R. P., W. Comparison and Evaluation of Retrospective Intermodality Image Registration Techniques. In: SPIE Medical Imaging: Image Processing, pp 332-347. Eds., M. H. Loew, and K. H. Hanson. SPIE, San Diego, CA, USA (1996).
103. Holzmann, G. J. The Logic of Bugs. In: Foundations of Software Engineering, pp 81-87. ACM, Charleston, SC, USA (2002).
104. Munson, J. C. Software Faults, Software Failures and Software Reliability Modeling. Information and Software Technology 38, 687-699 (1996).
105. Woodward, M. R. Insights into Software Testing. Software Focus 2, 93-103 (2001).