A performance evaluation protocol for symbol spotting systems in terms of recognition and location indices

International Journal on Document Analysis and Recognition

Volumn 12, Issue 2, 2009, Pages 83-96

  Rusiñol, Marçal ^a   Lladós, Josep ^a  

^a UNIVERSITAT AUTÒNOMA DE BARCELONA   (Spain)

Author keywords

Graphics recognition; Performance evaluation; Symbol spotting

Indexed keywords

DOCUMENT IMAGES; GRAPHICS RECOGNITION; LOCATION ACCURACY; PERFORMANCE EVALUATION; RECOGNITION ABILITIES; REGIONS OF INTEREST; SHAPE DESCRIPTORS; SINGLE-STEP; SYMBOL SPOTTING;

DATABASE SYSTEMS;

EID: 67650439217     PISSN: 14332833     EISSN: 14332825     Source Type: Journal    
DOI: 10.1007/s10032-009-0083-y     Document Type: Article

References (41)

1. Aksoy, S., Ye, M., Schauf, M., Song, M., Wang, Y., Haralick, R.: Algorithm performance contest. In: Proceedings of the Fifteenth International Conference on Pattern Recognition, ICPR00, pp. 870-876 (2000). doi: 10.1109/ICPR.2000.903054
2. Antonacopoulos, A., Bridson, D.: Performance analysis framework for layout analysis methods. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, ICDAR07, pp. 1258-1262 (2007). doi: 10.1109/ICDAR.2007.4377117
3. Antonacopoulos, A., Gatos, B., Bridson, D.: ICDAR 2005 page segmentation competition. In: Proceedings of the Eighth International Conference on Document Analysis and Recognition, ICDAR05, pp. 75-79 (2005). doi: 10.1109/ICDAR.2005.184
4. Antonacopoulos, A., Gatos, B., Bridson, D.: ICDAR 2007 page segmentation competition. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, ICDAR07, pp. 1279-1283 (2007) doi: 10.1109/ICDAR.2007.203
5. Antonacopoulos, A., Gatos, B., Karatzas, D.: ICDAR 2003 page segmentation competition. In: Proceedings of the Seventh International Conference on Document Analysis and Recognition, ICDAR03, pp. 688-692 (2003). doi: 10.1109/ICDAR.2003.1227750
6. Antonacopoulos, A., Karatzas, D., Bridson, D.: Ground truth for layout analysis performance evaluation. In: Document Analysis Systems, DAS06. Lecture Notes on Computer Science, vol. 3872, pp. 302-311. Springer, Berlin (2006). doi: 10.1007/11669487_27
7. Barber C., Dobkin D., Huhdanpaa H.: The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software 22(4), 469-483 (1996). doi: 10.1145/235815.235821
8. Buckland M., Gey F.: The relationship between recall and precision. J. Am. Soc. Inform. Sci. 45(1), 12-19 (1994)
9. Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 233-240 (2006). doi: 10.1145/1143844.1143874
10. Delalandre, M., Pridmore, T., Valveny, E., Locteau, H., Trupin, E.: Building synthetic graphical documents for performance evaluation. In: Graphics Recognition. Recent Advances and New Opportunities. Lecture Notes on Computer Science, vol. 5046, pp. 288-298. Springer, Berlin (2008). doi: 10.1007/978-3-540-88188-9_27
11. Fawcett T.: An introduction to ROC analysis. Pattern Recognit. Lett. 27(8), 861-874 (2006). doi: 10.1016/j.patrec.2005.10.010
12. Holz, F., Witschel, H., Heinrich, G., Heyer, G., Teresniak, S.: An evaluation framework for semantic search in p2p networks. In: Proceedings of the Seventh International Workshop on Innovative Internet Community Systems, I2CS07 (2007)
13. Hripcsak G., Rothschild A.: Agreement, the F-measure, and reliability in information retrieval. J. Am. Med. Inform. Assoc 12(3), 296-298 (2005). doi: 10.1016/j.jamia.2005.01.008
14. Hu M.: Visual pattern recognition by moment invariants. IRE Trans. Inform. Theory 8, 179-187 (1962)
15. Huijsmans D., Sebe N.: How to complete performance graphs in content-based image retrieval: Add generality and normalize scope. IEEE Trans. Pattern Anal. Mach. Intell. 27(2), 245-251 (2005). doi: 10.1109/ TPAMI.2005.30
16. Kang, B., Kim, H., Lee, S.: Performance analysis of semantic indexing in text retrieval. In: Computational Linguistics and Intelligent Text Processing. Lecture Notes on Computer Science, vol. 2945, pp. 433-436. Springer, Berlin (2004).doi: 10.1007/b95558
17. Kauppinen H., Seppänen T., Pietik̈inen M.: An experimental comparison of autoregressive and fourier-based descriptors in 2d shape classification. IEEE Trans. Pattern Anal. Mach. Intell. 17, 201-207 (1995). doi: 10.1109/34.368168
18. Lambert, G., Gao, H.: Discrimination properties of invariants using the line moments of vectorized contours. In: Proceedings of the 13th International Conference on Pattern Recognition, ICPR96, pp. 735-739 (1996). doi: 10.1109/ICPR.1996.546920
19. Liu W., Dori D.: A protocol for performance evaluation of line detection algorithms. Mach. Vis. Appl. 9(5), 240-250 (1997). doi: 10.1007/ s001380050045
20. Liu W., Dori D.: Incremental arc segmentation algorithm and its evaluation. IEEE Trans. Pattern Anal.Mach. Intell. 20(4), 424-431 (1998). doi: 10.1109/34.677280
21. Lopresti, D., Nagy, G.: Issues in ground-truthing graphic documents. In: Graphics Recognition Algorithms and Applications. Lecture Notes on Computer Science, vol. 2390, pp. 46-66. Springer, Berlin (2001). doi: 10.1007/3-540-45868-9
22. Lu, C., Shukla, M., Subramanya, S., Wu, Y.: Performance evaluation of desktop search engines. In: Proceedings of the IEEE International Conference on Information Reuse and Integration, IRI07, pp. 110-115 (2007). doi: 10.1109/IRI.2007.4296606
23. Lucas, S.: ICDAR 2005 text locating competition results. In: Proceedings of the Eighth International Conference on Document Analysis and Recognition, ICDAR05, pp. 80-84 (2005). doi: 10.1109/ICDAR.2005.231
24. Lucas S., Panaretos A., Sosa L., Tang A., Wong S., Young R., Ashida K., Nagai H., Okamoto M., Yamamoto H., Miyao H., Zhu J., Ou W., Wolf C., Jolion J., Todoran L., Worring M., Lin X.: ICDAR 2003 robust reading competitions: Entries, results, and future directions. Int. J. Doc. Anal. Recognit. 7(2), 105-122 (2005). doi: 10.1007/s10032-004-0134-3
25. Makhoul, J., Kubala, F., Schwartz, R., Weischedel, R.: Performance measures for information extraction. In: Proceedings of DARPA Broadcast News Workshop (1999)
26. Marcus, J.: A novel algorithm for HMM word spotting performance evaluation and error analysis. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP92, pp. 89-92 (1992). doi: 10.1109/ICASSP.1992.226113
27. Müller H., Müller W., Squire D., Marchand-Maillet S., Pun T.: Performance evaluation in content-based image retrieval: Overview and proposals. Pattern Recognit. Lett. 22(5), 593-601 (2001). doi: 10.1016/ S0167-8655(00)00118-5
28. Neumann, T., Bender, M., Michel, S., Weikum, G.: A reproducible benchmark for P2P retrieval. In: Proceedings of the First International Workshop on Performance and Evaluation of Data Management Systems, ExpDB06, pp. 1-8 (2006)
29. Phillips I., Chhabra A.: Empirical performance evaluation of graphics recognition systems. IEEE Trans. Pattern Anal. Mach. Intell. 21(9), 849-870 (1999). doi: 10.1109/34.790427
30. Rath, T., Manmatha, R.: Features for word spotting in historical manuscripts. In: Proceedings of the Seventh International Conf. on Document Analysis and Recognition, ICDAR03, pp. 218-222 (2003). doi: 10.1109/ICDAR.2003.1227662
31. van Rijsbergen C.: Information Retrieval. Butterworth-Heinemann Newton, MA, USA (1979)
32. Rusiñol, M., Lladós, J.: Word and symbol spotting using spatial organization of local descriptors. In: The Eighth IAPR International Workshop on Document Analysis Systems, DAS08, pp. 489-496 (2008). doi: 10.1109/DAS.2008.24
33. Smeaton, A., Over, P., Kraaij, W.: Evaluation campaigns and TRECVid. In: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 321-330 (2006). doi: 10.1145/1178677.1178722
34. Song J., Su F., Tai C., Cai S.: An object-oriented progressive-simplification-based vectorization system for engineering drawings: Model, algorithm, and performance. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1048-1060 (2002). doi: 10.1109/TPAMI.2002.1023802
35. Stoyan D., Stoyan H.: Fractals, Random Shapes and Point Fields (Methods of Geometrical Statistics). John Wiley & Sons, Chichester (1994)
36. Tabbone, S., Zuwala, D.: An indexing method for graphical documents. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, ICDAR07, pp. 789-793 (2007). doi: 10.1109/ ICDAR.2007.4377023
37. Tombre, K., Lamiroy, B.: Graphics recognition - from re-engineering to retrieval. In: Proceedings of the Seventh International Conference on Document Analysis and Recognition, ICDAR03, pp. 148-155 (2003)
38. Valveny, E., Dosch, P.A Symbol recognition contest: a synthesis. In: Graphics Recognition, Recent Advances and Perspectives. Lecture Notes on Computer Science, vol. 3088, pp. 368-385. Springer, Berlin (2004). doi: 10.1007/b99011
39. Valveny, E., Dosch, P.: Report on the second symbol recognition contest. In: Graphics Recognition. Ten Years Review and Future Perspectives. Lecture Notes on Computer Science, vol. 3926, pp. 381-397. Springer, Berlin (2006). doi: 10.1007/11767978
40. Valveny E., Dosch P., Winstanley A., Zhou Y., Yang S., Yan L., Liu W., Elliman D., Delalandre M., Trupin E., Adam S., Ogier J.: A general framework for the evaluation of symbol recognition methods. Int. J. Doc. Anal. Recognit. 9(1), 59-74 (2007). doi: 10.1007/s10032-006-0033-x
41. Wolf C., Jolion J.: Object count/area graphs for the evaluation of object detection and segmentation algorithms. Int. J. Doc. Anal. Recognit. 8(4), 280-296 (2006). doi: 10.1007/s10032-006-0014-0