Image search in patents: A review

International Journal on Document Analysis and Recognition

Volumn 16, Issue 4, 2013, Pages 309-329

  Bhatti, Naeem ^a   Hanbury, Allan ^b  

^a QUAID I AZAM UNIVERSITY   (Pakistan)

^b VIENNA UNIVERSITY OF TECHNOLOGY   (Austria)

Author keywords

Data charts; Diagrams; Patent images; Technical drawings

Indexed keywords

CIRCUIT DIAGRAMS; DATA CHARTS; DATA FLOW DIAGRAMS; DIAGRAMS; GRAPHICAL DESCRIPTION; PATENT IMAGES; SYMBOL RECOGNITION; TECHNICAL DRAWING;

GRAPHIC METHODS; IMAGE RETRIEVAL; PATTERN RECOGNITION;

PATENTS AND INVENTIONS;

EID: 84888313416     PISSN: 14332833     EISSN: 14332825     Source Type: Journal    
DOI: 10.1007/s10032-012-0197-5     Document Type: Review

References (101)

1. Adams, S.: Electronic non-text material in patent applications-some questions for patent offices, applicants and searchers. World Pat. Inf. 27(2), 99-103 (2005).
2. Antonacopoulos, A., Kennedy, D.: Information extraction from complex circular charts. Document Analysis and Recognition, International Conference on 0, 784-787 (2001).
3. Atanasova, G., Hristova, P.: Flow chart interpreter: an environment for software animation representation. In: CompSysTech '03: Proceedings of the 4th International Conference Conference on Computer Systems and Technologies, pp. 453-458 (2003).
4. Bakergem, D. V.: Image collections in the design studio. In: McCullough, M., Mitchell, W. J., Purcell, P. (eds.) The electronic design studio, pp. 261-271. MIT Press, Cambridge, MA, USA. http://dl. acm. org/citation. cfm?id=109149. 109168 (1990).
5. Barta, A., Vajk, I.: Document image analysis by probabilistic network and circuit diagram extraction. Inform. Int. J. Comput. Inform. 29, 291-301 (2005).
6. Berchtold, S., Kriegel, H. P.: S3: similarity search in cad database systems. In: SIGMOD (1997).
7. Blostein, D., Lank, E., Zanibbi, R.: Treatment of diagrams in document image analysis. In: Proceedings of the First International Conference on Theory and Application of Diagrams, pp. 330-344. Springer, London, UK (2000).
8. Browuer, W., Kataria, S., Das, S., Mitra, P., Giles, C. L.: Segregating and extracting overlapping data points in two-dimensional plots. In: JCDL, pp. 276-279 (2008).
9. Bruza, P. D., van der Weide, T.: The semantics of data flow diagrams. In: In Proceedings of the International Conference on Management of Data, pp. 66-78. McGraw-Hill Publishing Company (1993).
10. Butler, G., Grogono, P., Shinghal, R., Tjandra, I.: Analyzing the logical structure of data flow diagrams in software documents. In: ICDAR, vol. 2, pp. 575-578 (1995).
11. Chhabra, A. K.: Graphic symbol recognition: An overview. In: GREC '97: Selected Papers from the Second International Workshop on Graphics Recognition, Algorithms and Systems (1998).
12. Chou, P. A.: Recognition of equations using a 2-D stochastic context-free grammar. In: SPIE, vol. 1199, pp. 852-863 (1989).
13. Clayton, M., W. H.: Enhancing the sketchbook. In: Proceedings of the Association for Computer Aided Design in Architecture (ACADIA 91) (1991).
14. Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 603-619 (2002).
15. Cordella, L. P., Vento, M.: Symbol and shape recognition. In: GREC '99: Selected Papers from the Third International Workshop on Graphics Recognition, Recent, Advances, pp. 167-182 (2000).
16. Csurka, G., Renders, J. M., Jacquet, G.: XRCE's participation at patent image classification and image-based patent retrieval tasks of the Clef-IP 2011. In: Petras, V., Forner, P., Clough, P. D. (eds.) CLEF (Notebook Papers/Labs/Workshop) (2011).
17. Das, M., Manmatha, R., Riseman, E. M.: Indexing flower patent images using domain knowledge. IEEE Intell. Syst. 14(5), 24-33 (1999).
18. Datta, R., Joshi, D., Li, J., Wang, J. Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), 1-60 (2008).
19. Delalandre, M., Valveny, E., Llados, J.: Performance evaluation of symbol recognition and spotting systems: An overview. In: Document Analysis Systems, 2008. DAS '08. The Eighth IAPR International Workshop on, pp. 497-505 (2008).
20. Dodhiawala, R.: Blackboard Architectures and Applications. Academic Press Inc., Orlando (1989).
21. Eakins, J. P.: Trademark image retrieval. In: Lew, M. (ed.) Chapter 13 in principles of visual information retrieval, pp. 319-350. Springer, Berlin (2001).
22. Escalera, S., Fornés, A., Pujol, O., Lladós, J., Radeva, P.: Circular blurred shape model for multiclass symbol recognition. IEEE Trans. Syst. Man Cybern. Part B 41(2), 497-506 (2011).
23. Escalera, S., Fornés, A., Pujol, O., Radeva, P., Sánchez, G., Lladós, J.: Blurred shape model for binary and grey-level symbol recognition. Pattern Recognit. Lett. 30(15), 1424-1433 (2009).
24. Fathulla, K., Basden, A.: What is a diagram? Information Visualisation. International Conference on pp. 951-956 (2007).
25. Feng, G., Viard-Gaudin, C., Sun, Z.: On-line hand-drawn electric circuit diagram recognition using 2d dynamic programming. Pattern Recognit. 42(12), 3215-3223 (2009).
26. Fonseca, M. J., Ferreira, A., Jorge, J. A.: Content-based retrieval of technical drawings. In: Special Issue of International Journal of Computer Applications in Technology (IJCAT) (2004).
27. Fonseca, M. J., Ferreira, A., Jorge, J. A.: Sketch-based retrieval of complex drawings using hierarchical topology and geometry. Comput. Aided Des. 41(12), 1067-1081 (2009).
28. Fonseca, M. J., Jorge, J. A.: Towards content-based retrieval of technical drawings through high-dimensional indexing. Comput. Graph. 27(1), 61-69 (2003).
29. Forneands, A., Escalera, S., Llado ands, J., Valveny, E.: Symbol classification using dynamic aligned shape descriptor. In: ICPR, pp. 1957-1960 (2010).
30. Fornés, A., Lladós, J., Sánchez, G., Karatzas, D.: Rotation invariant hand-drawn symbol recognition based on a dynamic time warping model. IJDAR 13(3), 229-241 (2010).
31. Futrelle, R. P., Shao, M., Cieslik, C., Grimes, A. E.: Extraction, layout analysis and classification of diagrams in pdf documents. In: ICDAR, pp. 1007-1014 (2003).
32. Hanbury, A., Bhatti, N., Lupu, M., Mörzinger, R.: Patent image retrieval: a survey. In: Proceedings of the Patent Inforamtion Retrieval Workshop (PaIR) (2011).
33. Hoi, C. H., Chan, C. H., Huang, K., Lyu, M. R., King, I.: Biased support vector machine for relevance feedback in image retrieval. In: Proceedings International Joint Conference on Neural Networks (IJCNN), pp. 3189-3194 (2004).
34. Hou, S., Ramani, K.: Classifier combination for sketch-based 3d part retrieval. Comput. Graph. 31(4), 598-609 (2007).
35. Hou, S., Ramani, K.: Structure-oriented contour representation and matching for engineering shapes. Comput. Aided Des. 40(1), 94-108 (2008).
36. Huang, W., Tan, C. L., Leow, W. K.: Model-based chart image recognition. In: GREC, pp. 87-99 (2003).
37. Huet, B., Guarascio, G., Kern, N. J., Mérialdo, B.: Relational skeletons for retrieval in patent drawings. In: ICIP(2), pp. 737-740 (2001).
38. Jain, A., Vailaya, A.: Shape-based retrieval: A case study with trademark image databases. Pattern Recognit. 31, 1369-1390 (1998).
39. Joachims, T.: Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms. Kluwer Academic Publishers, Norwell (2002).
40. Joseph, S. H., Pridmore, T. P.: Knowledge-directed interpretation of mechanical engineering drawings. IEEE Trans. Pattern Anal. Mach. Intell. 14(9), 928-940 (1992).
41. Kataria, S., Browuer, W., Mitra, P., Giles, C. L.: Automatic extraction of data points and text blocks from 2-dimensional plots in digital documents. In: AAAI, pp. 1169-1174 (2008).
42. Kim, H. K., Kim, J. D., Sim, D. G., Oh, D. I.: A modified zernike moment shape descriptor invariant to translation, rotation and scale for similarity-based image retrieval. In: Multimedia and Expo, IEEE International Conference on, vol. 1, pp. 307-310 (2000).
43. Koch, S., Bosch, H., Giereth, M., Ertl, T.: Iterative integration of visual insights during scalable patent search and analysis. IEEE Transactions on Visualization and Computer Graphics 99(PrePrints) (2010). doi: http://doi. ieeecomputersociety. org/10. 1109/TVCG. 2010. 85.
44. Kopec, G. E., Chou, P. A.: Document image decoding using markov source models. IEEE Trans. Pattern Anal. Mach. Intell. 16(6), 602-617 (1994).
45. Kruskal, J. B., Liberman, M.: The symmetric time-warping problem: From continuous to discrete. In: Sankoff, D., Kruskal, J. B. (eds.) Time Warps, String Edits, and Macromolecules-The Theory and Practice of Sequence Comparison, chap. 4 (1999).
46. Lank, E. H.: A retargetable framework for interactive diagram recognition. In: ICDAR, p. 185. IEEE Computer Society, Washington, DC, USA (2003).
47. Latecki, L. J., Lakämper, R.: Convexity rule for shape decomposition based on discrete contour evolution. Comput. Vis. Image Underst. 73(3), 441-454 (1999).
48. van Leuken, R. H., Demirci, M. F., Hodge, V. J., Austin, J., Veltkamp, R. C.: Layout indexing of trademark images. In: Proceedings of the 6th ACM international conference on Image and video retrieval, CIVR '07, pp. 525-532 (2007).
49. Leung, W. H., Chen, T.: User-independent retrieval of free-form hand-drawn sketches. In: Proceedings of the IEEE ICASSP02, pp. 2029-2032. IEEE Press (2002).
50. Liang, S., Sun, Z.: Sketch retrieval and relevance feedback with biased SVM classification. Pattern Recogn. Lett. 29(12), 1733-1741 (2008).
51. Liang, S., Sun, Z., Li, B.: Effective sketch retrieval based on its contents. In: ICMLC, pp. 5266-5272 (2005).
52. Liang, Z., Nan, J., Lin, Q. Q.: Relevant feedback in content-based engineering drawing retrieval. In: CSSE, pp. 544-547 (2008).
53. List, J.: How drawings could enhance retrieval in mechanical and device patent searching. World Pat. Inf. 29(3), 210-218 (2007).
54. Liu, R., Baba, T., Masumoto, D.: Attributed graph matching based engineering drawings retrieval. In: Document Analysis Systems, pp. 378-388 (2004).
55. Liu, R., Wang, Y., Baba, T., Masumoto, D.: Shape detection from line drawings with local neighborhood structure. Pattern Recognit. 43(5), 1907-1916 (2010).
56. Lladós, J., Valveny, E., Sánchez, G., Martí, E.: Symbol recognition: Current advances and perspectives. In: Selected Papers from the Fourth International Workshop on Graphics Recognition Algorithms and Applications, GREC '01, pp. 104-127. Springer-Verlag, London, UK (2002).
57. Love, D., Barton, J.: Aspects of design retrieval performance using automatic GT coding of 2D engineering drawings. In: 4th International Conference on Integrated Design and Manufacture in, Mechanical Engineering (2004).
58. Lu, T., Tai, C. L., Yang, H., Cai, S.: A novel knowledge-based system for interpreting complex engineering drawings: Theory, representation, and implementation. IEEE Trans. Pattern Anal. Mach. Intell. 31(8), 1444-1457 (2009).
59. Lu, T., Yang, H., Yang, R., Cai, S.: Automatic analysis and integration of architectural drawings. Int. J. Doc. Anal. Recognit. 9(1), 31-47 (2007).
60. Lu, X., Kataria, S., Brouwer, W. J., Wang, J. Z., Mitra, P., Giles, C. L.: Automated analysis of images in documents for intelligent document search. Int. J. Doc. Anal. Recognit. (IJDAR) 12(2), 65-81 (2009).
61. Lu, X., Mitra, P., Wang, J. Z., Giles, C. L.: Automatic categorization of figures in scientific documents. In: JCDL, pp. 129-138. ACM (2006).
62. Mahmoudi, F., Shanbehzadeh, J., Eftekhari-Moghadam, A. M., Soltanian-Zadeh, H.: Image retrieval based on shape similarity by edge orientation autocorrelogram. Pattern Recognit. 36, 1725-1736 (2003).
63. Milner, R.: A Calculus of Communicating Systems. Springer New York Inc., Secaucus (1982).
64. Milner, R.: Communication and Concurrency. Prentice-Hall Inc., Upper Saddle River (1989).
65. Milner, R.: The polyadic pi-calculus: a tutorial. Technical report, Logic and Algebra of Specification (1991).
66. Osada, R., Funkhouser, T., Chazelle, B., Dobkin, D.: Shape distributions. ACM Trans. Graph. 21(4), 807-832 (2002).
67. Osborn, J. R.: Theory pictures as trails: Diagrams and the navigation of theoretical narratives. Cogn. Sci. Online 3(2), 14-44 (2005).
68. Park, J. H., Um, B. S.: A new approach to similarity retrieval of 2-D graphic objects based on dominant shapes. Pattern Recognit. Lett. 20(6), 591-616 (1999).
69. Parker, C., Chen, T.: Hierarchical matching for retrieval of hand-drawn sketches. In: ICME, pp. 29-32. IEEE Computer Society, Washington, DC, USA (2003).
70. Pasternak, B.: Processing imprecise and structural distorted line drawings by an adaptable drawing interpretation kernel. In: Proceedings of the IAPR Workshop on Document Analysis Systems, pp. 349-363 (1994).
71. Perronnin, F., Dance, C.: Fisher kernels on visual vocabularies for image categorization. In: CVPR, pp. 1-8 (2007).
72. Plimmer, B., Freeman, I.: A toolkit approach to sketched diagram recognition. In: Proceedings of the 21st British CHI Group Annual Conference on HCI (BCS-HCI), pp. 205-213. British Computer Society, Swinton, UK (2007).
73. Pu, J., Ramani, K.: On visual similarity based 2d drawing retrieval. Comput. Aided Des. 38(3), 249-259 (2006).
74. Rubine, D.: Specifying gestures by example. SIGGRAPH Comput. Graph. 25(4), 329-337 (1991).
75. Schietse, J., Eakins, J. P., Veltkamp, R. C.: Practice and challenges in trademark image retrieval. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, CIVR '07, pp. 518-524 (2007).
76. Seul, M., O'Gorman, L., Sammon, M. J.: Practical algorithms for image analysis: Description, examples, and code. Cambridge University Press, New York (2000).
77. Shukla, S., Samal, A.: Recognition and quality assessment of data charts in mixed-mode documents. Int. J. Doc. Anal. Recognit. 11(3), 111-126 (2008).
78. Sidiropoulos, P., Vrochidis, S., Kompatsiaris, I.: Adaptive hierarchical density histogram for complex binary image retrieval. In: Content-Based Multimedia Indexing (CBMI), 2010 International Workshop on, pp. 16. IEEE (2010).
79. Sifakis, J. (ed.): Automatic Verification Methods for Finite State Systems, International Workshop, Grenoble, France, June 12-14, 1989, Proceedings, Lecture Notes in Computer Science, vol. 407. Springer (1990).
80. Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349-1380 (2000).
81. Sousa, P., Fonseca, M. J.: Geometric matching for clip-art drawing retrieval. J. Vis. Commun. Image Represent. 20(2), 71-83 (2009).
82. Sousa, P., Fonseca, M. J.: Sketch-based retrieval of drawings using spatial proximity. J. Vis. Lang. Comput. 21(2), 69-80 (2010). Special Issue on Sketch Computation.
83. Szwoch, W.: Recognition, understanding and aestheticization of freehand drawing flowcharts. Int. Conf. Doc. Anal. Recognit. 2, 1138-1142 (2007).
84. Szwoch, W.: Aestheticization of flowcharts. In: Stapleton, G., Howse, J., Lee, J. (eds.) Diagrammatic Representation and Inference, Lecture Notes in Computer Science, vol. 5223, pp. 423-426. Springer, Berlin (2008).
85. Terrades, O., Tabbone, S., Valveny, E.: A review of shape descriptors for document analysis. In: ICDAR, vol. 1, pp. 227-231 (2007).
86. Tiwari, A., Bansal, V.: Patseek: Content based image retrieval system for patent database. In: ICEB, pp. 1167-1171 (2004).
87. Tombre, K.: Analysis of engineering drawings: State of the art and challenges. In: GREC, pp. 257-264 (1998).
88. Trucco, E., Verri, A.: Introductory Techniques for 3-D Computer Vision. Prentice Hall PTR, Upper Saddle River (1998).
89. Valveny, E., Tabbone, S., Ramos, O., Philippot, E.: Graphics recognition. Recent advances and new opportunities. Chap. Performance Characterization of Shape Descriptors for Symbol Representation, pp. 278-287. Springer, Berlin, Heidelberg (2008).
90. Vrochidis, S., Papadopoulos, S., Moumtzidou, A., Sidiropoulos, P., Pianta, E., Kompatsiaris, I.: Towards content-based patent image retrieval: A framework perspective. World Pat. Inf. 32(2), 94-106 (2010).
91. Wanner, L., Baeza-Yates, R., Brgmann, S., Codina, J., Diallo, B., Escorsa, E., Giereth, M., Kompatsiaris, Y., Papadopoulos, S., Pianta, E., Piella, G., Puhlmann, I., Rao, G., Rotard, M., Schoester, P., Serafini, L., Zervaki, V.: Towards content-oriented patent document processing. World Pat. Inf. 30(1), 21-33 (2008).
92. Weber, N., Henrich, A.: Retrieval of technical drawings in dxf format-concepts and problems. In: Lernen-Wissen-Adaption Workshop (LWA), pp. 213-220 (2007).
93. Wenyin, L., Zhang, W., Yan, L.: An interactive example-driven approach to graphics recognition in engineering drawings. Int. J. Doc. Anal. Recognit. 9(1), 13-29 (2007).
94. Yan, L., Wenyin, L.: Engineering drawings recognition using a case-based approach. In: ICDAR '03: Proceedings of the Seventh International Conference on Document Analysis and Recognition, pp. 190-194 (2003).
95. Yang, M., Qiu, G., Huang, J., Elliman, D.: Near-duplicate image recognition and content-based image retrieval using adaptive hierarchical geometric centroids. In: ICPR, pp. 958-961 (2006).
96. Yuan, Z., Pan, H., Zhang, L.: A novel pen-based flowchart recognition system for programming teaching, pp. 55-64 (2009).
97. Zhang, D., Lu, G.: Shape based image retrieval using generic fourier descriptors. Signal Proces. Image Commun. 17, 825-848 (2002).
98. Zhang, H., Petkovic, D.: Content-based representation and retrieval of visual media: A state-of-the-art review. Multimed Tools Appl 3, 179-202 (1996).
99. Zhiyuan, Z., Juan, Z., Bin, X.: An outward-appearance patent-image retrieval approach based on the contour-description matrix. In: FCST, pp. 86-89. IEEE Computer Society, Washington, DC, USA (2007).
100. Zhou, Y., Tan, C. L.: Chart analysis and recognition in document images. In: ICDAR, pp. 1055-1058 (2001).
101. Zhou, Y. P., Tan, C. L.: Hough technique for bar charts detection and recognition. In: Document Images, International Conference on Image Processing, ICIP 2000, pp. 494-497 (2000).