Multilingual information retrieval: From research to practice

Multilingual Information Retrieval: From Research to Practice

Volumn , Issue , 2012, Pages 1-217

  Peters, Carol ^a   Braschler, Martin ^b   Clough, Paul ^c  

^a ISTI CNR   (Italy)

^b ZURICH UNIVERSITY OF APPLIED SCIENCES   (Switzerland)

^c UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION RETRIEVAL; STUDENTS;

CROSS LANGUAGES; CROSS-LANGUAGE INFORMATION RETRIEVAL; GRADUATE STUDENTS; INFORMATION ACCESS SYSTEMS; MULTI-LINGUAL INFORMATION RETRIEVAL; MULTILINGUAL INFORMATION RETRIEVALS (MLIR); RETRIEVAL SYSTEMS; TEXT RETRIEVAL METHODS;

SEARCH ENGINES;

EID: 84955154765     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-642-23008-0     Document Type: Book

References (284)

1. stannual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1998). ACM Press: 64–71
2. EMIR (1994) Final report of the EMIR project number 5312. Commission of the European Union, Brussels
3. Ferro N, Peters C (2008) From CLEF to TrebleCLEF: the evolution of the cross-language evaluation forum. In: Proc. NTCIR-7 Workshop Meeting, December 16–19 2008, NII, Tokyo. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings7/
4. FIRE (2008) First workshop of the forum for information retrieval evaluation. http://www.isical. ac.in/~fire/2008/working_notes.html
5. FIRE (2010) Working notes FIRE 2010, 19-21 February 2010, DAIICT, Gandhinagar. http://www.isical.ac.in/~fire/2010/working_notes.html
6. Gey FC, Kando N, Peters C (2005) Cross-language information retrieval: the way ahead. J. Inf. Process. & Manag. 41(3): 415–431
7. Gey FC, Kando N, Lin C-Y, Peters C (2006) New directions in multilingual information access. SIGIR 2006 workshop report. ACM SIGIR Forum 40(2): 31–39
8. Gey FC, Kando N, Karlgren J (2009) Information access in a multilingual world: Transitioning from research to real-world applications. ACM SIGIR Forum 43(2): 24–28
9. Grefenstette G. (ed) (1998) Cross-language information retrieval. The Kluwer International Series on Information Retrieval, Kluwer Academic Publishers, Boston
10. Harman D (2003) The development and evolution of TREC and DUC. In Proc. 3rd NTCIR workshop on research in information retrieval, question answering, and summarization. NII, Tokyo. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings3/
11. Harman D, Braschler M, Hess M, Kluck M, Peters C, Schäuble P. Sheridan P, (2001) CLIR evaluation in TREC. In Peters C (ed) op.cit.: 7–23
12. Harman D, Kando N, Majumder P, Mitra M, Peters C (eds) (2010) Special issue on Indian language information retrieval. ACM Trans. Asian Lang. Inform. Process. 9(3)
13. Hovy E, Ide N, Frederkin R (eds) (1999) Multilingual information management: current levels and future abilities, NSF/EC/DARPA, http://www.cs.cmu.edu/~ref/mlim/index.html
14. ISO (1985) ISO Standard 5964-1985: Guidelines for the establishment and development of multilingual thesauri. First edition 1985-02-15. International Organisation for Standardisation, Technical Committee ISO/TC 46
15. ISO/IEC (1993) ISO/IEC International Standard 10646-1:1993(E): Information technology-Universal Multiple-Octet Coded Character Set (UCS)-Part 1: Architecture and basic multilingual plane. International Organization for Standardization, Geneva 1993
16. Kando N (2001). Overview of 2nd NTCIR workshop. In: Proc. 2nd NTCIR workshop on research in Chinese and Japanese text retrieval and text summarization, Tokyo, May 2000–March 2001. NII, Tokyo. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings2/overview-kando.pdf.
17. Kando N, Kuriyama K, Nozue T, Eguchi K, Kato H, Hidaka S, Adachi J (1999) The NTCIR workshop: the first evaluation workshop on Japanese text retrieval and cross-lingual information retrieval. In: Proc. 4th international workshop on information retrieval with Asian languages (IRAL’99), Nov. 11-12, 1999, Taipei, Taiwan.
18. Kando N, Mitamura T, Sakai T (2008) Introduction to the NTCIR-6 special issue. ACM Trans. Asian Lang. Inform. Process. 7(2): 1–3
19. Oard DW (ed) (2003) The surprise language exercises. ACM Trans. Asian Lang. Proc. 2(3-4): 79–84
20. Oard DW, Gey FC (2003) The TREC-2002 Arabic-English CLIR track. In: The eleventh text retrieval conference. TREC 2002. NIST special publication 500-251: 17–26
21. Peters C (ed) (2001) Cross-language information retrieval and evaluation. 1st workshop of crosslanguage evaluation forum, CLEF 2000. Springer LNCS 2069
22. Salton G (1971) Automatic processing of foreign language documents. Prentice-Hill: Englewood Cliffs, NJ
23. Schäuble P, Smeaton A (1998) An international research agenda for digital libraries: Summary report of the series of joint NSF-EU working groups on future directions for digital libraries research, 1998. http://www.ercim.eu/publication/ws-proceedings/DELOS-B/dl_sum_report.pdf
24. Abdou S, Savoy J (2006) Statistical and comparative evaluation of various indexing and search models. In: Proc. AIRS2006. Springer-Verlag LNCS 4182: 362–373
25. Amati G, Rijsbergen CJV (2002) Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inf. Syst. 20(4): 357–389
26. Baeza-Yates R, Ribeiro-Neto B (2011) Modern information retrieval, 2nd edn. ACM Press, New York
27. Braschler M, Ripplinger B (2004) How effective is stemming and decompounding for German text retrieval? Inf. Retr. 7(3–4): 291–316
28. Braschler M, Gonzalo J (2009) Best practices in system and user-oriented multilingual information access. TrebleCLEF Project: http://www.trebleclef.eu/
29. Braschler M, Peters C (2004) Cross-language evaluation forum: objectives, results, achievements. Inf. Retr. 7(1–2): 7–31
30. Brin S, Page L (1998) The anatomy of a large-scale hypertextual Web search engine. Comput. Netw. ISDN Syst. 30(1–7): 107–117
31. Broder AZ (2000) Identifying and filtering near-duplicate documents. In: Proc. 11th Annual Symposium on Combinatorial Pattern Matching (COM ’00). Springer-Verlag, London: 1–10
32. Chowdhury GG (2010) Introduction to modern information retrieval. Neal-Schuman Publishers.
33. Croft B, Metzler D, Strohman T (2009) Search engines: information retrieval in practice. Addison Wesley.
34. Dolamic L, Savoy J (2009) When stopword lists make the difference. J. of Am. Soc. for Inf. Sci. 61 (1): 200–203
35. Dunning T (1994) Statistical identification of language. Technical Report, CRL Technical Memo MCCS-94-273
36. El-Khair IA (2007) Arabic information retrieval. Annu. Rev. of Inf. Sci. and Technol. 41(1): 505–533
37. Erickson JC (1997) Options for the presentation of multilingual text: use of the Unicode standard. Library Hi Tech, 15(3/4): 172–188
38. Harman D (1991) How effective is suffixing?. J. of the Am. Soc. for Inf. Sci. 42(1): 7–15
39. Hiemstra D, de Jong F (1999) Disambiguation strategies for cross-language information retrieval. In: Proc. 3rd European Conference on Research and Advanced Technology for Digital Libraries (ECDL ’99). Springer-Verlag, London: 274–293
40. Hull DA (1996) Stemming algorithms-A case study for detailed evaluation. J. of the Am. Soc. for Inf. Sci. 47(1): 70–84
41. Kaszkiel M, Zobel J (1997) Passage retrieval revisited. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR ’97). ACM, New York: 178–185
42. Manning CD, Raghavan P, Schäutze H (2008) Introduction to information retrieval. Cambridge University Press
43. McNamee P (2009) JHU ad hoc experiments at CLEF 2008. In: Proc. 9th Workshop of the Cross-Language Evaluation Forum (CLEF 2008). Springer-Verlag LNCS 5706: 170–177
44. McNamee P and Mayfield J (2003) JHU/APL experiments in tokenization and non-word translation. In: Proc. 4th Workshop of the Cross-Language Evaluation Forum (CLEF 2003). Springer-Verlag LNCS 3237: 85–97
45. Mitra M, Buckley C, Singhal A, Cardie C (1997) An analysis of statistical and syntactic phrases. In Proc. 5th International Conference in Computer-Assisted Information Retrieval (RIAO 1997): 200–214
46. Ponte J, Croft B (1998) A language modeling approach to information retrieval. In Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR ’98). ACM, New York: 275–281
47. Porter MF (1980) An algorithm for suffix stripping. Program, 14(3):130–137. Reprint in: Spärck Jones K and Willett P (eds): Readings in Information Retrieval. Morgan Kaufmann Publishers, San Francisco: 313–316
48. Qiu Y, Frei H-P (1993) Concept based query expansion. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR ’93). ACM, New York: 160–169
49. Robertson SE (1977) The probability ranking principle in IR. J. of Doc. 33(4): 294–304
50. Robertson SE, van Rijsbergen CJ, Porter MF (1980): Probabilistic models of indexing and searching. Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR ’80). ACM, New York: 35–56
51. Robertson SE, Maron ME, Cooper WS (1982) Probability of relevance: a unification of two competing models for document retrieval. Info. Tech: R and D. 1: 1–21
52. Salton G, Wong A, Yang C (1975) A vector space model for automatic indexing. Commun. of the ACM 18(11): 613–620
53. Salton G, Buckley C (1990) Improving retrieval performance by relevance feedback. J. of Am. Soc. for Inf. Sci. 41(4): 288–297
54. Savoy J (2005) Comparative study of monolingual and multilingual search models for use with Asian languages. ACM Trans. on Asian Lang. Inf. Proc. 42(2): 163–189
55. Schäuble P (1997) Multimedia information retrieval. Content-based information retrieval from large text and audio databases. Kluwer Academic Publishers.
56. Sedgewick R, Wayne K (2011) Algorithms, Fourth Edition, Section 3.4 “Hash Tables”. Addison-Wesley Professional Singhal A, Buckley C, Mitra M (1996) Pivoted document length normalization. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR ’96). ACM, New York: 21–29
57. Spärck Jones K, Willett P (1997) Readings in information retrieval. Morgan Kaufmann Tague-Sutcliffe J (ed) (1996) Evaluation of information retrieval systems. J. Am. Soc. for Inf. Sci. 47(1)
58. Walker S, Robertson SE, Boughanem M, Jones GJF, Spärck Jones K (1998) Okapi at TREC-6, automatic ad hoc, VLC, routing, filtering and QSDR. In Voorhees EM, Harman DK, (eds) The Sixth Text REtrieval Conference (TREC-6). NIST Special Publication 500–240: 125–136
59. Ubiquity (2002) Talking with Terry Winogard. Ubiquity Magazine 3(23)
60. Ballesteros LA, Croft WB (1997) Phrasal translation and query expansion techniques for crosslanguage information retrieval. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR 1997), ACM Press: 84–91
61. Belkin NJ, Kantor P, Cool C, Quatrain R (1994) Combining evidence for information retrieval. In: Proc. of the 2nd Text REtrieval Conference (TREC-2). NIST Special Publication 500-215: 35–43
62. Braschler M (2004a) Combination approaches for multilingual text retrieval. J. Inf. Retr. 7(1/2): 183-204
63. Braschler M (2004b) Robust multilingual information retrieval. Doctoral Thesis, Institut interfacultaire d’informatique, Universite´ de Neuchâtel
64. Braschler M, Gonzalo J (2009) Best practices in system and user-oriented multilingual information access. TrebleCLEF Project: http://www.trebleclef.eu/
65. Braschler M, Schäuble P (1998) Multilingual information retrieval based on document alignment techniques. In: Proc. 2nd European Conference on Research and Advanced Technology for Digital Libraries, Second European Conference (ECDL 1998), Springer-Verlag: 183–197
66. Buckley C, Mitra M, Walz J, Cardie C (1998) Using clustering and super concepts within SMART: TREC-6. In: Proc. of the 6th Text REtrieval Conference (TREC-6), NIST Special Publication 500-240: 107–124
67. Darwish K, Oard DW (2003) Probabilistic structured query methods. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR 2003). ACM Press: 338–344
68. Ferro N, Peters C (2010) CLEF 2009 ad hoc track overview: TEL and Persian tasks. In: Multilingual Information Access Evaluation I. Text Retrieval Experiments, Springer LNCS 6241: 13–35
69. Fox EA, Shaw JA (1993) Combination of multiple searches. In: Proc. of the 2nd Text Retrieval Conference (TREC-2), NIST Special Publication 500-215: 243–252
70. Gey FC (2005) How similar are Chinese and Japanese for cross-language information retrieval? In: Proc. of NTCIR-5 Workshop Meeting, December 6–9, 2005, Tokyo, Japan.
71. Gollins T, Sanderson M (2001) Improving cross language retrieval with triangulated translation. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR 2001). ACM Press: 90–95
72. Grefenstette G (1998) (ed): Cross-language information retrieval. Kluwer Academic Publishers
73. Hedlund T, Airio E, Kekustalo H, Lehtokangas R, Pirkola A, Järvelin K (2004) Dictionary-based cross-language information retrieval: Learning experience from CLEF. J. Inf. Retr. 7: 97–117
74. Hiemstra D, de Jong F (1999) Disambiguation strategies for cross-language information retrieval. In: Proc. of the 3rd European Conference on Research and Advanced Technology for Digital Libraries (ECDL 1999). Springer-Verlag: 274–293
75. Littman ML, Dumais ST, Landauer TK (1998) Automatic cross-language information retrieval using latent semantic indexing. Chapter 5. In: Grefenstette (ed) Cross-Language Information Retrieval. Kluwer Academic Publishers: 51–62
76. Mandl T, Womser-Hacker C, Di Nunzio GM, Ferro N (2008) How robust are multilingual information retrieval systems? In: Proc. ACM Symposium on Applied Computing (SAC 2008): 1132–1136
77. McNamee P, Mayfield J (2002) Comparing cross-language query expansion techniques by degrading translation resources. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR 2002). ACM Press: 159–166
78. McNamee P, Mayfield J (2004) Character n-gram tokenization for European language text retrieval. J. Inf. Retr. 7: 73–97
79. Moreau N (2009) Best Practices in language resources for multilingual information access. TrebleCLEF project: http://www.trebleclef.eu/
80. Nie J-Y (2010) Cross-language information retrieval. Morgan & Claypool Nie J-Y, Simard M, Isabelle P, Durand R (1999) Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the Web. Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR ’99). ACM Press: 74–81
81. Pirkola A (1998) The effects of query structure and dictionary setups in dictionary-based crosslanguage information retrieval. In: Proc. ACM SIGIR conference on research and development in information retrieval (SIGIR 1998). ACM Press: 55–63
82. Qiu Y, Frei H-P (1993) Concept based query expansion. In: Proc. of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993). ACM Press: 160–169
83. Savoy J (2004) Combining multiple strategies for effective monolingual and cross-language retrieval. J. Inf. Retr. 7(1/2): 119–146
84. Savoy J, Dolamic L (2009) How effective is Google’s translation service in search? Commun. of the ACM 52(10): 139–143
85. Sheridan P, Braschler M, Schäuble P (1997) Cross-language information retrieval in a multilingual legal domain. In: 1st European Conference on Research and Advanced Technology for Digital Libraries (ECDL 1997), LNCS 1324: 253–268
86. Si L, Callan J (2005) Multilingual retrieval by combining multiple ranked lists. In: Peters C (ed) Proc. of the 6th Workshop of the Cross-Language Evaluation Forum (CLEF 2005). Available at http://www.clef-campaign.org Yu K, Tsujii J (2009) Bilingual dictionary extraction from Wikipedia. In Proc. of Machine Translation Summit XII (MTS-2009). Available at http://www.mt-archive.info/MTS-2009-Yu.pdf
87. Marchionini G (1995) Information seeking in electronic environments. Cambridge University Press
88. Marchionini G (2006) Exploratory search: from finding to understanding. Commun. of the ACM 49(4):41–46
89. Marlow J, Clough P, Dance K (2007) Multilingual needs of cultural heritage website visitors: A case study of Tate Online. In: Proc. of International Cultural Heritage Informatics Meeting (ICHIM07). Available at http://www.archimuse.com/ichim07/papers/marlow/marlow.html. Cited 15 Feb 2011
90. Marlow J, Clough P, Cigarràn Recuero J, Artiles J (2008a) Exploring the effects of language skills on multilingual web search. In: Proc. of the 30th European Conference on IR Research (ECIR’08). LNCS 5478: 126–137
91. Marlow J, Clough P, Ireson N, Cigarràn Recuero J, Artiles J, Debole F (2008b) The MultiMatch project: Multilingual/multimedia access to cultural heritage on the web, In: Proc. of Museums on the Web Conference (MW2008). Available at http://www.archimuse.com/mw2008/papers/marlow/marlow.html. Cited 15 Feb 2011
92. McCracken DD, Wolfe RJ (2004) User-centred website development: A human-computer interaction approach. Pearson Education Inc.
93. Minelli S, Marlow J, Clough P, Cigarran J, Gonzalo J, Oomen J (2007) Gathering requirements for multilingual search of audiovisual material in cultural heritage. In: Proc. of Workshop on User Centricity–state of the art (16th IST Mobile and Wireless Communications Summit), Budapest, Hungary
94. Morville P, Rosenfeld L (2006) Information architecture for the world wide web: Designing largescale web sites. 3rd Edition. O’Reilly
95. Mulhem P, Nigay L (1996) Interactive information retrieval systems: from user centred interface design to software design. In: Proc. of the 19th Annual international ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’96). ACM Press. Zurich, Switzerland: 326–334
96. Nielsen J (1994) Heuristic evaluation. In: Nielsen J, Mack RL (eds), Usability Inspection Methods. John Wiley & Sons, New York
97. Oard D, Gonzalo J (2002) The CLEF 2001 interactive track. In: Proc. of 2nd Workshop of the Cross-Language Evaluation Forum (CLEF 2001). LNCS 2406: 372–382
98. Oard D, Gonzalo J, Sanderson M, López-Ostenero F, Wang J (2004) Interactive cross-language document selection. Inf. Retr. 7(1–2):205–228
99. Oard DW, He D, Wang J (2008) User-assisted query translation for interactive cross-language information retrieval. Inf. Process. & Manag. 44(1):180–211
100. Ogden WC, Davis MW (2000) Improving cross-language text retrieval with human interactions. In: Proc. of the Hawaii International Conference on System Science (HICSS-33). Vol. 3
101. Ogden W, Cowie J, Davis M, Ludovik E, Nirenburg S, Molina-Salgado H (1999) Keizai: An interactive cross-language text retrieval system. Machine Translation Summit VII, Workshop on Machine Translation for Cross-Language Information Retrieval, Singapore
102. Peñas A, Gonzalo J, Verdejo F (2001) Cross-language information access through phrase browsing. 6th International Conference of Natural Language for Information Systems (NLDB’01), Madrid, Spain
103. Petrelli D, Hansen P, Beaulieu M, Sanderson M (2002) User requirement elicitation for Cross-Language Information Retrieval. New Rev. of Inf. Behav. Res. 3:17–35
104. Petrelli D, Hansen P, Beaulieu M, Sanderson M, Demetriou G, Herring P (2004) Observing users–designing Clarity: A case study on the user-centred design of a cross-language retrieval system. J. of the Am. Society for Inf. Sci. and Technology 55(10):923–934
105. Petrelli D, Clough P (2006) Using concept hierarchies in text-based image retrieval: A user evaluation, In: Accessing Multilingual Information Repositories. LNCS 4022: 297–306
106. Preece J, Rogers Y, Sharp H (2002) Interaction design: Beyond human-computer interaction. Wiley, New York
107. Resnik P (1997) Evaluating multilingual gisting of Web pages. In: Working Notes of the AAAI97 workshop on Cross-Language Text and Speech Retrieval
108. Resnick M, Vaughan M, (2006) Best practices and future visions for search user interfaces. J. of the Am. Soc. for Inf. Sci. and Technol. 57(6):781–787
109. Rieh HY, Rieh SY (2005). Web searching across languages: Preference and behavior of bilingual academic users in Korea. Libr. & Inf. Sci. Res. 27(2):249–263
110. Rieh SY, Xie H (2006) Analysis of multiple query reformulations on the Web: The interactive information retrieval context. Inf. Process. & Manag. 42(3):751–768
111. Robins D (2000) Interactive information retrieval: Context and basic notions. Informing Sci. 3:57–62
112. Rubin J (1994) Handbook of usability testing: how to plan, design, and conduct effective tests. Wiley, New York
113. Ruthven I (2009) Interactive information retrieval. Annu. Rev. of Inf. Sci. and Technol. 42 (1):43–91
114. Savourel Y (2001) XML internationalization and localization. Sams Publishing, Indianapolis
115. Shen S, Woolley M, Prior S (2006) Towards culture-centred design. Interact. with Comput. 18 (4):820–852
116. Shneiderman B, Byrd D, Croft B (1998) Sorting out searching: a user-interface framework for text searches. Commun. of the ACM 41(4):95–98
117. Steinerová J, (2008) Seeking relevance in academic information use. Inf. Res. 13(4) paper 380. Available at http://InformationR.net/ir/13-4/paper380.html. Cited 15 Feb 2011
118. Sutcliffe A, Ennis M (1998) Towards a cognitive theory of information retrieval, Interacting with computers. HCI and Inf. Retr. 10(3):321–351
119. Taylor R (1968) Question negotiation and information seeking in libraries. Coll. and Res. Libr. 29 (3):178–194
120. Tombros A, Sanderson M (1998) Advantages of query-biased summaries in IR. In: Proc. of the 21st ACM Conference of the Special Interest Group in Information Retrieval: 2–10
121. Vakkari P (2003) Task-based information searching. Ann. Rev. of Inf. Sci. and Technol. 37:413–464
122. Wildemuth BM (2006) Evidence-based practice in search interface design. J. of the Am. Soc. for Inf. Sci. & Technol. 57(6):825–828
123. Wilson TD, (2000) Human information behaviour. Informing Sci. 3(2): 49–56. Available at http://inform.nu/Articles/Vol3/v3n2p49-56.pdf. Cited 15 Feb 2011
124. Wilson ML, Kules B, Schraefel MC, Shneiderman B (2010) From keyword search to exploration: Designing future search interfaces for the web. Found. and Trends in Web Sci. 2(1):1–97.
125. Zoe LR, DiMartino D (2000) Cultural diversity and end-user searching: An analysis by gender and language background. Res. Strateg. 17(4):291–305
126. Alonso O, Rose DE, Stewart B (2008) Crowdsourcing for relevance evaluation. ACM SIGIR Forum. 42(2): 9–15
127. Aslam JA, Yilmaz E, Pavlu V (2005) A geometric interpretation of R-precision and its correlation with average precision. In: Proc. 28th ACM SIGIR conference on research and development in information retrieval (SIGIR 2005). ACM Press: 573–574
128. Bailey P, Craswell N, Soboroff I, Thomas P, de Vries AP, Yilmaz E (2008) Relevance assessment: Are judges exchangeable and does it matter? In: Proc. 31st ACM SIGIR conference on research and development in information retrieval. ACM Press: 667–674
129. Borlund P (2003) The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. J Inf. Res. 8(3)
130. Borlund P (2009) User-centred evaluation of information retrieval systems. Chapter 2. In: G€oker A, Davies J (eds) Information Retrieval: Searching in the 21st Century. John Wiley & Sons: 21–37
131. Braschler M (2004a) CLEF 2003–Overview of results. In: Comparative evaluation of multilingual information access systems. 4th workshop of the Cross-Language Evaluation Forum, CLEF 2003, Springer LNCS 3237: 44–63
132. Braschler M (2004b) Robust multilingual information retrieval. Doctoral Thesis, Institut interfacultaire d’informatique, Universite´ de Neuchâtel
133. Braschler M, Gonzalo J (2009) Best practices in system and user-oriented multilingual information access. TrebleCLEF Project: http://www.trebleclef.eu/
134. Braschler M, Peters C. (eds) (2004) Cross-language evaluation forum. J Inf.Retr. 7(1–2) 2004
135. Buckley C, Voorhees E (2004) Retrieval evaluation with incomplete information. In Proc. 27th ACM SIGIR conference on research and development in information retrieval (SIGIR 2004). ACM Press: 25–32
136. Carterette B, Pavlu V, Kanoulas E, Aslam JA, Allan J (2008) Evaluation over thousands of queries. In: Proc. 31st ACM SIGIR conference on research and development in information retrieval (SIGIR 2009). ACM Press: 651–658
137. Cleverdon CW (1967) The Cranfield tests on index language devices. In: Aslib Proc. 19(6): 173–192
138. Cleverdon CW (1970) The effect of variations in relevance assessments in comparative experimental tests of index languages. Technical Report No. 3, Cranfield Institute of Technology, Cranfield, UK, 1970
139. Clough P, Sanderson M, Müller H (2004) The CLEF cross language image retrieval track (ImageCLEF) 2004. In: Image and Video Retrieval (CIVR 2004), Springer, LNCS 3115: 243–251
140. Clough P, Gonzalo J, Karlgren J (2010a) Creating re-useable log files for interactive CLIR, In SIGIR 2010 Workshop on the Simulation of Interaction (SimInt), Geneva, Switzerland, 23 July 2010
141. Clough P, Müller H, Sanderson M, (2010b) Seven years of image retrieval evaluation. In:Müller H, Clough P, Deselaers Th, Caputo, B. (eds) (2010) ImageCLEF experimental evaluation in visual information retrieval. The Information Retrieval Series. Springer 32: 3–18
142. Cormack GV, Palmer CR, Clarke CLA (1998) Efficient construction of large test collections. In: Proc. 21st ACM SIGIR conference on research and development in information retrieval (SIGIR 1998). ACM Press: 282–289
143. Croft B, Metzler D, Strohman T (2009) Evaluating search engines. In: Search engines: Information retrieval in practice 1st ed., Addison Wesley: 269–307
144. Dunlop M (2000) Reflections on MIRA: Interactive evaluation in information retrieval. J. Am. Soc. for Inf. Sci. 51(14): 1269–1274
145. Geva S, Kamps J, Peters C, Sakai T, Trotman A, Voorhees E (eds) (2009) Proc. SIGIR 2009 workshop on the future of IR evaluation. SIGIR 2009, Boston USA. http://staff.science.uva.nl/~kamps/publications/2009/geva:futu09.pdf
146. Gonzalo J, Oard DW (2003). The CLEF 2002 interactive track. In: Advances in cross-language information retrieval. 3rd workshop of the Cross-Language Evaluation Forum, CLEF 2002. Springer LNCS 2785: 372–382
147. Hansen P, (1998) Evaluation of IR user interface–Implications for user interface design. Human IT. http://www.hb.se/bhs/it
148. Harman DK (ed) (1992) Evaluation issues in information retrieval. J. Inform. Process. & Manag. 28(4)
149. Harman DK (2005) The TREC test collections. In: Voorhees EM, Harman DK (eds) TREC: experiment and evaluation in information retrieval, MIT Press, 2005
150. Howe J (2006). The rise of crowdsourcing. Wired, June 2006. http://www.wired.com/magazine/
151. Hull D (1993) Using statistical testing in the evaluation of retrieval experiments. In: Proc. 16th ACM SIGIR conference on research and development in information retrieval (SIGIR 1993). ACM Press: 329–338
152. Järvelin K, Kekäläinen J (2000) IR evaluation methods for retrieving highly relevant documents. In: Proc. 23rd ACM SIGIR conference on research and development in information retrieval (SIGIR 2000). ACM Press: 41–48
153. Jansen BJ, Spink A (2006) How are we searching the World Wide Web? A comparison of nine search engine transaction logs. J. Inform. Process. & Manag. 42(1): 248–263
154. Kando N, Kuriyama K, Nozue T, Eguchi K, Kato H, Hidaka S (1999) Overview of IR tasks at the first NTCIR workshop. In Proc. First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition: 11–44
155. Karlgren J, Gonzalo J (2010) Interactive image retrieval. In Müller H, Clough P, Deselaers Th, Caputo, B. (eds) ImageCLEF. Experimental evaluation in visual information retrieval. The Information Retrieval Series. Springer 32: 117–139
156. Kelly D. (2009) Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval. 228 p.
157. Lesk ME, Salton G. (1969) Relevance assessments and retrieval system evaluation. J Inf. Storage and Retr. 4: 343–359
158. Leung C, Ip H (2000) Benchmarking for content-based visual information search. In Fourth International Conference on Visual Information Systems (VISUAL’2000), LNCS 1929, Springer-Verlag: 442–456
159. Mandl T (2009) Easy tasks dominate information retrieval evaluation results. In Lecture Notes in Informatics. Datenbanksysteme in Business, Technologie und Web (BTW): 107–116
160. Mandl T, Womser-Hacker C, Di Nunzio GM, Ferro N (2008) How robust are multilingual information retrieval systems? Proc. SAC 2008-ACM Symposium on Applied Computing: 1132–1136
161. Mizzaro S (1998) Relevance: The whole history. J. of Am. Soc. for Inf. Sci. 48(9): 810–832
162. Müller H, Müller W, Squire DM, Marchand-Maillet S, Pun T, (2001) Performance evaluation in content-based image retrieval: overview and proposals. Pattern Recognit. Lett. 22(5): 593–601
163. Müller H, Clough P, Deselaers Th, Caputo B, (eds) (2010) ImageCLEF. Experimental evaluation in visual information retrieval. The Information Retrieval Series, Springer 32: 495 p.
164. Peters C (ed) (2000) First results of the CLEF 2000 cross-language text retrieval system evaluation campaign. Working notes for the CLEF 2000 workshop, ERCIM-00-W01: 142
165. Petrelli D (2007) On the role of user-centred evaluation in the advancement of interactive information retrieval. J. Inform. Process & Manag. 44(1): 22–38
166. van Rijsbergen CJ (1979) Evaluation. In: Information Retrieval 2nd ed., Butterworths Robertson S (2006) On GMAP: and other transformations. Proc. 15th ACM international conference on information and knowledge management (CIKM ’06). ACM Press: 78–83
167. Robertson S (2008) On the history of evaluation in IR. J. of Inf. Sci. 34(4): 439–456
168. Salton G (1968) Automatic information organization and retrieval. McGraw Hill Text Sanderson M, Braschler M (2009) Best practices for test collection creation and information retrieval system evaluation, TrebleCLEF Project: http://www.trebleclef.eu
169. Sanderson M, Zobel J (2005) Information retrieval system evaluation: effort, sensitivity, and reliability. In: Proc. 28th ACM SIGIR conference on research and development in information retrieval (SIGIR 2005). ACM Press: 162–169
170. Sanderson M, Paramita M, Clough P, Kanoulas E (2010) Do user preferences and evaluation measures line up? In: Proc. 33rd ACM SIGIR conference on research and development in information retrieval (SIGIR 2010). ACM Press: 555–562
171. Saracevic T (1995) Evaluation of evaluation in information retrieval. In: 18th ACM SIGIR conference on research and development in information retrieval (SIGIR 1995), ACM Press: 138–146
172. Schäuble P, (1997) Multimedia information retrieval: Content-based information retrieval from large text and audio databases, Kluwer Academic Publishers Schamber L (1994) Relevance and information behaviour. Annual Review of Information Science and Technology, 29: 3–48
173. Smith JR (1998) Image retrieval evaluation. In: Proc IEEE Workshop on Content-based Access of Image and Video Libraries:112–113
174. Spärck Jones K (1981) Information Retrieval Experiment. Butterworth-Heinemann Ltd. 352 p.
175. Spärck Jones K, van Rijsbergen CJ (1975) Report on the need for and provision of an “ideal” information retrieval test collection. British Library Research and Development report 5266. Cambridge: Computer Laboratory, University of Cambridge Su LT (1992) Evaluation measures for interactive information retrieval. J. Inform. Process & Manag. 28(4): 503–516
176. Tague-Sutcliffe J (1992) The pragmatics of information retrieval experimentation, revisited. J. Inform. Process & Manag. 28(4): 467–490
177. Tague-Sutcliffe J (ed) (1996) Evaluation of information retrieval systems, J. Am. Soc. for Inf. Sci. 47(1)
178. Tomlinson S (2010) Sampling precision to depth 10000 at CLEF 2009. In: Multilingual information access evaluation Part I: Text retrieval experiments: 10th workshop of the Cross-Language Evaluation Forum, CLEF 2009, Springer, LNCS 6241: 78–85
179. Voorhees EM (2000) Variations in relevance judgments and the measurement of retrieval effectiveness. J Inform. Process. & Manag. 36(5): 697–716
180. Voorhees EM (2002) The philosophy of information retrieval evaluation. In: Evaluation of crosslanguage information retrieval systems: 2nd workshop of the Cross-Language Evaluation Forum, CLEF 2001, Springer, LNCS 2406: 355–370
181. Voorhees EM (2006) Overview of TREC 2006. Fifteenth text retrieval conference (TREC 2006) Proc. NIST Special Publication SP500-272. http://trec.nist.gov/pubs/trec15/
182. Voorhees EM, Buckley C (2002) The effect of topic set size on retrieval experiment error. In: Proc. 25th ACM SIGIR conference on research and development in information retrieval (SIGIR 2002). ACM Press: 316–323
183. Voorhees EM, Garofolo JS (2005) Retrieving noisy text. In Voorhees EM, Harman DK (eds). TREC. experimentation and evaluation in information retrieval. MIT Press: 183–197
184. Voorhees EM, Harman DK (eds) (2005) TREC: experiment and evaluation in information retrieval. The MIT Press, Cambridge, MA Womser-Hacker C (2002) Multilingual topic generation within the CLEF 2001 experiments. In: Evaluation of cross-language information retrieval systems: 2nd workshop of the Cross-Language Evaluation Forum, CLEF 2001, Springer, LNCS 2406: 255–262
185. Zobel J (1998) How reliable are the results of large-scale information retrieval experiments? In: Proc. 21st ACM SIGIR conference on research and development in information retrieval (SIGIR 1998). ACM Press: 307–314
186. Bernardi R, Calvanese D, Dini L, Di Tomaso V, Frasnelli E, Kugler U, Plank B (2006) Multilingual search in libraries. The case-study of the Free University of Bozen-Bolzano. In: Proc. 5th International Conference on Language Resources and Evaluation-LREC 2006, Genoa
187. Bilal D, Bachir I (2007) Children’s interaction with cross-cultural and multilingual digital libraries: I & II. Understanding interface design representations. J. Inf. Process. and Manag. 43: 47–80
188. Byrne W, Doermann D, Franz F, Gustman S, Hajı˘c J, Oard D, Picheny M, Psutka J, Ramabhadran B, Soergel D, Ward T, Zhu W-J (2004) Automatic recognition of spontaneous speech for access to multilingual and historical archives. IEEE Trans. Speech and Audio Process. 12 (4): 420–435
189. Chau M, Qin J, Zhou Y, Tseng C, Chen H (2008) SpidersRUs: Creating specialized search engines in multiple languages. Decis. Support Syst. 45(3): 621–640
190. Chen J, Bao Y (2009) Information access across languages on the web: From search engines to digital libraries. In: Proc. Am. Soc. for Inf. Sci. and Technol. 46: 1–14
191. Chung W (2008) Web searching in a multilingual world. Commun. ACM 51(5): 32–40
192. Chung W, Zhang Y, Huang Z, Wang G, Ong TH, Chen H (2004) Internet searching and browsing in a multilingual world: an experiment on the Chinese business intelligence portal (CBizPort). J. Am. Soc. Inf. Sci. and Technol. 55(9): 818–831
193. Cleveland A, Pan D, Chen J, Yu X, Philbrick J, O’Neill M, Smith L (2008) Analysis of the health information needs and health related Internet usage of a Chinese population in the United States. J. Libr. and Inf. Service 52(3): 112–11
194. Clough P, Sanderson M (2006) User experiments with the Eurovision cross-language image retrieval system. J. Am. Soc. for Inf. Sci. 57(5): 697–708
195. Clough P, Eleta I (2010) Investigating language skills and field of knowledge on multilingual information access in digital libraries. Int. J. Digit. Libr. Syst. 1(1): 89–103
196. Clough P, Gonzalo J, Karlgren J, Barker E, Artiles J, Peinado V (2008) Large-scale interactive evaluation of multilingual information access systems-the iCLEF Flickr challenge. In: Proc. of Workshop on novel methodologies for evaluation in information retrieval, 30th European Conference on Information Retrieval, Glasgow, 30 March-3rd April 2008
197. Colowick S, (2008) Multilingual search with PanImages. MultiLingual 19(2):61–63. Available at http://turing.cs.washington.edu/PanImMultilingual.pdf. Cited 15 Apr 2011
198. de Jong F, Oard DW, Heeren W, Ordelman R (2008) Access to recorded interviews: a research agenda. ACM J. Comput. in Cult. Herit. 1(1). Article 3
199. Del Bimbo A (1999) Visual information retrieval. Morgan Kaufmann Publishers Inc., San Francisco Delezoide B, Le Borgne H (2007) SemanticVox: A multilingual video search engine. In: CIVR’07 Proc. of 6th ACM International Conference on Image and Video Retrieval: 81–84
200. Depeursinge A, Müller H (2010) Fusion techniques for combining textual and visual information retrieval. In: ImageCLEF: Experimental Evaluation in Information Retrieval. The Information Retrieval Series. Springer: 95–114
201. Dini L, Peters W, Liebwald D, Schweighofer E, Mommers L, Voermans, W (2005) Cross-lingual legal information retrieval using a WordNet architecture. In: Proc. of 10th International Conference on Artificial Intelligence and Law (ICAIL ’05). ACM, New York: 163–167
202. Dumais S, Banko M, Brill E, Lin J, Ng A (2002) Web question answering: Is more always better? In: Proc. ACM SIGIR conference on research and development in informatiion retrieval, SIGIR 2002: 291–298
203. Duncker E (2002) Cross-cultural usability of the library metaphor. In: Proc. of 2nd ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL ’02). ACM, New York: 223–230
204. Eakins J, Briggs P, Burford B (2004) Image retrieval interfaces: A user perspective. In: Proc. CIVR 2004: 628–637
205. Enser PGB (1995) Pictorial information retrieval. J. of Documentation 51(2): 126–170
206. Federico M, Jones GJF (2004) The CLEF 2003 cross-language spoken document retrieval track. In: Proc. CLEF 2003. Comparative Evaluation of Multilingual Information Access Systems, Springer LNCS 3237: 646–652
207. Federico M, Bertoldo N, Levow G-A, Jones GJF (2005) CLEF 2004 Cross-language spoken document retrieval track. In: Proc. CLEF 2004. Multilingual Information Access for Text, Speech and Images, Springer LNCS 3491: 817–820
208. Feldman S, Sherman C (2003) The high cost of not finding information. Tech. Rep. 29127, IDC, April 2003
209. Ferro N, Peters C (2010) CLEF 2009 ad hoc track overview: TEL and Persian tasks. In: Proc. CLEF 2009. Multilingual Information Access Evaluation I. Text Retrieval Experiments, Springer LNCS 6241: 13–35
210. Forner P, Giampiccolo D, Magnini B, Peñas A, Rodrigo A, Sutcliffe R (2010) Evaluating multilingual question answering systems at CLEF. In: Proc. 7th International Conference on Language Resources and Evaluation. LREC 2010, Malta: 2774–2781
211. Fujii A., Iwayama M, Kando N (2004) Test collections for patent-to-patent retrieval and patent map generation in NTCIR-4 workshop. In: Proc. 4th International Conference on Language Resources and Evaluation. LREC 2004: 1643–1646
212. Fujii A, Iwayama M, Kando N (2007) Introduction to the special issue on patent processing. J. Inf. Process. and Manag. 43(5): 1149–1153
213. Fukimoto J, Kato T, Masui F, Mori T, (2007) Overview of the 4th question answering challenge at NTCIR workshop-6. NTCIR-6 Workshop Proc., National Institute of Informatics, Tokyo. Available at http://research.nii.ac.jp/ntcir/ntcir-ws6/OnlineProceedings/NTCIR/75-revised-20070521.pdf. Cited 15 Apr 2011
214. Garofolo J, Fiscus J, Fisher W (1997) Design and preparation of the 1996 Hub-4 broadcast news benchmark test corpora, Proc. DARPA Speech Recognition Workshop, February 1997
215. Garofolo J, Auzanne CGP, Voorhees E (2000) The TREC spoken document retrieval track: A success story. In: Proc. RIAO 2000: 1–20
216. Goodrum A (2000) Image information retrieval: An overview of current research. Informing Sci. 3 (2): 63–66
217. Hauptmann AG, Scheytt P, Wactlar HD, Kennedy PE (1998) Multi-lingual Informedia: A demonstration of speech recognition and information retrieval across multiple languages. In: Information Retrieval Across Multiple Languages, BNTUW-98 Proc. of DARPA Workshop on Broadcast News Understanding Systems
218. Hauptmann AG, Jin R, Ng TD (2002) Multi-modal information retrieval from broadcast video using OCR and speech recognition. In: JCDL ’02 Proc. of 2nd ACM/IEEE-CS joint conference on Digital Libraries: 160–161
219. Hawking D (2004) Challenges in enterprise search. In: Proc. Fifteenth Australasian Database Conference (ADC 2004): 15–24
220. Hecht B, Gergle D (2010) The tower of Babel meets Web 2.0: user-generated content and its applications in a multilingual context. In: Proc. 28th International Conference on Human Factors in Computing Systems (CHI ’10). ACM, New York: 291–300
221. Hersh WR, Donohoe LC (1998) SAPHIRE international: a tool for cross-language information retrieval. In: Proc. 1998 AMIA Annual Symposium: 673–677
222. Hersh WR, Müller H, Jensen JJ, Yang J, Gorman PN, Ruch P (2006). Advancing biomedical image retrieval: development and analysis of a test collection. J. Am. Med. Inform. Assoc. 13: 488–496
223. Hirschman L, Gaizauskas R (2001) Natural language question answering: The view from here. Nat. Lang. Eng. 7(4): 275–300
224. Hsu DF, Taksa I (2005) Comparing rank and score combination methods for data fusion in information retrieval. J. Inf. Retr. 8(3): 449–480
225. Huijbregts MAH, Ordelman RJF, de Jong FMG (2007) Annotation of heterogeneous multimedia content using automatic speech recognition. In: Proc. Semantic and Digital Media Technologies 2nd International Conference on Semantic Multimedia (SAMT’07). Falcidieno B, Spagnuolo M, Avrithis Y, Kompatsiaris I, Buitelaar P (eds). Springer-Verlag, Berlin, Heidelberg: 78–90
226. Jones GJF (2000) Applying machine translation resources for cross-language information access from spoken documents. In: Proc. MT 2000: Machine Translation and Multilingual Applications in the New Millennium, Exeter, U.K: 4-1–4-9
227. Jones GJF, Larson M, Marchand-Maillet S, (2008). Multilingual/multimedia information retrieval. In: State-of-the-Art. Del. 1.1 MultiMatch Project. http://www.multimatch.eu/docs/publicdels/1.1.3.pdf
228. Kahn Jr CE (2009) Multilingual retrieval of radiology images. Radio Graphics 29: 23–29
229. Karlgren J, Gonzalo J (2010) Interactive image retrieval. In: ImageCLEF-Experimental Evaluation in Visual Information Retrieval. The Information Retrieval Series, (32). Springer, 117–138. ISBN 978-3-642-15180-4
230. Katz B, Borchardt G, Felshin S, Shen Y, Zaccak G (2007) Answering English questions using foreign-language, semi-structured sources. In: Proc. International Conference on Semantic Computing (ICSC ’07). IEEE Computer Society, Washington, DC: 439–445
231. Koehn P (2005) Europarl: A parallel corpus for statistical machine translation. In: Proc. of the Machine Translation Summit X: 79–86
232. Koolen M, Adriaans F, Kamps J, de Rijke, M (2006) A cross-language approach to historic document retrieval. In: Proc. 28th European Conference on IR Research (ECIR 2006): 407–419
233. Larson M, Newman E, Jones GJF (2009) Overview of Video CLEF 2008: automatic generation of topic-based feeds. In: Proc. CLEF 2008 Evaluating Systems for Multilingual and Multimodal Information Access. Springer LNCS 5706: 906–917
234. Larson M, Newman E, Jones GJF (2010) Overview of Video CLEF 2009: new perspectives on speech-based multimedia content. In: Proc. CLEF 2009 Multilingual Information Access Evaluation II. Multimedia Experiments, Springer LNCS 6242: 354–368
235. Lazarinis F, Vilares J, Tait J, Efthimiadis EN (2009) Current research issues and trends in non-English web searching. J. Inf. Retr. 12(3): 230–250
236. Liu F, Ackerman M, Fontelo P (2006) BabelMeSH: development of a cross-language tool for Medline/PubMed. In: Proc. AMIA Annual Symposium: 1012
237. Lu WH, Lin RS, Chan YC, Chen KH (2008) Using Web resources to construct multilingual medical thesaurus for cross-language medical information retrieval. Decis. Support Syst. 45 (3): 585–595
238. Lyu MR, Song J, Cai M (2005) A comprehensive method for multilingual video text detection, localization, and extraction. IEEE Trans. Circuits and Syst. for Video Technol. 15(2): 243–255
239. Maeda A (2004) Cross-language information access on the Web: A tool to help learning foreign languages. In: Cantoni L, McLoughlin C (eds), Proc. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004: 5449–5454
240. Magnini B, Giampiccolo D, Aunimo L, Ayache C, Osenova P, Penas A, de Rijke M, Sacaleanu B, Santos D, Sutcliffe R (2006) The multilingual question answering track at CLEF. In: Proc. 5th International Conference on Language Resources and Evaluation (LREC’2006), Genoa, Italy: 1156–1163
241. Marlow J, Clough P, Dance K. (2007) Multilingual needs of cultural heritage website visitors: A case study of Tate Online, In: Trant J, Bearman D (eds). Proc. International Cultural Heritage Informatics Meeting (ICHIM07) http://www.archimuse.com/ichim07/papers/marlow/marlow.html. Cited 15 Apr 2011
242. Meng W, Yu CT, Liu K (2002) Building efficient and effective metasearch engines. ACM Computing Surveys 34(1): 48–89
243. Min J, Jiang J, Leveling J, Jones GJF, Way A, (2010) DCU’s experiments for the NTCIR-8 IR4QA Task. In: Proc. NTCIR-8 Workshop. National Institute of Informatics, Tokyo. http://research. nii.ac.jp/ntcir/workshop/OnlineProceedings8/NTCIR/08-NTCIR8-IR4QA-MinJ.pdf Cited 15 Apr 2011
244. Mitamura T, Shima H, Sakai T, Kando N, Mori T, Takeda K, Lin C-Y, Song R, Lin C-J, Lee C-W (2010) Overview of the NTCIR-8 ACLIA tasks: advanced cross-lingual information access. In: Proc. NTCIR-8 Workshop. National Institute of Informatics, Tokyo. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings8/NTCIR/01-NTCIR8-OV-CLQA-MitamuraT.pdf. Cited 15 Apr 2011
245. Montiel-Ponsoda E, Aguado-de-Cea G, Gómez-Pérez A, Peters W (2010) Enriching ontologies with multilingual information. J. Nat. Lang. Eng. Available on Cambridge Journals Online, 09 June 2010
246. Müller H, Kalpathy-Cramer J (2010) The medical image retrieval task. In: ImageCLEF: Experimental Evaluation in Information Retrieval. The Information Retrieval Series. Springer: 239–257
247. Müller H, Clough P, Deselaers T, Caputo B (eds) (2010) ImageCLEF-Experimental evaluation of visual information retrieval. The Information Retrieval Series. Springer. http://dx.doi.org/10.1007/978-3-642-15181-1
248. Noguera E, Llopis F, Ferrandez A, Escapa A (2007) Evaluation of open-domain question answering systems within a time constraint. In: Proc. 21st International Conference on Advanced Information Networking and Applications Workshops, AINAW ’07, Niagara Falls, Ontario: 260–265
249. Nomoto M, Fukushige Y, Sato M, Suzuki H, (2004) Are we making progress?-An analysis of NTCIR QAC1 and 2. In: Proc. NTCIR-4 Workshop, National Institute of Informatics, Tokyo. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings4/. Cited 15 Apr 2011
250. O’Leary DE (2008) A multilingual knowledge management system: A case study of FAO and WAICENT. Decis. Support Syst. 45(3): 641–661
251. Oard D (1997) Serving users in many languages: Cross-language information retrieval for digital libraries. D-Lib Magazine Dec.1997. http://www.dlib.org/dlib/december97/oard/12oard.html. Cited 15 Apr 2011
252. Ordelman R, de Jong F, Heeren W (2006) Exploration of audiovisual heritage using audio indexing technology. In: Proc. 1st ECAI Workshop on Intelligent Technologies for Cultural Heritage Exploitation: 36–39
253. Pavani AMB (2001) A model of multilingual digital library. Ciência da Informação, Brasília 30 (3): 73–81
254. Pecina P, Hoffmannová P, Jones GJF, Zhang Y, Oard DW (2008) Overview of the CLEF-2007 cross-language speech retrieval track. In: Proc. CLEF 2007. Advances in Multilingual and Multimodal Information retrieval, Springer LNCS 5152: 674–686
255. Peruginelli G (2008) Multilingual legal information access: an overview. In: Chiocchetti E, Voltmer L, (eds), Harmonising Legal Terminology, Bolzano: EURAC, 2007, ISBN: 9788888906393: 6–34
256. Peters W, Sagri MT, Tiscornia D (2007) The structuring of legal knowledge in LOIS. Artif. Intell. and Law 15(2): 117–135
257. Peterson T, (2002) The importance of being multilingual. Business Week September 4 2002. http://www.businessweek.com/bwdaily/dnflash/sep2002/nf2002094_2752.htm. Cited 15 Apr 2011
258. Pingali P, Jagarlamudi J, Varma V (2006) WebKhoj: Indian language IR from multiple character encodings. In: Proc. World Wide Web Conference Series–WWW 2006: 801–809
259. Potthast P, Barrón-Cedeño A, Stein B, Rosso P (2011) Cross-language plagiarism detection. Lang. Resour. and Eval. 45(1): 45–62
260. Rasmussen EM (1997) Indexing images. Annu. Rev. Inf. Sci. and Technol. 32: 169–196
261. Rautiainen M, Seppänen T, Ojala T (2006) Advancing content-based retrieval effectiveness with cluster-temporal browsing in multilingual video databases. In: Proc. International Conference on Multimedia Computing and Systems/International Conference on Multimedia and Expo: 377–380
262. Reid, NH (1999) Photographic archives: Aberdeen, Dundee and St Andrews. In: Coppock T (ed) Making information available in digital format: perspectives from practitioners, The Stationery Office, Edinburgh: 106–119
263. Roda G, Tait J, Piroi F, Zenz V (2010) CLEF-IP 2009: retrieval experiments in the intellectual property domain. In: Proc. CLEF 2009. Multilingual Information Access Evaluation I. Text Retrieval Experiments, Springer, LNCS 6241: 385–409
264. Rodrigo A, Peñas A, Verdejo F (2009) Overview of the answer validation exercise 2008. In: Proc. CLEF 2008. Evaluating Systems for Multilingual and Multi-modal Information Accessl, Springer LNCS 5706: 296–313
265. Rydberg-Cox JA, (2005) The cultural heritage language technologies consortium. D-Lib Magazine 11(5). http://www.dlib.org/dlib/may05/rydberg-cox/05rydberg-cox.html. Cited 15 Apr 2011
266. Sagri MT, Tiscornia D (2004) Semantic lexicons for accessing legal information. In: Proc. EGOV2004, LNCS 3183, Springer: 72–81
267. Saha S, Fernandez A (2007) Language barriers in health care. J. Gen. Intern. Medicine 22: 281–282
268. Sanderson M, Clough P (2002) Eurovision-an image-based CLIR system. In: Proc. Workshop of 25th ACM SIGIR Conference on Research and Development in Information Retrieval, Workshop 1: Cross-Language Information Retrieval: A Research Roadmap: 56–59
269. Santos D, Cabral LM (2010) GikiCLEF: expectations and lessons learned. In: Proc. CLEF 2009. Multilingual Information Access Evaluation I: Text Retrieval Experiments, LNCS 6241, Springer: 212–222
270. Sasaki Y, Lin C-J, Chen K-H, Chen H-H (2007) Overview of the NTCIR-6 cross-lingual question answering (CLQA) task. In: Proc. NTCIR-6 Workshop, National Institute of Informatics, Tokyo. http://research.nii.ac.jp/ntcir/ntcir-ws6/OnlineProceedings/NTCIR/72-revised-20070604.pdf. Cited 15 Apr 2011
271. Savino P, Peters C (2004) ECHO: a digital library for historical film archives. Int. J. on Digit. Libr. 4(1): 3–7
272. Sheridan P, Weschler M, Schäuble P (1997a) Cross-language speech retrieval: establishing a baseline performance. In: Proc. 20th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’97): 99–108
273. Sheridan P, Braschler M, Schäuble P (1997b) Cross-language information retrieval in a multilingual legal domain. In: Peters C, Thanos C (eds) Proc. First European Conference on Research and Advanced Technology for Digital Libraries (ECDL ’97), Springer: 253–268
274. Smeaton A, Over P, Kraij W (2009) High-Level Feature Detection from Video in TRECVid: A 5-Year Retrospective of Achievements. In: A. Divakaran (ed), Multimedia Content Analysis, Signals and Communication Technology, Springer Science + Business Media: 151–174
275. Smeulders A, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. Pattern Analysis and Mach. Intell. 22(12): 1349–1380
276. Steinberger R, Pouliquen B, Widiger A, Ignat C, Erjavec T, Tufis¸ D, Varga D (2006) The JRCAcquis: A multilingual aligned parallel corpus with 20+ languages. In: Proc. 5th International Conference on Language Resources and Evaluation (LREC’2006). Genoa, Italy, 24–26 May 2006
277. Tran TT (2009) Understanding cultural barriers in hepatitis B virus infection. Cleveland Clinic J. of Medicine 73(3): 10–13
278. Turmo J, Comas PR, Rosset S, Lamel L, Moreau, Mostefa N (2009) Overview of QAST 2008. In: Proc. CLEF 2008. Evaluating Systems for Multilingual and Multi-modal Information Access, Springer LNCS 5706: 314–324
279. Udupa R, Khapra M (2010) Improving the multilingual user experience of Wikipedia using crosslanguage name search. In: Proc. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT ’10): 492–500
280. Voorhees EM (2001) The TREC question answering track. Nat. Lang. Eng. 7(4): 361–378.
281. Voorhees EM, Tice D (2000) Building a question answering test collection. In Proc. 23rd ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR 2000: 200–207
282. Witten IH, Bainbridge D (2003) How to build a digital library. San Francisco, CA: Morgan Kaufman Publishers
283. Wu D, Luo B, He D (2010) How multilingual digital information is used: A study in Chinese academic libraries. In: Proc. International Conference on Management and Service Science (MASS 2010): 1–4
284. Zhang J, Lin S (2007) Multiple language supports in search engines. Online Inf. Review 31(4): 516–532