A Comparative Study of Ontology Matching Systems via Inferential Statistics

IEEE Transactions on Knowledge and Data Engineering

Volumn 31, Issue 4, 2019, Pages 615-628

  Mohammadi, Majid ^a   Hofman, Wout ^b   Tan, Yao Hua ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b TNO   (Netherlands)

Author keywords

Bergmann; Friedman; Holm; McNemar; Nemenyi; Ontology alignment evaluation; paired t test; post hoc; Quade; Shaffer; Wilcoxon signed rank

Indexed keywords

JOB ANALYSIS; ROBUSTNESS (CONTROL SYSTEMS); STATISTICAL METHODS; STATISTICS;

BENCHMARK TESTING; BERGMANN; FRIEDMAN; GEOSCIENCES; HOLM; MCNEMAR; NEMENYI; ONTOLOGY ALIGNMENT; POST-HOC; QUADE; SHAFFER; SIGNED RANK; T-TESTS; TASK ANALYSIS;

ONTOLOGY;

EID: 85047821778     PISSN: 10414347     EISSN: 15582191     Source Type: Journal    
DOI: 10.1109/TKDE.2018.2842019     Document Type: Article

References (46)

1. A. Isaac, S. Wang, C. Zinn, H. Matthezing, L. Van der Meij, and S. Schlobach, "Evaluating thesaurus alignments for semantic interoperability in the library domain, " IEEE Intell. Syst., vol. 24, no. 2, pp. 76-86, Mar./Apr. 2009.
2. P. P. Talukdar, Z. G. Ives, and F. Pereira, "Automatically incorporating new sources in keyword search-based data integration, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2010, pp. 387-398.
3. M. Ba and G. Diallo, "Large-scale biomedical ontology matching with ServOMap, " Innovation Res. BioMed. Eng., vol. 34, no. 1, pp. 56-59, 2013.
4. U. Bharambe, S. S. Durbha, and R. L. King, "Geospatial ontologies matching: An information theoretic approach, " in Proc. IEEE Int. Geosci. Remote Sens. Symp., 2012, pp. 2918-2921.
5. D. H. Wolpert and W. G. Macready, "No free lunch theorems for optimization, " IEEE Trans. Evol. Comput., vol. 1, no. 1, pp. 67-82, Apr. 1997.
6. D. H. Wolpert, "What the no free lunch theorems really mean; how to improve search algorithms, " in Santa Fe Inst. Work. Paper, 2012, Art. no. 12.
7. J. Demsar, "Statistical comparisons of classifiers over multiple data sets, " J. Mach. Learn. Res., vol. 7, no. Jan, pp. 1-30, 2006.
8. S. Garcia and F. Herrera, "An extension on "statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons, " J. Mach. Learn. Res., vol. 9, no. Dec, pp. 2677-2694, 2008.
9. S. García, A. Fernández, J. Luengo, and F. Herrera, "Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power, " Inf. Sci., vol. 180, no. 10, pp. 2044-2064, 2010.
10. D. A. Hull, "Information retrieval using statistical classification, " PhD thesis, Dept. Statistics, Citeseer, 1994.
11. T. G. Dietterich, "Approximate statistical tests for comparing supervised classification learning algorithms, " Neural Comput., vol. 10, no. 7, pp. 1895-1923, 1998.
12. B. T. Nski, M. S. Etek, Z. Telec, and T. Lasota, "Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms, " Int. J. Appl. Math. Comput. Sci, vol. 22, no. 4, pp. 867-881, 2012.
13. J. Derrac, S. García, D. Molina, and F. Herrera, "A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms, " Swarm Evol. Comput., vol. 1, no. 1, pp. 3-18, 2011.
14. J. Euzenat, P. Shvaiko et al., Ontology Matching, vol. 18. Berlin, Germany: Springer, 2007.
15. M. Mohammadi, A. A. Atashin, W. Hofman, and Y. Tan, "Comparison of ontology alignment algorithms across single matching task via the McNemar test, " ACM Trans. Knowl. Discovery Data, vol. 12, no. 4, Jun. 2018.
16. J. Euzenat, M.-E. Roşoiu, and C. Trojahn, "Ontology matching benchmarks: Generation, stability, and discriminability, " Web Semantics: Sci. Serv. Agents World Wide Web, vol. 21, pp. 30-48, 2013.
17. C. Meilicke, R. García-Castro, F. Freitas, W. R. Van Hage, E. Montiel-Ponsoda, R. R. De Azevedo, H. Stuckenschmidt, O.Svábzamazal, V. Svátek, A. Tamilin et al., "MultiFarm: A benchmark for multilingual ontologymatching, " Web Semantics: Sci. Services Agents World WideWeb, vol. 15, pp. 62-68, 2012.
18. F. Wilcoxon, "Individual comparisons by ranking methods, " Biometrics Bulletin, vol. 1, no. 6, pp. 80-83, 1945.
19. Q. McNemar, "Note on the sampling error of the difference between correlated proportions or percentages, " Psychometrika, vol. 12, no. 2, pp. 153-157, 1947.
20. D. J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures. Boca Raton, FL, USA: CRC Press, 2003.
21. M. Friedman, "The use of ranks to avoid the assumption of normality implicit in the analysis of variance, " J. Amer. Statistical Assoc., vol. 32, no. 200, pp. 675-701, 1937.
22. D. Quade, "Using weighted rankings in the analysis of complete blocks with additive block effects, " J. Amer. Statistical Assoc., vol. 74, no. 367, pp. 680-683, 1979.
23. C. M. Jarque and A. K. Bera, "Efficient tests for normality, homoscedasticity and serial independence of regression residuals, " Econ. Lett., vol. 6, no. 3, pp. 255-259, 1980.
24. J. Euzenat, A. Ferrara, L. Hollink, A. Isaac, C. Joslyn, V. Malaisé, C. Meilicke, A. Nikolov, J. Pane, M. Sabou et al., "Results of the ontology alignment evaluation initiative 2009, " in Proc. 4th Int. Conf. Ontology Matching, 2009, pp. 73-126.
25. W. Li and Q. Sun, "GMap: Results for OAEI 2015, " Ontology Matching, vol. 1, 2015, Art. no. 150.
26. A. Benavoli, G. Corani, F. Mangili, M. Zaffalon, and F. Ruggeri, "A Bayesian Wilcoxon signed-rank test based on the Dirichlet process, " in Proc. 31st Int. Conf. Int. Conf.Mach. Learn., 2014, pp. 1026-1034.
27. M. W. Fagerland, S. Lydersen, and P. Laake, "The McNemar test for binary matched-pairs data: Mid-p and asymptotic are better than exact conditional, " BMC Med. Res. Methodology, vol. 13, no. 1, 2013, Art. no. 1.
28. A. L. Edwards, "Note on the correction for continuity in testing the significance of the difference between correlated proportions, " Psychometrika, vol. 13, no. 3, pp. 185-187, 1948.
29. H. Lancaster, "Significance tests in discrete distributions, " J. Amer. Statistical Assoc., vol. 56, no. 294, pp. 223-234, 1961.
30. J. H. Drew, "Modern data analysis: A first course in applied statistics, " Technometrics, vol. 33, no. 4, pp. 487-488, 1991.
31. J. W. Mauchly, "Significance test for sphericity of a normal n-variate distribution, " Ann. Math. Statist., vol. 11, no. 2, pp. 204-209, 1940.
32. R. L. Iman and J. M. Davenport, "Approximations of the critical region of the fbietkan statistic, " Commun. Statist.-Theory Methods, vol. 9, no. 6, pp. 571-595, 1980.
33. J. Hodges, E. L. Lehmann et al., "Rank methods for combination of independent experiments in analysis of variance, " Ann. Math. Statist., vol. 33, no. 2, pp. 482-497, 1962.
34. E. Jiménez-Ruiz and B. C. Grau, "LogMap: Logic-based and scalable ontology matching, " in Proc. Int. Semantic Web Conf., 2011, pp. 273-288.
35. W. E. Djeddi and M. T. Khadir, "XMAP: A novel structural approach for alignment of OWL-Full ontologies, " in Proc. Int. Conf. Mach. Web Intell., 2010, pp. 368-373.
36. P. Nemenyi, "Distribution-free multiple comparisons, " PhD thesis, Dept. Math., Princeton Univ., Princeton, NJ, USA, 1963.
37. S. Holm, "A simple sequentially rejective multiple test procedure, " Scandinavian J. Statist., vol. 6, pp. 65-70, 1979.
38. J. P. Shaffer, "Modified sequentially rejective multiple test procedures, " J. Amer. Statistical Assoc., vol. 81, no. 395, pp. 826-831, 1986.
39. B. Bergmann and G. Hommel, "Improvements of general multiple test procedures for redundant systems of hypotheses, " in Multiple Hypothesenprüfung/Multiple Hypotheses Testing. Berlin, Germany: Springer, 1988, pp. 100-115.
40. R. R. Bouckaert, "Estimating replicability of classifier learning experiments, " in Proc. 21st Int. Conf. Mach. Learn., 2004, Art. no. 15.
41. Z. Dragisic, K. Eckert, J. Euzenat, D. Faria, A. Ferrara, R. Granada, V. Ivanova, E. Jiménez-Ruiz, A. O. Kempf, P. Lambrix et al., "Results of the ontology alignment evaluation initiative 2014, " in Proc. 9th Int. Conf. Ontology Matching, 2014, pp. 61-104.
42. M. Gulic and B. Vrdoljak, "CroMatcher-results for OAEI 2013, " in Proc. 8th Int. Conf. Ontology Matching, 2013, pp. 117-122.
43. P. Wang and B. Xu, "Lily: Ontology alignment results for OAEI 2008, " in Proc. 3rd Int. Conf. Ontology Matching, 2008, pp. 167-175.
44. C. Meilicke, "MAMBA-results for the OAEI 2015, " in Proc. 10th Int. Workshop Ontology Matching-Ontology Alignment Evaluation Initiative, 2015, Art. no. 181.
45. D. Faria, C. Pesquita, E. Santos, M. Palmonari, I. F. Cruz, and F. M. Couto, "The agreementMakerLight ontology matching system, " in Proc. On Move Meaningful Internet Syst. Confederated Int. Conf., 2013, pp. 527-541.
46. M. El Abdi, H. Souid, M. Kachroudi, and S. B. Yahia, "CLONA results for OAEI, " in Proc. 10th Int. Workshop Ontology Matching-Ontology Alignment Evaluation Initiative, 2015.