Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications

Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications

Volumn , Issue , 2014, Pages 1-298

  Balasubramanian, Vineeth ^a   Ho, Shen Shyang ^b   Vovk, Vladimir ^c  

^a INDIAN INSTITUTE OF TECHNOLOGY HYDERABAD   (India)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84880086948     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/C2012-0-00234-7     Document Type: Book

References (396)

1. Abe N., Mamitsuka H. Query learning strategies using boosting and bagging. Proceedings of the 15th international conference on machine learning (ICML) 1998.
2. Adamskiy D., Nouretdinov I., Mitchell A., Coldham N., Gammerman A. Applying conformal prediction to the bovine TB diagnosing. Proceedings of 12th INNS EANN-SIG international conference and seventh IFIP WG 12.5 international conference 2011, vol. II:449-454.
3. Adler A. Sample images can be independently restored from face recognition templates. IEEE Canadian conference on electrical and computer engineering, 2003 2003, vol. 2:1163-1166.
4. Aggarwal C.C. A framework for change diagnosis of data streams. Proceedings of the ACM SIGMOD international conference on management of data 2003, 575-586. ACM. A.Y. Halevy, Z.G. Ives, A. Doan (Eds.).
5. Alt H. The computational geometry of comparing shapes. Lecture notes in computer science 2009, vol. 5760:235-248. Springer, Berlin. S. Albers, H. Alt, S. Näher (Eds.).
6. Ambroise C., McLachlan G.J. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the national academy of sciences of the USA 2002, vol. 99:6562-6566.
7. Ambroladze A., Parrado-Hernández E., Shawe-Taylor J. Tighter PAC-Bayes bounds. Proceedings of advances in neural information processing systems 2006.
8. Bache K, Lichman M UCI machine learning repository 2013, University of California, Irvine, school of information and computer sciences. http://archive.ics.uci.edu/ml.
9. Auld T., Moore A.W., Gull S. Bayesian neural networks for internet traffic classification. IEEE Trans Neural Netw 2007, 18:223-239.
10. Axelsson S. The base-rate fallacy and the difficulty of intrusion detection. ACM Trans Inf Syst Secur 2000, 3(3):185-205.
11. Balasubramanian V, Chakraborty S, Panchanathan S. Generalized query by transduction for online active learning. In: IEEE international conference on computer vision (ICCV), workshop on online learning for computer vision; 2009.
12. Balasubramanian V.N., Gouripeddi R., Panchanathan S., Vermillion J., Bhaskaran A., Siegel R.M. Support vector machine based conformal predictors for risk of complications following a coronary drug eluting stent procedure. Proceedings of the IEEE conference on computers in cardiology 2009.
13. Baram Y., El-Yaniv R., Luz K. Online choice of active learning algorithms. J Mach Learn Res 2004, 5.
14. Barbará D., Domeniconi C. Detecting outliers using transduction and statistical testing. Proceedings of the 12th annual SIGKDD international conference on knowledge discovery and data mining 2006, 55-64.
15. Barlow H.B. Unsupervised learning. Neural Commun 1989, 1:295-311.
16. Baskiotis N., Sebag M. C4. 5 competence map: a phase transition-inspired approach. Proceedings of the 21st international conference on machine learning 2004, 10. ACM.
17. Basseville M., Nikiforov I.V. Detection of abrupt changes: theory and application 1993, Prentice Hall.
18. Bay S.D., Pazzani M.J. Characterizing model errors and differences. Proceedings of the 17th international conference on machine learning 2000, 49-56. Morgan Kaufmann, San Francisco, CA.
19. Bellotti T., Luo Z., Gammerman A. Reliable classification of childhood acute leukaemia from gene expression data using confidence machines. Proceedings of IEEE international conference on granular computing 2006, 148-153.
20. Bellotti T., Luo Z., Gammerman A., Van Delft F.W., Saha V. Qualified predictions for microarray and proteomics pattern diagnostics with confidence machines. Int J Neural Syst 2005, 15:247-258.
21. Bellotti T. Confidence machines for microarray classification and feature selection [Ph.D. thesis]. Royal Holloway University of London; 2005.
22. Bellotti T., Crook J. Support vector machines for credit scoring and discovery of significant features. Expert Syst Appl 2009, 36:3302-3308.
23. Bellotti T., Luo Z., Gammerman A. Strangeness minimisation feature selection with confidence machine. Intelligent data engineering and automated learning IDEAL 2006, 978-985.
24. Bellotti T., Luo Z., Gammerman A., van Delft F.W., Saha V. Qualified predictions for microarray and proteomics pattern diagnostics with confidence machines. Int J Neural Syst 2005, 15:1-12.
25. Bender R., Berg G., Zeeb H. Tutorial: using confidence curves in medical research. Biomed J 2005, 47:237-247.
26. Benjamini Y., Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J Roy Stat Soc Ser B 1995, 57:289-300.
27. Birattari M., Bontempi H., Bersini H. Local learning for data analysis. Proceedings of the eighth Belgian-Dutch conference on machine learning, Wageningen, The Netherlands 1998, 55-61.
28. Blok HJ. On the nature of the stock market: simulations and experiments [Ph.D. thesis]. Canada: University of British Columbia; 2000.
29. Blum A., Chawla S. Learning from labeled and unlabeled data using graph mincuts. Proceedings of international conference on machine learning (ICML) 2001.
30. Blum A., Langley P. Selection of relevant features and examples in machine learning. Artif Intell 1997, 97:245-271.
31. Blum A., Mitchell T. Combining labeled and unlabeled data with co-training. Proceedings of the 11th annual conference on computational learning theory, Madison, Wisconsin 1998, 92-100. ACM Press, New York, USA. P. Bartlett, Y. Mansour (Eds.).
32. Boehm B. Software engineering economics 1981, Prentice Hall.
33. Bolle R.M., Connell J.H., Pankanti S., Ratha N.K., Senior A.W. Guide to biometrics 2004, Springer.
34. Bonwell C, Eison J. Active learning: creating excitement in the classroom. ASHE-ERIC higher education report; 1991.
35. Boser BE, Guyon IM, Vapnik VN. A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on computational learning theory. Pittsburgh: ACM; 1992. p. 144-52.
36. Bosnić Z., Kononenko I. Estimation of individual prediction reliability using the local sensitivity analysis. Appl Intell 2007, 29:187-203.
37. Bosnić Z., Kononenko I. Comparison of approaches for estimating reliability of individual regression predictions. Data Knowl Eng 2008, 67:504-516.
38. Bosnić Z., Rodrigues P.P., Kononenko I., Gama J. Correcting streaming predictions of an electricity load forecast system using a prediction reliability estimate. Man Mach Interact 2011, 2:343-350.
39. Bosnić Z., Vračar P., Radović M.D., Devedžić G., Filipović N., Kononenko I. Mining data from hemodynamic simulations for generating prediction and explanation models. IEEE Trans Inf Technol Biomed 2012, 16:248-254.
40. Bosnič Z., Kononenko I. Estimation of individual prediction reliability using the local sensitivity analysis. Appl Intell 2008, 29(3):187-203.
41. Bousquet O., Elisseeff A. Stability and generalization. J Mach Learn Res 2002, 2:499-526.
42. Box G., Jenkins G.M., Reinsel G.C. Time series analysis: forecasting and control 1994, Prentice-Hall. 3rd ed.
43. Breierova L., Choudhari M. An introduction to sensitivity analysis 1996, MIT System Dynamics in Education Project.
44. Breiman L., Friedman J.H., Olshen R.A., Stone C.J. Classification and regression trees 1984, Chapman and Hall.
45. Brinker K. Incorporating diversity in active learning with support vector machines. Proceedings of the international conference on machine learning (ICML) 2003.
46. Campbell C., Cristianini N., Smola A. Query learning with large margin classifiers. Proceedings of the international conference on machine learning (ICML) 2000.
47. Carney J., Cunningham P. Confidence and prediction intervals for neural network ensembles. Proceedings of the international joint conference on neural networks, Washington, USA 1999, 1215-1218.
48. Cauwenberghs G., Poggio T. Incremental support vector machine learning. Advances in neural information processing systems (NIPS) 2000.
49. Champod C., Meuwly D. The inference of identity in forensic speaker recognition. Speech Commun 2000, 31:193-203.
50. Chandola V., Banerjee A., Kumar V. Anomaly detection: a survey. ACM Comput Surv 2009, 41(3):1-58.
51. Chang C.-C., Lin C.-J. LIBSVM: a library for support vector machines; 2001. http://www.csie.ntu.edu.tw/~cjlin/libsvm.
52. Chapelle O., Scholkopf B., Zien A. Semi-supervised learning 2006, MIT Press.
53. Chapelle O., Vapnik V.N. Model selection for support vector machines. Proceedings of advances in neural information processing systems 1999, vol. 12:230-237.
54. Cheng J., Wang K. Active learning for image retrieval with Co-SVM. Pattern recognition 2007.
55. Chu F., Wang Y., Zaniolo C. An adaptive learning approach for noisy data streams. Proceedings of the fourth IEEE international conference on data mining 2004, 351-354. IEEE Computer Society.
56. Chu F., Zaniolo C. Fast and light boosting for adaptive mining of data streams. Lecture notes in computer science 2004, vol. 3056:282-292. Springer. H. Dai, R. Srikant, C. Zhang (Eds.).
57. Cohn D. Neural network exploration using optimal experiment design. Advances in neural information processing systems (NIPS) 1994.
58. Cohn D., Atlas L., Ladner R. Improving generalization with active learning. Mach learn 1994, 15(2):201-221.
59. Cohn D., Ghahramani Z., Jordan M. Active learning with statistical models. J Artif Intell Res 1996, 4:129-145.
60. Corani G., Zaffalon M. Credal model averaging: an extension of bayesian model averaging to imprecise probabilities. Machine learning and knowledge discovery in databases 2008, 257-271. Springer.
61. Cortez P., Rio M., Rocha M., Sousa P. Multiscale internet traffic forecasting using neural networks and time series methods. Expert Syst 2012, 29(2):143-155.
62. Cover T.M., Hart P. Nearest neighbor pattern classification. IEEE Trans Inf Theory 1967, 13:21-27.
63. Cox D.R., Hinkley D.V. Theoretical statistics 1974, Chapman and Hall, London.
64. Dasgupta S., Hsu D. Proceedings of the international conference on machine learning (ICML) 2008.
65. Dashevskiy M. Prediction with performance guarantees [Ph.D. thesis]. Royal Holloway, UK: Department of Computer Science, University of London; 2010.
66. Dashevskiy M., Luo Z. Network traffic demand prediction with confidence. GLOBECOM 2008, 1453-1457.
67. Dashevskiy M., Luo Z. Reliable probabilistic classification and its application to internet traffic. Lecture notes in computer science 2008, vol. 5226:380-388. Springer.
68. Dashevskiy M., Luo Z. Predictions with confidence in applications. The sixth international conference on machine learning and data mining (MLDM 2009) 2009, 775-786.
69. Dashevskiy M., Luo Z. Reliable probabilistic classification of internet traffic. Int J Inf Acquisition 2009, 6(2):133-146.
70. Dashevskiy M., Luo Z. Time series prediction with performance guarantee. IET Commun 2011, 5:1044-1051.
71. Feres de Souza B., de Carvalho A.C.P.L.F., Calvo R., Ishii R.P. Multiclass SVM model selection using particle swarm optimization. Proceedings of the sixth international conference on hybrid intelligent systems 2006.
72. Desharnais JM. Analyse statistique de la productivite des projects de development en informatique a partir de la technique de points de fonction [M.Sc. thesis]. Montreal Universite du Quebec; 1988.
73. Dessimoz D., Champod C. Handbook of biometrics. Linkages between biometrics and forensic science 2008, Springer.
74. Devetyarov D. Confidence and Venn machines and their applications to proteomics [Ph.D. thesis]. Royal Holloway: University of London; 2011.
75. Devetyarov D., Nouretdinov I., Burford B., Camuzeaux S., Gentry-Maharaj A., Tiss A., et al. Conformal predictors in early diagnostics of ovarian and breast cancers. Prog Artif Intell 2012, 1:245-257.
76. Devetyarov D., Nouretdinov I., Gammerman A. Confidence machine and its application to medical diagnosis. BIOCOMP 2009, 448-454.
77. Devroye L., Györfi L., Lugosi G. A probabilistic theory of pattern recognition. Applications of mathematics 1996, vol. 31. Springer, New York.
78. Diaconis P., Freedman D. On the consistency of Bayes estimates (with discussion). Ann Stat 1986, 14:1-67.
79. Diacu F. Is failure to predict a crime? October 2012. New York Times; October 2012. http://www.nytimes.com/2012/10/27/opinion/a-failed-earthquake-prediction-a-crime.html.
80. Diamond G.A., Forester J.S. Analysis of probability as an aid in the clinical diagnosis of coronary artery disease. New Eng J Med 1979, 300:3-50.
81. Diehl C., Cauwenberghs G. SVM incremental learning, adaptation and optimization. International joint conference on neural networks (IJCNN) 2003.
82. Ding C., Peng H. Minimum redundancy feature selection from microarray gene expression data. Proceedings of the computer society bioinformatics conference 2003, IEEE.
83. Domingos P., Pazzani M. Beyond independence: conditions for the optimality of the simple bayesian classifier. Proceedings of the 13th international conference on machine learning 1996, 105-112.
84. Dreger H., Feldmann A., Mai M., Paxson V., Sommer R. Dynamic application-layer protocol analysis for network intrusion detection. Proceedings of the 15th conference on USENIX security symposium, Berkeley, CA, USA 2006, vol. 15. USENIX Association.
85. Duda R.O., Hart P.E., Stork D.G. Pattern classification 2000, Wiley, New York. 2nd ed.
86. Džeroski S., Ženko B. Is combining classifiers with stacking better than selecting the best one?. Mach Learn 2004, 54(3):255-273.
87. Edelman S. Representation is representation of similarity. Behav Brain Sci 1998, 21:449-498.
88. Efroymson M.A. Multiple regression analysis 1960, Wiley.
89. Eklund M., Norinder U., Boyer S., Carlsson L. Application of conformal prediction in QSAR. Artificial intelligence applications and innovations 2012, 166-175. Springer.
90. Elidan G., Ninio M., Friedman N., Schuurmans D. Data perturbation for escaping local maxima in learning. Proceedings of the 18th national conference on artificial intelligence and 14th conference on innovative applications of artificial intelligence, Edmonton, Alberta, Canada 2002, 132-139. AAAI Press.
91. Proceedings of the 13th European conference on machine learning (ECML 2002), Helsinki, Finland, August 19-23. Lecture notes in computer science 2002, vol. 2430. Springer, London. T. Elomaa, H. Mannila, H. Toivonen (Eds.).
92. En-Najjary T., Urvoy-Keller G., Pietrzyk M., Costeux J.-L. Application-based feature selection for internet traffic classification. 22nd international teletraffic congress (ITC) 2010, 1-8.
93. Erman J, Mahanti A, Arlitt M. Traffic classification using clustering algorithms. In: Proceedings of the 2006 SIGCOMM workshop on mining network data; 2006. p. 281-6.
94. Eskin E. Anomaly detection over noisy data using learned probability distributions. Proceedings of the seventh international conference on machine learning 2000, 255-262.
95. Este A., Gringoli F., Salgarelli L. Support vector machines for TCP traffic classification. Comput Netw 2009, 53(14):2476-2490.
96. Fawcett T. An introduction to ROC analysis. Pattern Recogn Lett 2006, 27(8):861-874. [special issue: ROC analysis in pattern recognition].
97. Fedorova V., Gammerman A., Nouretdinov I., Vovk V. Plug-in martingales for testing exchangeability on-line. Proceedings of the 29th international conference on machine learning (ICML) 2012.
98. Ferri C, Hernández-Orallo J. Cautious classifiers. In: Proceedings of the first international workshop on ROC analysis in artificial intelligence (ROCAI-2004); 2004. p. 27-36.
99. Fleuret F. Fast binary feature selection with conditional mutual information. J Mach Learn Res 2004, 5:1531-1555.
100. Frank A, Asuncion A. UCI machine learning repository; 2010.
101. Fraser D.A.S. Sequentially determined statistically equivalent blocks. Ann Math Stat 1951, 22:372-381.
102. Fraser D.A.S. Nonparametric tolerance regions. Ann Math Stat 1953, 24:44-55.
103. Fraser D.A.S., Wormleighton R. Nonparametric estimation IV. Ann Math Stat 1951, 22:294-298.
104. Freund Y., Sebastian Seung H., Shamir E., Tishby N. Selective sampling using the query by committee algorithm. Mach learn 1997, 28(2-3):133-168.
105. Freund Y., Shapire R.E. Experiments with a new boosting algorithm. Proceedings of the 13th international conference on machine learning (ICML) 1996, 148-156.
106. Freund Y., Schapire R.E. A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 1997, 55:119-139.
107. Friedman F.H., Hastie T., Tibshirani R. Additive logistic regression: a statistical view of boosting. Ann Stat 2005, 28:337-407.
108. Friedman J.H. Greedy function approximation: a gradient boosting machine. Ann Stat 2001, 29:1189-1232.
109. Friedman J.H. Stochastic gradient boosting. Comput Stat Data Anal 2002, 38:367-378.
110. Fujii A., Tokunaga T., Inui K., Tanaka H. Selective sampling for example-based word sense disambiguation. Comput Linguist 1998, 24(4):573-597.
111. Furui S. Recent advances in speaker recognition. Pattern Recogn Lett 1997, 18:859-872.
112. Gammerman A., Nouretdinov I., Burford B., Chervonenkis A., Vovk V., Luo Z. Clinical mass spectrometry proteomic diagnosis by conformal predictors. Stat Appl Genet Mol Biol 2008, 7(2).
113. Gammerman A., Vovk V., Burford B., Nouretdinov I., Luo Z., Chervonenkis A., et al. Serum proteomic abnormality predating screen detection of ovarian cancer. Comput J 2009, 52(3):326-333.
114. Gammerman A., Vovk V. Hedging predictions in machine learning: the second Computer Journal lecture. Comput J 2007, 50:151-163.
115. Gammerman A., Vovk V., Vapnik V. Learning by transduction. Proceedings of the 14th conference on uncertainty in artificial intelligence, Madison, WI 1998, 148-155. Morgan Kaufmann, San Francisco, USA. G.F. Cooper, S. Moral (Eds.).
116. Gelman A., Carlin J.B., Stern H.S., Rubin D.B. Bayesian data analysis 2003, Chapman and Hall/CRC.
117. Giacinto G., Roli F. Dynamic classifier selection based on multiple classifier behaviour. Pattern Recogn 2001, 34:1879-1881.
118. Gibbs A.L., Su F.E. On choosing and bounding probability metrics. Int Stat Rev 2002, 70(3):419-435.
119. Girshik M.A., Rubin H. A Bayes approach to a quality control model. Ann Math Stat 1952, 23(1):114-125.
120. Gold C., Sollich P. Model selection for support vector machine classification. Neurocomputing 2003, 55:221-249.
121. Gonzalez S., Vega J., Murari A., Pereira A., Dormido-Canto S., Ramirez J.M. Automatic location of l/h transition times for physical studies with a large statistical basis. Plasma Phys Control Fusion 2012, 54(6):065009. JET-EFDA contributors.
122. Gonzalez S., Vega J., Pereira A., Pastor I. Region selection and image classification methodology using a non-conformity measure. Prog Artif Intell 2012, 1(3):215-222.
123. Gonzalez-Rodriguez J., Rose P., Ramos D., Toledano D.T., Ortega-Garcia J. Emulating DNA: rigorous quantification of evidential weight in transparent and testable forensic speaker recognition. IEEE Trans Audio Speech Lang Process 2007, 15:2104-2115.
124. Goodrich M., Tamassia R. Data structures and algorithms in JAVA 1998, John Wiley and Sons, Inc.
125. Gopalakrishnan K., Papadopoulos H. Reliable pavement backcalculation with confidence estimation. Sci Iran 2011, 18(6):1214-1221.
126. Granger C.W.J., Joyeux R. An introduction to long-memory time series models and fractional differencing. J Time Ser Anal 1980, 1:15-29.
127. Grother P. Face recognition vendor test (FRVT). Supplemental report NISTIR 7083; 2004.
128. Gu C., Zhang S., Xue X. Encrypted internet traffic classification method based on host behavior. Int J Digit Content Technol Appl 2011, 5(3):167-174.
129. Guo Y. Active instance sampling via matrix partition. Advances of neural information processing systems (NIPS) 2010.
130. Guo Y., Greiner R. Optimistic active learning using mutual information. Proceedings of the international joint conference on artificial intelligence (IJCAI) 2007.
131. Guo Y., Schuurmans D. Discriminative batch mode active learning. Advances of neural information processing systems (NIPS) 2007.
132. Gutta S., Wechsler H. Face recognition using asymmetric faces. First international conference on biometric authentication (ICB) 2004.
133. Guttman I. Statistical tolerance regions: classical and Bayesian 1970, Griffin, London.
134. Guyon I., Elisseeff A. An introduction to variable and feature selection. J Mach Learn Res 2003, 3:1157-1182.
135. Guyon I., Weston J., Barnhill S., Vapnik V. Gene selection for cancer classification using support vector machines. Mach Learn 2002, 46(1-3):389-422.
136. Halck O.M. Using hard classifiers to estimate conditional class probabilities. Proceedings of the 13th European conference on machine learning 2002, 124-134. Springer, Berlin. T. Elomaa, H. Mannila, H. Toivonen (Eds.).
137. Hammer B, Strickert M, Villmann T. Supervised neural gas for learning vector quantization. In: Proceedings of the fifth German workshop on artificial life (GWAL-5), Berlin: Akademische Verlagsgesellschaft-infix-IOS Press; 2002. p. 9-16.
138. Hand D.J. Breast cancer diagnosis from proteomic mass spectrometry data: a comparative evaluation. Stat Appl Genet Mol Biol 2008, 7(2).
139. Hastie T., Rosset S., Tibshirani R., Zhu J. The entire regularization path for the support vector machine. J Mach Learn Res 2004, 5:1391-1415.
140. Hastie T., Tibshirani R., Friedman J. The elements of statistical learning: data mining, inference, and prediction 2009, Springer, New York. 2nd ed.
141. Hastie T., Tibshirani R., Friedman J., Franklin J. The elements of statistical learning: data mining, inference and prediction. Math Intell 2005, 27(2):83-85.
142. Hawkins D. Identification of outliers 1980, Chapman and Hall, London.
143. Heskes T. Practical confidence and prediction intervals. Adv Neural Inf Process Syst 1997, 9:176-182.
144. Ho S.S., Wechsler H. Transductive confidence machine for active learning. Proceedings of the international joint conference on neural networks'03, Portland, OR 2003.
145. Ho S.-S. A martingale framework for concept change detection in time-varying data streams. ICML 2005, 321-327. ACM. L. De Raedt, S. Wrobel (Eds.).
146. Ho S-S. Learning from data streams using transductive inference and martingale [Ph.D. thesis]. George Mason University; 2006.
147. Ho S.-S., Wechsler H. Adaptive support vector machine for time-varying data streams using martingale. IJCAI 2005, 1606-1607. Professional Book Center. L.P. Kaelbling, A. Saffiotti (Eds.).
148. Ho S.-S., Wechsler H. On the detection of concept changes in time-varying data stream by testing exchangeability. Proceedings of the 21st annual conference on uncertainty in artificial intelligence (UAI-05) 2005, 267-274. AUAI Press.
149. Ho S.-S., Wechsler H. Detecting change-points in unlabeled data streams using martingale. 20th international joint conference on artificial intelligence (IJCAI), Hyderabad, India 2007.
150. Ho S.-S., Wechsler H. Query by transduction. IEEE Trans Pattern Anal Mach Intell 2008, 30(9):1557-1571.
151. Ho S.-S., Wechsler H. A martingale framework for detecting changes in data streams by testing exchangeability. IEEE Trans Pattern Anal Mach Intell 2010, 32(12):2113-2127.
152. Hofmann T., Buhmann J. Active data clustering. Advances in neural information processing systems (NIPS) 1998.
153. Hoi Steven C.H., Jin R., Lyu M. Batch mode active learning with applications to text categorization and image retrieval. IEEE Trans Knowl Data Eng 2009, 21(9):1233-1248.
154. Hoi S., Jin R., Zhu J., Lyu M. Batch mode active learning and its application to medical image classification. International conference on machine learning (ICML) 2006.
155. Hoi S.C.H., Jin R., Zhu J., Lyu M.R. Semi-supervised SVM batch mode active learning for image retrieval. IEEE CVPR 2008.
156. Holub A, Perona P, Burl M. Entropy-based active learning for object recognition. In: IEEE conference on computer vision and pattern recognition (CVPR) workshops; 2008.
157. Hüllermeier E. Case-based approximate reasoning 2007, vol. 44. Springer.
158. Hulten G., Spencer L., Domingos P. Mining time-changing data streams. Proceedings of the seventh ACM SIGKDD international conference on knowledge discovery and data mining 2001, 97-106. ACM.
159. Ivina O. Conformal prediction of air pollution concentrations for the barcelona metropolitan region [Ph.D. thesis]. University of Girona; 2012.
160. Ivina O., Nouretdinov I., Gammerman A. Valid predictions with confidence estimation in an air pollution problem. Prog Artif Intell 2012, 1(3):235-243.
161. Jaeger J., Sengupta R., Ruzzo W.L. Improved gene selection for classification of microarrays. Pacific symposium of biocomputing 2003, vol. 8:53-64.
162. Joachims T. Transductive inference for text classification using support vector machines. International conference on machine learning (ICML), Bled, Slowenien 1999, 200-209.
163. John G.H., Langley P. Estimating continuous distributions in Bayesian classifiers. Proceedings of the 11th conference on uncertainty in artificial intelligence 1995, Morgan Kaufmann, San Francisco, USA. P. Besnard, S. Hanks (Eds.).
164. Johnson H, Sadat F, Foster G, Kuhn R, Simard M, Joanis E, et al. Portage: with smoothed phrase tables and segment choice models. In: Proceedings of the workshop on statistical machine translation. Association for Computational Linguistics; 2006. p. 134-7.
165. Joshi A., Porikli F., Papanikolopoulos N. Breaking the interactive bottleneck in multi-class classification with active selection and binary feedback. IEEE conference on computer vision and pattern recognition (CVPR) 2010.
166. Joshi A., Porikli F., Papanikolopoulos N. Multi-class batch-mode active learning for image classification. International conference on robotics and automation (ICRA) 2010.
167. Jost L. Combining significance levels from multiple experiments or analyses. and ; 2009. http://www.physics/statsarticlesindex.htm.
168. Kääriäinen M. Sinuhe: statistical machine translation using a globally trained conditional exponential family translation model. Proceedings of the 2009 conference on empirical methods in natural language processing 2009, vol. 2:1027-1036. Association for Computational Linguistics.
169. Kapoor A, Hua G, Akbarzadeh A, Baker S. Which faces to tag: adding prior constraints into active learning. In: IEEE international conference on computer vision (ICCV) workshops; 2009.
170. Karagiannis T., Papagiannaki K., Faloutsos M. BLINC: multilevel traffic classification in the dark. Proceedings of ACM SIGCOMM 2005, 229-240.
171. Kearns M., Ron D. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Comput 1999, 11(6):1427-1453.
172. Kemperman J.H.B. Generalized tolerance limits. Ann Math Stat 1956, 27:180-186.
173. Keogh E., Lin J., Fu A. HOT SAX: efficiently finding the most unusual time series subsequence. Proceedings of the fifth IEEE international conference on data mining 2005, 226-233.
174. Kifer D., Ben-David S., Gehrke J. Detecting change in data streams. Proceedings of the 13th international conference on very large data bases 2004, 180-191. Morgan Kaufmann. M.A. Nascimento, M.T. Özsu, D. Kossmann, R.J. Miller, J.A. Blakeley, K. Bernhard Schiefer (Eds.).
175. Kim H., Claffy K.C., Fomenkov M., Barman D., Faloutsos M., Lee K. Internet traffic classification demystified: myths, caveats, and the best practices. Proceedings of the 2008 ACM CoNEXT conference, New York, USA 2008, ACM.
176. Kleijnen J. Experimental designs for sensitivity analysis of simulation models. Eurosim 2001 conference 2001.
177. Klinkenberg R. Learning drifting concepts: examples selection vs example weighting. Intell Data Anal 2004, 8(3):281-300. [special issue on incremental learning systems capable of dealing with concept drift].
178. Klinkenberg R., Joachims T. Detecting concept drift with support vector machines. Proceedings of the 17th international conference on machine learning 2000, 487-494. Morgan Kaufmann. P. Langley (Ed.).
179. Kolter J.Z., Maloof M.A. Dynamic weighted majority: a new ensemble method for tracking concept drift. ICDM 2003, 123-130. IEEE Computer Society.
180. Kononenko I., Šimec E., Robnik-Šikonja M. Overcoming the myopia of inductive learning algorithms with RELIEFF. Appl Intell 1997, 7:39-55.
181. Kononenko I. Semi-naive Bayesian classifier. Proceedings of the European working session on learning-91, Porto, Portugal 1991, 206-219. Springer-Verlag, Berlin-Heidelberg-New York. Y. Kodratoff (Ed.).
182. Kothari R., Jain V. Learning from labeled and unlabeled data using a minimal number of queries. IEEE Trans Neural Netw 2003.
183. Krishnamurthy B., Sen S., Zhang Y., Chen Y. Sketch-based change detection: methods, evaluation, and applications. Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement 2003, 234-247. ACM.
184. Kukar M. Transductive reliability estimation for medical diagnosis. Artif Intell Med 2003, 29:81-106.
185. Kukar M. Quality assessment of individual classifications in machine learning and data mining. Knowledge and information systems 2006, 9(3):364-384.
186. Kukar M., Groselj C. Transductive machine learning for reliable medical diagnostics. J Med Syst 2005, 29(1):13-32.
187. Kukar M., Kononenko I. Reliable classifications with machine learning. Proceedings of 13th European conference on machine learning, ECML 2002 2002, 219-231. Springer-Verlag, Berlin. T. Elomaa, H. Mannila, H. Toivonen (Eds.).
188. Lachiche N., Flach P. Improving accuracy and cost of two-class and multi-class probabilistic classifiers using ROC curves. Proceedings of the 20th international conference on machine learning (ICML 2003) 2003, 416-423.
189. Lambrou A., Papadopoulos H., Gammerman A. Evolutionary conformal prediction for breast cancer diagnosis. Information technology and applications in biomedicine 2009.
190. Lambrou A., Papadopoulos H., Gammerman A. Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE Trans Inf Technol Biomed 2011, 15(1):93-99.
191. Lambrou A., Papadopoulos H., Kyriacou E., Pattichis C.S., Pattichis M.S., Gammerman A., et al. Assessment of stroke risk based on morphological ultrasound image analysis with conformal prediction. Proceedings of the sixth IFIP international conference on artificial intelligence applications and innovations, IFIP AICT 339 2010, 146-153. Springer.
192. Lambrou A., Papadopoulos H., Kyriacou E., Pattichis M.S., Gammerman A., Nicolaides A. Evaluation of the risk of stroke with confidence predictions based on ultrasound carotid image analysis. Int J Artif Intell Tool 2012, 21(4).
193. Lambrou A, Papadopoulos H, Nouretdinov I, Gammerman A. Reliable probability estimates based on support vector machines for large multiclass datasets. In: Iliadis L, Maglogiannis I, Papadopoulos H, Karatzas K, Sioutas S. editors. Proceedings of the AIAI 2012 workshop on conformal prediction and its applications. IFIP advances in information and communication technology, vol. 382. Berlin: Springer; 2012. p. 182-91.
194. Langford J. Tutorial on practical prediction theory for classification. J Mach Learn Res 2005, 6:273-306.
195. Laxhammar R. Conformal anomaly detection: Detecting abnormal trajectories in surveillance applications [Ph.D. thesis]. University of Skövde; 2014.
196. Laxhammar R, Falkman G. Conformal prediction for distribution-independent anomaly detection in streaming vessel data. In: Proceedings of the first international workshop on novel data stream pattern mining techniques. ACM; 2010. p. 47-55.
197. Laxhammar R., Falkman G. Sequential conformal anomaly detection in trajectories based on Hausdorff distance. Proceedings of the 14th international conference on information fusion 2011, 153-160.
198. Laxhammar R, Falkman G Inductive Conformal Anomaly Detection for Sequential Detection of Anomalous Sub-Trajectories. In Annals of Mathematics and Artificial Intelligence: Special Issue on Conformal Prediction and its Applications 2013, September 2013.
199. Laxhammar R, Falkman G. Online Learning and Sequential Anomaly Detection in Trajectories. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 99, (2013) PrePrints, September 2013.
200. Lei Jing, Robins James, Wasserman Larry Distribution free prediction sets. Journal of the American Statistical Association 2013, 108:278-287. Preliminary version published as Technical Report arXiv:1111.1418 [math.ST].
201. Lei Jing, Wasserman Larry. Distribution free prediction bands. Technical Report [stat.ME], arXiv.org e-Print archive, March 2012. Published in the Journal of the Royal Statistical Society B, 2014;76:71-96. arxiv:1203.5422.
202. Leland W.E., Taqqu M.S., Willinger W., Wilson D.V. On the self-similar nature of ethernet traffic (extended version). IEEE/ACM Trans Netw 1994, 2(1):1-15.
203. Lewis D., Gale W. A sequential algorithm for training text classifiers. Proceedings of the ACM SIGIR conference 1994.
204. Li F., Wechsler H. Open set face recognition using transduction. IEEE Trans Pattern Anal Mach Intell 2005, 27:1686-1697.
205. Li F., Wechsler H. Face authentication using recognition-by-parts, boosting and transduction. Int J Artif Intell Pattern Recogn 2009, 23:545-573.
206. Li H., Wang S., Qi F. SVM model selection with the VC bound. Comput Inf Sci 2005, 3314:1067-1071.
207. Li M., Sethi Ishwar K. Confidence-based active learning. IEEE Trans Pattern Anal Mach Intell 2006, 28(8):1251-1261.
208. Li M., Vitányi P. An introduction to Kolmogorov complexity and its applications 1997, Springer-Verlag, New York. 2nd ed.
209. Li W, Abdin K, Dann R, Moore AW. Approaching real-time network traffic classification. Technical report RR-06-12. Department of Computer Science, Queen Mary, University of London; 2006.
210. Li W., Canini M., Moore A.W., Bolla R. Efficient application identification and the temporal and spatial stability of classification schema. Comput Netw 2009, 53(6):790-809.
211. Li Y., Fang B., Guo L., Chen Y. Network anomaly detection based on TCM-KNN algorithm. Proceedings of the second ACM symposium on information, computer and communications security, ASIACCS '07, New York, NY, USA 2007, 13-19. ACM.
212. Li Y., Guo L. An active learning based TCM-KNN algorithm for supervised network intrusion detection. Comput Secur 2007, 26(7-8):459-467.
213. Li Z., Yuan R., Guan X. Accurate classification of the internet traffic based on the SVM method. Proceedings of IEEE international conference on communications, ICC 2007, 1373-1378.
214. Liere R., Tadepalli P. Active learning with committees for text categorization. Proceedings of the 14th national conference on artificial intelligence 1997.
215. Lindenbaum M., Markovitch S., Rusakov D. Selective sampling for nearest neighbor classifiers. Mach Learn 2004, 54(2):125-152.
216. Lomasky R., Brodley C., Aernecke M., Walt D., Friedl M. Active class selection. Proceedings of the European conference on machine learning (ECML) 2007.
217. Loughin T.M. A systematic comparison of methods for combining p-values from independent tests. Comput Stat Data Anal 2004, 47(3):467-485.
218. MacKay David J.C. Information-based objective functions for active data selection. Neural Comput 1992, 4(4):590-604.
219. Madhukar A., Williamson C. A longitudinal study of p2p traffic classification. Proceedings of the 14th IEEE international symposium on modeling, analysis, and simulation (MASCOTS'06) 2006, 179-188.
220. Mak M.W., Zhang W.D., He M.X. A new two-stage scoring normalization approach for speaker verification. International symposium on multimedia, video and speech processing 2001.
221. Makili L, Vega J, Dormido-Canto S. Active learning using conformal predictors: application to image classification. In: Seventh workshop on fusion data processing validation and analysis; 2012.
222. Makili L., Vega J., Dormido-Canto S. Incremental support vector machines for fast reliable image recognition. Fusion Eng Des 2013, 88(6):1170-1173.
223. Makili L., Vega J., Dormido-Canto S., Pastor I., Murari A. Computationally efficient {SVM} multi-class image recognition with confidence measures. Fusion Eng Des 2011, 86(6-8):1213-1216. Proceedings of the 26th symposium of fusion technology (SOFT-26).
224. McCallum A., Nigam K. Employing EM and pool-based active learning for text classification. Proceedings of the international conference on machine learning (ICML) 1998.
225. McCullagh P., Vovk V., Nouretdinov I., Devetyarov D., Gammerman A. Conditional prediction intervals for linear regression. Proceedings of the eighth international conference on machine learning and applications, Miami, FL, December 13-15 2009, 131-138. http://www.stat.uchicago.edu/~pmcc/reports/predict.pdf.
226. Melluish T., Saunders C., Nouretdinov I., Vovk V. Comparing the Bayes and typicalness frameworks. Lecture notes in computer science 2001, vol. 2167:360-371. Springer, Heidelberg. L. De Raedt, P.A. Flach (Eds.).
227. Melville P., Mooney R. Diverse ensembles for active learning. Proceedings of the international conference on machine learning (ICML) 2004.
228. Melville P., Yang S., Saar-Tsechansky M., Mooney R. Active learning for probability estimation using Jensen-Shannon divergence. European conference on machine learning (ECML) 2005.
229. Meyn S., Negrete-Pincetic M., Wang G., Kowli A., Shafieepoorfard E. The value of volatile resources in electricity markets. 2010 49th IEEE conference on decision and control (CDC) 2010, 1029-1036.
230. Miettinen O.S. Theoretical epidemiology: principles of occurrence research in medicine 1985, Wiley, New York.
231. Mitchell T.M. Machine learning 1997, McGraw-Hill, Boston, MA.
232. Mitra P., Murthy C.A., Pal Sankar K. A probabilistic active support vector learning algorithm. IEEE Trans Pattern Anal Mach Intell 2004, 26(3):413-418.
233. Momma M., Bennett K.P. Pattern search method for model selection of support vector regression. Proceedings of the second SIAM international conference on data mining 2002.
234. Monteleoni C., Kaariainen M. Practical online active learning for classification. IEEE conference on computer vision and pattern recognition (CVPR) 2007.
235. Moore AW., Discrete content-based classification - a data set. Technical report. Cambridge: Intel Research; 2005.
236. Moore AW, Papagiannaki D. Toward the accurate identification of network applications. In: Proceedings of the sixth passive and active measurement workshop; 2005. p. 50-60.
237. Moore A.W., Zuev D. Internet traffic classification using bayesian analysis techniques. Proceedings of the 2005 ACM SIGMETRICS, New York, USA 2005, 50-60. ACM.
238. Moore AW, Zuev D, Crogan M. Discriminators for use in flow-based classification. Technical report RR-05-13. Department of Computer Science, Queen Mary, University of London; 2005.
239. Morris B.T., Trivedi M.M. A survey of vision-based trajectory learning and analysis for surveillance. IEEE Trans Circuits Syst Video Technol 2008, 18(8):1114-1127.
240. Musicant D.R. Normally distributed clustered datasets 1998, Computer Sciences Department, University of Wisconsin, Madison. http://www.cs.wisc.edu/dmi/svm/ndc/.
241. Nappi M., Wechsler H. Robust re-identification using randomness and statistical learning: Quo Vadis. Pattern Recogn Lett 2012, 33:1820-1827.
242. National Institute of Standards and Technology. Digital library of mathematical functions. ; 6 May 2013. http://dlmf.nist.gov/.
243. Nguyen T.T.T., Armitage G. A survey of techniques for internet traffic classification using machine learning. IEEE Commun Surv Tutor 2008, 10(4):56-76.
244. Nigam K., McCallum A.K., Thrun S., Mitchell T. Text classification from labeled and unlabeled documents using EM. Mach Learn 2000, 39(2-3):103-134.
245. Nischenko I., Jordaan E.M. Confidence of SVM predictions using a strangeness measure. International joint conference on neural networks (IJCNN) 2006, 1239-1246.
246. Nouretdinov I., Burford B., Gammerman A. Application of inductive confidence machine to ICMLA competition data. Proceedings of the eighth international conference on machine learning and applications 2009, 435-438.
247. Nouretdinov I., Costafreda S.G., Gammerman A., Chervonenkis A., Vovk V., Vapnik V. Machine learning classification with confidence: application of transductive conformal predictors to MRI-based diagnostic and prognostic markers in depression. Neuroimage 2011, 56:809-813.
248. Nouretdinov I., Devetyarov D., Vovk V., Burford B., Camuzeaux S., Gentry-Maharaj A., et al. Multiprobabilistic prediction in early medical diagnoses. Ann Math Artif Intell 2013, 1-20. 10.1007/s10472-013-9367-5.
249. Nouretdinov I., Gammerman A., Qi Y., Klein-Seetharaman J. Determining confidence of predicted interactions between HIV-1 and human proteins using conformal method. Pacific symposium on biocomputing 2012, vol. 17:311-322.
250. Nouretdinov I., Li G., Gammerman A., Luo Z. Application of conformal predictors to tea classification based on electronic nose. Proceedings of the sixth conference on artificial intelligence applications and innovations 2010, 303-310.
251. Nouretdinov I., Melluish T., Vovk V. Ridge regression confidence machine. Proceedings of the 18th international conference on machine learning, San Francisco 2001, 385-392. Morgan Kaufmann.
252. Nouretdinov I, Vovk V, V'yugin V, Gammerman A. Transductive confidence machine is universal. Technical report CLRC-TR-02-05. Computer Learning Research Centre, Royal Holloway, University of London; 2002.
253. Nouretdinov I., V'yugin V., Gammerman A. Transductive confidence machine is universal. Lecture notes in artificial intelligence 2003, vol. 2842. Springer, Berlin. R. Gavaldà, K.P. Jantke, E. Takimoto (Eds.).
254. Nouretdinov IR. Offline Nearest Neighbour Transductive Confidence Machine. In: Poster and workshop proceedings of the eighth industrial conference on data mining; 2008. p. 16-24.
255. Olona-Cabases M. The probability of a correct diagnosis. Nuclear cardiology in everyday practice 1994, 348-357. Kluwer, Dordrecht, NL. J. Candell-Riera, D. Ortega-Alcalde (Eds.).
256. Osugi T., Kun D., Scott S. Balancing exploration and exploitation: a new algorithm for active machine learning. IEEE international conference on data mining (ICDM) 2005.
257. Özöǧür S, Shawe-Taylor J, Weber GW, Ögel ZB. Pattern analysis for the prediction of fungal pro-peptide cleavage sites. Discrete Appl Math 2008. [special issue on Netw Comput Biol]. doi:10.1016/j.dam.2008.06.043.
258. Özöǧür-Akyüz S., Hussain Z., Shawe-Taylor J. Prediction with the SVM using test point margins. Data Min 2010, 147-158.
259. Page E.S. On problem in which a change in a parameter occurs at an unknown point. Biometrika 1957, 44:248-252.
260. Papadimitriou S., Brockwell A., Faloutsos C. Adaptive, hands-off stream mining. Proceedings of the 29th international conference on very large data base 2003, 560-571.
261. Papadopoulos H., Gammerman A., Vovk V. Confidence predictions for the diagnosis of acute abdominal pain. Artificial Intelligence Applications and Innovations 2009, vol. III:175-184.
262. Papadopoulos H., Gammerman A., Vovk V. Reliable diagnosis of acute abdominal pain with conformal prediction. Eng Intell Syst 2009, 17:127-137.
263. Papadopoulos H., Haralambous H. Neural networks regression inductive conformal predictor and its application to total electron content prediction. Lecture notes in computer science 2010, vol. 6352:32-41. Springer, Berlin/Heidelberg. K. Diamantaras, W. Duch, L. Iliadis (Eds.).
264. Papadopoulos H., Haralambous H. Reliable prediction intervals with regression neural networks. Neural Netw 2011, 24(8):842-851.
265. Papadopoulos H, Papatheocharous E, Andreou AS. Reliable confidence intervals for software effort estimation. In: Proceedings of the workshops of the fifth IFIP conference on artificial intelligence applications and innovations (AIAI-2009), April 23-25, Thessaloniki, Greece; 2009. p. 211-20.
266. Papadopoulos H., Proedrou K., Vovk V., Gammerman A. Inductive confidence machines for regression. ECML 2002, 345-356.
267. Papadopoulos H., Vovk V., Gammerman A. Regression conformal prediction with nearest neighbours. J Artif Intell Res 2011, 40(1):815-840.
268. Park J., Tyan H.-R., Kuo C.J. Ga-based internet traffic classification technique for QOS provisioning. Proceedings of the 2006 international conference on intelligent information hiding and multimedia (IIH-MSP '06) 2006, 251-254.
269. Paxson V., Floyd S. Wide-area traffic: the failure of poisson modeling. IEEE/ACM Trans Netw 1995, 3:226-244.
270. Davis R.A., Brockwell P.J. Time series: theory and methods 1991, Springer.
271. Petraco N.D.K., Shenkin P., Speir J., Diaczuk P., Pizzola P.A., Gambino C., Petraco N. Addressing the national academy of sciences challenge: a method for statistical pattern comparison of striated tool marks. J Forens Sci 2012, 57(4):900-911.
272. Pfahringer B., Bensuasan H., Giraud-Carrier C. Meta-learning by landmarking various learning algorithms. Proceedings of the 17th international conference on machine learning 2000, Morgan Kaufmann, San Francisco, CA.
273. Phillips P.J. The Good, the bad, and the ugly face challenge problem. Image Vis Comput 2012, 30:177-185.
274. Phillips P.J., Wechsler H., Huang J., Rauss P. The FERET database and evaluation procedure for face recognition algorithms. Image Vis Comput 1998, 16:295-306.
275. Piatetsky-Shapiro G. Data scientists responsibility for predictions, October 2012. KDD Nuggets; October 2012. http://www.kdnuggets.com/polls/2012/responsibility-for-predictions.html.
276. Piciarelli C., Micheloni C., Foresti G.L. Trajectory-based anomalous event detection. IEEE Trans Circuits Syst Video Technol 2008, 18(11):1544-1554.
277. Platt J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 1999.
278. Poggio T., Rifkin R., Mukherjee S., Niyogi P. General conditions for predictivity in learning theory. Nature 2004, 428:419-422.
279. Poggio T., Smale S. The mathematics of learning: dealing with data. Notices of ASM 2003, 537-544.
280. Proedrou K, Nouretdinov I, Vovk V, Gammerman A Transductive confidence machines for pattern recognition. Proceedings of the 13th European conference on machine learning 2002, 381-390. Springer, London, (Lecture Notes in Computer Science). T. Elomaa, H. Mannila, H. Toivonen (Eds.).
281. Qi G., Hua X., Rui Y., Tang J., Zhang H. Two-dimensional active learning for image classification. IEEE conference on computer vision and pattern recognition (CVPR) 2008.
282. Quinlan J.R. C4.5: programs for machine learning 1993, vol. 1. Morgan kaufmann.
283. Reiner A., Yekutieli D., Benjamini Y. Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 2003, 19(3):368-375.
284. Rényi A. Théorie des éléments saillants d'une suite d'observations. Annales Scientifiques de l'Université de Clermont-Ferrand 2, Série Mathématiques 1962, 8:7-13. http://www.numdam.org/.
285. Reynolds D.A., Quatieri T.F., Dunn R.B. Speaker verification using adaptive Gaussian mixture models. Digit Signal Process 2000, 10:19-41.
286. Rigollet P., Vert R. Optimal rates for plug-in estimators of density level sets. Bernoulli 2009, 14:1154-1178.
287. Riveiro M. Visual analytics for maritime anomaly detection [Ph.D. thesis]. Örebo University; 2011.
288. Ross M.E., Mahfouz R., Onciu M., Liu H.-C., Zhou X., Song G., et al. Gene expression profiling of pediatric acute myelogenous leukaemia. Blood 2004, 104(12):3679-3687.
289. Ross M.E., Zhou X., Song G., Shurtleff S.A., Girtman K., Williams W.K., et al. Classification of pediatric acute lymphoblastic leukaemia by gene expression profiling. Blood 2003, 102:2951-2959.
290. Roy N., McCallum A. Toward optimal active learning through sampling estimation of error reduction. Proceedings of the international conference on machine learning (ICML) 2001.
291. Rubinstein Y.D., Hastie T. Discriminative vs. informative learning. Knowledge and data discovery (KDD) 1997, 49-53.
292. Ji G.-R., Dong Z., Wang D.-F., Han P., Xu D.-P. Wind speed conformal prediction in wind farm based on algorithmic randomness theory. 2008 international conference on machine learning and cybernetics 2008, vol. 1:131-135.
293. Rumelhart D.E., McClelland J.L. Parallel distributed processing. Foundations 1986, vol. 1. MIT Press, Cambridge.
294. Ryabko D. Relaxing i.i.d. assumption in online pattern recognition. Technical report CS-TR-03-11. Department of Computer Science, Royal Holloway, University of London; 2003.
295. Saar-Tsechansky M., Melville P., Provost F. Active feature-value acquisition. Manage Sci 2009, 55(4):664-684.
296. Sanderson C. Biometric person recognition: face, speech and fusion June 2008, VDM Verlag.
297. Saunders C., Gammerman A., Vovk V. Transduction with confidence and credibility. Proceedings of the 16th international joint conference on artificial intelligence 1999, vol. 2:722-726. Morgan Kaufmann, San Francisco, CA.
298. Saunders C., Gammerman A., Vovk V. Computationally efficient transductive machines. Lecture notes in computer science 2000, vol. 1968:325-333. Springer, Berlin.
299. Scheffé H., Tukey J.W. Nonparametric estimation I: validation of order statistics. Ann Math Stat 1945, 16:187-192.
300. Scheffer T., Decomain C., Wrobel S. Active hidden Markov models for information extraction. Proceedings of the international conference on advances in intelligent data analysis (CAIDA) 2001.
301. Schleif F.-M., Villmann T., Kostrzewa M., Hammer B., Gammerman A. Cancer informatics by prototype networks in mass spectrometry. Artif Intell Med 2009, 45(2-3):215-228.
302. Schohn G., Cohn D. Less is more: Active learning with support vector machines. Proceedings of the international conference on machine learning (ICML) 2000.
303. Schölkopf B., Smola A. Learning with kernels 2002, MIT Press, Cambridge, MA.
304. Schölkopf B., Williamson R., Smola A., Shawe-Taylor J., Platt J. Support vector method for novelty detection. Advances in neural information processing systems 12 1999, 582-588. Morgan Kaufmann.
305. Sculley D. Online active learning methods for fast label-efficient spam filtering. Fourth conference on email and antispam 2007.
306. Seber G.A.F., Lee A.J. Linear regression analysis 2003, Wiley, Hoboken, NJ. 2nd ed.
307. Seewald A., Furnkranz J. An evaluation of grading classifiers. Proceedings of the fourth international symposium on advances in intelligent data analysis 2001, 115-124.
308. Sen S., Spatscheck O., Wang D. Accurate, scalable in-network identification of p2p traffic using application signatures. Proceedings of the 13th international conference on World Wide Web (WWW'04) 2004, 512-521.
309. Settles B. Active learning literature survey. Technical report 1648. University of Wisconsin-Madison; 2010.
310. Settles B. From theories to queries: active learning in practice. In: JMLR workshop on active learning and experimental design; 2011.
311. Settles B., Craven M. An analysis of active learning strategies for sequence labeling tasks. Proceedings of the conference on empirical methods in natural language processing (EMNLP) 2008.
312. Settles B, Craven M, Friedland L. Active learning with real annotation costs. In: Proceedings of the neural information processing systems (NIPS) workshop on cost-sensitive learning; 2008.
313. Settles B., Craven M., Ray S. Multiple-instance active learning. Advances in neural information processing systems (NIPS) 2008.
314. Shafer G., Vovk V. Probability and finance: it's only a game! 2001, Wiley, New York.
315. Shafer G., Vovk V. A tutorial on conformal prediction. J Mach Learn Res 2008, 9:371-421.
316. Shahmuradov I., Solovyev V., Gammerman A. Plant promoter prediction with confidence estimation. Nucleic Acid Res 2005, 33(3).
317. Shannon C.E. A mathematical theory of communication. ACM SIGMOBILE Mobile Comput Commun Rev 2001, 5(1):3-55.
318. Shawe-Taylor J. Classification accuracy based on observed margin. Algorithmica 1998, 22:57-172.
319. Shawe-Taylor J., Cristianini Nello Kernel methods for pattern analysis 2004, Cambridge University Press, Cambridge, UK.
320. Shewhart W.A. The application of statistics as an aid in maintaining quality of a manufactured products. J Am Stat Assoc 1925, 20:546-548.
321. Shewhart W.A. Economic control of quality of manufactured product. American Society for Quality Control 1931.
322. Shi L, Zhao Y. Batch mode sparse active learning. In: IEEE international conference on data mining (ICDM) workshops; 2010.
323. Shi L., Zhao Y., Tang J. Batch mode active learning for networked data. ACM transactions on embedded computing systems 2011.
324. Shiryaev A.N. On optimum methods in quickest detection problems. Theory Probab Appl 1963, 8:22-46.
325. Simard M., Cancedda N., Cavestro B., Dymetman M., Gaussier E., Goutte C., et al. Translating with non-contiguous phrases. Proceedings of the conference on human language technology and empirical methods in natural language processing 2005, 755-762. Association for Computational Linguistics.
326. Sinha P., Balas B., Ostrovsky Y., Russell R. Face recognition by humans: nineteen results all computer vision researchers should know about. Proc IEEE 2006, 94(11):1948-1962.
327. Smirnov E., Sprinkhuizen-Kuyper I., Nalbantov G., Vanderlooy S. Version space support vector machines. Front Artif Intell Appl 2006, 141:809.
328. Smirnov E.N., Nalbantov G.I., Kaptein A.M. Meta-conformity approach to reliable classification. Intell Data Anal 2009, 13(6):901-915.
329. Smirnov E., Nikolaev N., Nalbantov G. Single-stacking conformity approach to reliable classification. Lecture notes in computer science 2010, vol. 6304:161-170. Springer. D. Dicheva, D. Dochev (Eds.).
330. Smyth P., Gray A., Fayyad U. Retrofitting decision tree classifiers using kernel density estimation. Proceedings of the 12th international conference on machine learning, Tahoe City, California, USA 1995, 506-514. Morgan Kaufmann, San Francisco, USA. A. Prieditis, S.J. Russell (Eds.).
331. Specht D.F., Romsdahl H. Experience with adaptive probabilistic neural networks and adaptive general regression neural networks. Proceedings of IEEE international conference on neural networks, Orlando, USA 1994, IEEE Press, Piscataway, USA. S.K. Rogers (Ed.).
332. Specia L., Saunders C., Turchi M., Wang Z., Shawe-Taylor J. Improving the confidence of machine translation quality estimates. Proceedings of MT summit XII 2009.
333. Steele M. Stochastic calculus and financial applications 2001, Springer Verlag.
334. Stolfo S.J., Fan W., Lee W., Prodromidis A., Chan P.K. Cost-based modeling for fraud and intrusion detection: results from the jam project. Proceedings of the DARPA information survivability conference and exposition, 2000 (DISCEX '00) 2000, vol. 2:130-144.
335. Sugiyama M., Rubens N. Active learning with model selection in linear regression. SIAM data mining conference (SDM) 2008.
336. Šajn L., Kukar M. Image processing and machine learning for fully automated probabilistic evaluation of medical images. Comput Method Prog Biomed 2011, 104:75-86.
337. Štrumbelj E., Bosnić Z., Kononenko I., Zakotnik B., Grašič C. Explanation and reliability of prediction models: the case of breast cancer recurrence. Knowl Inf Syst 2010, 24:305-324.
338. Taneja I.J. On generalized information measures and their applications. Adv Electron Elect Phys 1995, 76:327-416.
339. Nguyen Hieu T., Smeulders A. Active learning using pre-clustering. Proceedings of the twenty-first international conference on machine learning 2004, 79. ACM.
340. Tong S. Active learning: theory and applications [Ph.D. thesis]. Stanford University; 2001.
341. Tong S., Chang E. Support vector machine active learning for image retrieval. Proceedings of the ninth ACM international conference on multimedia 2001.
342. Tong S., Koller D. Support vector machine active learning with applications to text classification. J Mach Learn Res 2002, 2:45-66.
343. Topuz V. Traffic demand prediction using ann simulator. In: Proceedings of the 11th international conference, KES 2007 and XVII Italian workshop on neural networks conference on knowledge-based Intelligent Information and Engineering Systems: Part I; 2007. p. 864-70.
344. Tsuda K., Raetsch M., Mika S., Mueller K. Learning to predict the leave-one-out error of kernel based classifiers. Lecture notes in computer science 2001, 227-331. Springer, Berlin/Heidelberg.
345. Tsybakov A.B. Introduction to nonparametric estimation 2010, Springer, New York.
346. Tukey J.W. Sunset salvo. Am Stat 1986, 40:72-76.
347. Tukey J.W. Nonparametric estimation II: statistically equivalent blocks and tolerance regions - the continuous case. Ann Math Stat 1947, 18:529-539.
348. Tukey J.W. Nonparametric estimation III: statistically equivalent blocks and tolerance regions - the discontinuous case. Ann Math Stat 1948, 19:30-39.
349. Tusher V.G., Tibshirani R., Chu G. Significance analysis of microarrays applied to the ionising radiation response. Proc Natl Acad Sci USA 2001, 98:5116-5121.
350. Vanderlooy S., Sprinkhuizen-Kuyper I.G. A comparison of two approaches to classify with guaranteed performance. Lecture notes in computer science 2007, vol. 4702:288-299. Springer, Berlin. J.N. Kok, J. Koronacki, R. López de Mántaras, S. Matwin, D. Mladenic, A. Skowron (Eds.).
351. Vanderlooy S., Sprinkhuizen-Kuyper I.G., Smirnov E.N., Jaap van den Herik H. The ROC isometrics approach to construct reliable classifiers. Intell Data Anal 2009, 13(1):3-37.
352. Vapnik V. Statistical learning theory 1998, Wiley, New York.
353. Vapnik V. The nature of statistical learning theory 2000, Springer. 2nd ed.
354. Vega J., Murari A., Pereira A., Gonzalez S., Pastor I. Accurate and reliable image classification by using conformal predictors in the TJ-II thomson scattering. Rev Sci Instrum 2010, 81(10). 10E118-10E118-4.
355. Venn J. The logic of chance 1866, Macmillan, London.
356. Verdoolaege G., Vega J., Murari A., Van Oost G. Identification of confinement regimes in tokamak plasmas by conformal prediction on a probabilistic manifold. IFIP advances in information and communication technology 2012, vol. 382:244-253. Springer. L.S. Iliadis, I. Maglogiannis, H. Papadopoulos, K. Karatzas, S. Sioutas (Eds.).
357. Viola P., Jones M. Rapid object detection using a boosted cascade of simple features. Proceedings of the computer vision and pattern recognition conference (CVPR) 2001.
358. Vovk V. Aggregating strategies. In: Proceedings of the third annual workshop on computational learning theory; 1990.
359. Vovk V. On-line confidence machines are well-calibrated. Proceedings of the 43rd annual symposium on foundations of computer science, Los Alamitos, CA 2002, 187-196. IEEE Computer Society.
360. Vovk V. Conditional validity of inductive conformal predictors. Mach Learn 2013, 92:349-376.
361. Vovk V. Cross-conformal predictors. Ann Math Artif Intell 2013, (1012-2443):20. 10.1007/s10472-013-9368-4.
362. Vovk V. Kernel ridge regression. Empirical inference: festschrift in honor of Vladimir N. Vapnik 2013, Springer, Heidelberg, [chapter 11]. B. Schölkopf, Z. Luo, V. Vovk (Eds.).
363. Vovk V, Fedorova V, Nouretdinov I, Gammerman A. Criteria of efficiency for conformal prediction, in preparation.
364. Vovk V., Gammerman A., Saunders C. Machine-learning applications of algorithmic randomness. Proceedings of the 16th international conference on machine learning, San Francisco, CA 1999, 444-453. Morgan Kaufmann.
365. Vovk V., Gammerman A., Shafer G. Algorithmic learning in a random world 2005, Springer, New York.
366. Vovk V., Nouretdinov I., Gammerman A. Testing exchangeability on-line. ICML 2003, 768-775. AAAI Press. T. Fawcett, N. Mishra (Eds.).
367. Vovk V, Petej I. Venn-Abers predictors, in preparation.
368. Wackernagel H. Multivariate geostatistics: an introduction with applications 2003, Springer, Berlin.
369. Wald A. Sequential analysis 1947, Wiley, NY.
370. Wald A. An extension of Wilks' method for setting tolerance limits. Ann Math Stat 1943, 14:45-55.
371. Wand M.P., Jones M.C. Kernel smoothing 1995, Chapman and Hall, London.
372. Wang H., Lin Ch., Yang F., Hu X. Hedged predictions for traditional chinese chronic gastritis diagnosis with confidence machine. Comput Biol Med 2009, 39(5).
373. Wang H., Fan W., Yu P.S., Han J. Mining concept-drifting data streams using ensemble classifiers. Proceedings of the ninth ACM SIGKDD international conference on knowledge discovery and data mining 2003, 226-235. ACM. L. Getoor, T.E. Senator, P. Domingos, C. Faloutsos (Eds.).
374. Wasserman L. Frasian inference. Stat Sci 2011, 26:322-325.
375. Wechsler H. Reliable face recognition methods 2007, Springer.
376. Weigend A.S., Nix D.A. Predictions with confidence intervals (local error bars). Proceedings of the international conference on neural information processing, Seoul, Korea 1994, 847-852.
377. Weston J., Mukherjee S., Chapelle O., Pontil M., Poggio T., Vapnik V. Feature selection for SVMs. Proceedings of NIPS 2000, 668-674.
378. Widmer G., Kubat Miroslav Learning in the presence of concept drift and hidden contexts. Mach Learn 1996, 23(1):69-101.
379. Wilks S.S. Determination of sample sizes for setting tolerance limits. Ann Math Stat 1941, 12:91-96.
380. Williams N., Zander S., Armitage G. A preliminary performance comparison of five machine learning algorithms for practical ip traffic flow classification. SIGCOMM Comput Commun Rev 2006, 36(5):5-16.
381. Witten I.H., Frank E., Hall M.A. Data mining: practical machine learning tools and techniques 2011, Morgan Kaufmann.
382. Wold S., Ruhe A., Wold H., Dunn W.J. The collinearity problem in linear regression. the partial least squares (PLS) approach to generalized inverses. SIAM J Sci Stat Comput 1984, 5(3):735-743.
383. Wolpert D.H. Stacked generalization. Neural Netw 1992, 5:241-259.
384. Xu Z., Yu K., Tresp V., Xu X., Wang J. Representative sampling for text classification using support vector machines. European conference on information retrieval 2003.
385. Yang F., Zhen Wang H., Mi H., de Lin C., Wen Cai W. Using random forest for reliable classification and cost-sensitive learning for medical diagnosis. BMC Bioinf 2009, 10(1).
386. Yang M., Nouretdinov I., Luo Z., Gammerman A. Feature selection by conformal predictor. Proceedings of 12th INNS EANN-SIG international conference and seventh IFIP WG 12.5 international conference 2011, vol. II:449.
387. Yeoh E.-J., Ross M.E., Shurtleff S.A., Williams W.K., Patel D., Mahfouza R., et al. Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukaemia by gene expression profiling. Cancer Cell 2002, 1:133-143.
388. Yoon S.-H., Park J.-W., Park J.-S., Oh Y.-S., Kim M.-S. Internet application traffic classification using fixed ip-port. Proceedings of the 12th Asia-Pacific network operations and management conference on management enabling the future internet for changing business and new computing services 2009, 21-30.
389. Yu L., Liu H. Feature selection for high-dimensional data: a fast correlation-based filter solution. Proceedings of the 20th international conference on machine learning (ICML-2003) 2003, 856-863.
390. Yu L., Liu H. Efficient feature selection via analysis of relevance and redundancy. J Mach Learn Res 2004, 5:1205-1224.
391. Zaffalon M., Wesnes K., Petrini O. Reliable diagnoses of dementia by the naive credal classifier inferred from incomplete cognitive data. Artif Intell Med 2003, 29(1):61-79.
392. Zander S., Nguyen T., Armitage G. Automated traffic classification and application identification using machine learning. IEEE conference on local computer networks 2005, 250-257.
393. Zhang T. A leave-one-out cross validation bound for kernel methods with application in learning. Lecture notes in computer science: 14th annual conference on computation learning theory 2001, 2111:427-443.
394. Zhang X., Zhao D., Chen L., Min W. Batch mode active learning based multi-view text classification. International conference on Fuzzy systems and knowledge discovery 2009.
395. Zhu X. Semi-supervised learning literature survey. Computer sciences technical report 1530. University of Wisconsin-Madison; 2005.
396. Zhu X, Lafferty J, Ghahramani Z. Combining active learning and semi-supervised learning using gaussian fields and harmonic functions. In: Proceedings of the ICML workshop on the continuum from labeled to unlabeled data; 2003.