Bayesian mixed-effects inference on classification performance in hierarchical data sets

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 3133-3176

  Brodersen, Kay H ^a,b   Mathys, Christoph ^a   Chumbley, Justin R ^a   Daunizeau, Jean ^a   Ong, Cheng Soon ^b   Buhmann, Joachim M ^b   Stephan, Klaas E ^a  

^a UNIVERSITY OF ZURICH   (Switzerland)

^b ETH ZURICH   (Switzerland)

Author keywords

Balanced accuracy; Bayesian inference; Beta binomial; Group studies; Normal binomial

Indexed keywords

BALANCED ACCURACY; BAYESIAN INFERENCE; BETA-BINOMIAL; GROUP STUDY; NORMAL-BINOMIAL;

BAYESIAN NETWORKS; HIERARCHICAL SYSTEMS; INFERENCE ENGINES;

CLASSIFICATION (OF INFORMATION);

EID: 84870931397     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (68)

1. A. Agresti and D. B. Hitchcock. Bayesian inference for categorical data analysis. Statistical Methods and Applications, 14(3):297-330, Dec. 2005. ISSN 1618-2510. doi: 10.1007/ s10260-005-0121-y. (Pubitemid 41783361)
2. R. Akbani, S. Kwek, and N. Japkowicz. Applying support vector machines to imbalanced datasets. In Machine Learning: ECML 2004, pages 39-50. 2004. (Pubitemid 41050079)
3. J. H. Albert. Empirical Bayes estimation of a set of binomial probabilities. Journal of Statistical Computation and Simulation, 20(2):129-144, 1984. ISSN 0094-9655.
4. J. H. Albert and A. K. Gupta. Estimation in contingency tables using prior information. Journal of the Royal Statistical Society. Series B (Methodological), 45(1):60-69, Jan. 1983. ISSN 00359246.
5. T. Bayes and R. Price. An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society, 53:370-418, 1763. ISSN 0261-0523. doi: doi:10.1098/rstl. 1763.0053.
6. M. Beal. Variational Algorithms for Approximate Bayesian Inference. Ph.D. thesis, University College London, United Kingdom, 2003.
7. C. F. Beckmann, M. Jenkinson, and S. M. Smith. General multilevel linear modeling for group analysis in fMRI. NeuroImage, 20(2):1052-1063, Oct. 2003. ISSN 1053-8119. doi: 10.1016/ S1053-8119(03)00435-X. (Pubitemid 37268124)
8. T. E. J. Behrens, M. W. Woolrich, M. E. Walton, and M. F. S. Rushworth. Learning the value of information in an uncertain world. Nature Neuroscience, 10:1214-1221, 2007. (Pubitemid 47339952)
9. D. A. Berry and R. Christensen. Empirical Bayes estimation of a binomial parameter via mixtures of Dirichlet processes. The Annals of Statistics, 7(3):558-568, May 1979. ISSN 0090-5364. doi: 10.1214/aos/1176344677.
10. C. M. Bishop. Pattern Recognition and Machine Learning. Springer New York., 2007.
11. L. Breiman and P. Spector. Submodel selection and evaluation in regression. The x-random case. International Statistical Review/Revue Internationale de Statistique, page 291-319, 1992.
12. K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann. The balanced accuracy and its posterior distribution. In Proceedings of the 20th International Conference on Pattern Recognition, pages 3121-3124. IEEE Computer Society, 2010a. ISBN 1051-4651. doi: 10.1109/ICPR.2010.764.
13. K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann. The binormal assumption on precision-recall curves. In Proceedings of the 20th International Conference on Pattern Recognition, pages 4263-4266. IEEE Computer Society, 2010b. ISBN 1051-4651. doi: 10.1109/ICPR.2010.1036.
14. C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1-27:27, 2011.
15. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16(3):321-357, 2002. (Pubitemid 43057176)
16. G. F. Cooper and E. Herskovits. A Bayesian method for the induction of probabilistic networks from data. Machine Learning, 9(4):309-347, Oct. 1992. ISSN 0885-6125. doi: 10.1007/BF00994110.
17. G. V. Cormack. Email spam filtering: a systematic review. Foundations and Trends in Information Retrieval, 1(4):335-455, Apr. 2008. ISSN 1554-0669. doi: 10.1561/1500000006.
18. D. D. Cox and R. L. Savoy. Functional magnetic resonance imaging (fMRI) "brain reading": Detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage, 19(2):261-270, 2003. ISSN 1053-8119. doi: 10.1016/S1053-8119(03)00049-1. (Pubitemid 39665678)
19. J. J. Deely and D. V. Lindley. Bayes empirical Bayes. Journal of the American Statistical Association, 76(376):833-841, Dec. 1981. ISSN 01621459. doi: 10.2307/2287578.
20. O. Demirci, V. P. Clark, V. A. Magnotta, N. C. Andreasen, J. Lauriello, K. A. Kiehl, G. D. Pearlson, and V. D. Calhoun. A review of challenges in the use of fMRI for disease classification / characterization and a projection pursuit application from a multi-site fMRI schizophrenia study. Brain Imaging and Behavior, 2(3):207-226, Aug. 2008. ISSN 1931-7557. doi: 10.1007/s11682-008- 9028-1.
21. P. Dixon. Models of accuracy in repeated-measures designs. Journal of Memory and Language, 59 (4):447-456, Nov. 2008. ISSN 0749-596X. doi: 10.1016/j.jml.2007.11.004.
22. B. Efron and C. Morris. Limiting the risk of Bayes and empirical Bayes estimators - part I: the Bayes case. Journal of the American Statistical Association, pages 807-815, 1971.
23. B. Efron and C. Morris. Limiting the risk of Bayes and empirical Bayes estimators - part II: the empirical Bayes case. Journal of the American Statistical Association, page 130-139, 1972.
24. P. J. Everson and E. T. Bradlow. Bayesian inference for the beta-binomial distribution via polynomial expansions. Journal of Computational and Graphical Statistics, 11(1):202-207, Mar. 2002. ISSN 10618600.
25. S. Fazli, M. Danoczy, J. Schelldorfer, and K.-R. Mǔller. L1-penalized linear mixed-effects models for high dimensional data with application to BCI. NeuroImage, 56(4):2100-2108, June 2011. ISSN 1053-8119. doi: 16/j.neuroimage.2011.03.061.
26. K. J. Friston, A. P. Holmes, K. J. Worsley, J. P. Poline, C. D. Frith, and R. S. J. Frackowiak. Statistical parametric maps in functional imaging: A general linear approach. Human Brain Mapping, 2(4):189-210, 1995. (Pubitemid 124008664)
27. K. J. Friston, K. E. Stephan, T. E. Lund, A.Morcom, and S. Kiebel. Mixed-effects and fMRI studies. NeuroImage, 24(1):244-252, Jan. 2005. ISSN 1053-8119. doi: 10.1016/j.neuroimage.2004.08. 055. (Pubitemid 39611045)
28. F. Galton. Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15:246-263, 1886.
29. A. Gelfand and A. Smith. Sampling-based approaches to computing marginal densities. Journal of the American Statistical Association, 85(410):398-409, 1990.
30. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2 edition, July 2003. ISBN 9781584883883.
31. S. Geman and D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, vol. 30, p. 712-727, 6(1):721-741, 1984.
32. H. Goldstein. Multilevel Statistical Models, volume 847. Wiley, 2010.
33. I. J. Good. On the estimation of small frequencies in contingency tables. Journal of the Royal Statistical Society. Series B (Methodological), 18(1):113-124, 1956. ISSN 0035-9246.
34. B. S. Griffin and R. G. Krutchkoff. An empirical Bayes estimator for P[success] in the binomial distribution. The Indian Journal of Statistics, Series B, 33(3/4):217-224, Dec. 1971. ISSN 05815738.
35. M. G. Gustafsson, M. Wallman, U. Wickenberg Bolin, H. Gǒransson, M. Fryknǎs, C. R. Andersson, and A. Isaksson. Improving Bayesian credibility intervals for classifier error rates using maximum entropy empirical priors. Artificial Intelligence in Medicine, 49(2):93-104, June 2010. ISSN 0933-3657. doi: 10.1016/j.artmed.2010.02.004.
36. S. A. Harrison and F. Tong. Decoding reveals the contents of visual working memory in early visual areas. Nature, 458(7238):632-635, 2009. ISSN 0028-0836. doi: 10.1038/nature07832.
37. W. James and C. Stein. Estimation with quadratic loss. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, page 361, 1961.
38. N. Japkowicz and S. Stephen. The class imbalance problem: A systematic study. Intelligent Data Analysis, 6(5):429-449, 2002.
39. F. V. Jensen and T. D. Nielsen. Bayesian Networks and Decision Graphs. Springer, 2007. ISBN 9780387682815.
40. R. E. Kass and A. E. Raftery. Bayes factors. Journal of the American Statistical Association, 90 (430):773-795, 1995. ISSN 01621459.
41. A. Knops, B. Thirion, E. M. Hubbard, V. Michel, and S. Dehaene. Recruitment of an area involved in eye movements during mental arithmetic. Science (New York, N.Y.), 324(5934):1583-1585, May 2009. ISSN 1095-9203. doi: 10.1126/science.1171599.
42. R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In International Joint Conference on Artificial Intelligence, volume 14, pages 1137-1145. Lawrence Erlbaum Associates Ltd., 1995.
43. I. Krajbich, C. Camerer, J. Ledyard, and A. Rangel. Using neural measures of economic value to solve the public goods free-rider problem. Science (New York, N.Y.), 326(5952):596-599, Oct. 2009. ISSN 1095-9203. doi: 10.1126/science.1177302.
44. J. Langford. Tutorial on practical prediction theory for classification. Journal of Machine Learning Research, 6:273-306, 2005.
45. P. S. Laplace. Memoire sur la probabilit'e des causes par les ́ev́enemens. De l'Imprimerie Royale, 1774.
46. J. C. Lee and D. J. Sabavala. Bayesian estimation and prediction for the beta-binomial model. Journal of Business & Economic Statistics, 5(3):357-367, July 1987. ISSN 07350015. doi: 10.2307/1391611.
47. T. Leonard. Bayesian methods for binomial data. Biometrika, 59(3):581-589, Dec. 1972. doi: 10.1093/biomet/59.3.581.
48. D. J. C. MacKay. Bayesian interpolation. Neural Computation, 4:415-447, 1992.
49. D. Madigan and A. E. Raftery. Model selection and accounting for model uncertainty in graphical models using Occam's window. Journal of the American Statistical Association, 89(428):1535- 1546, Dec. 1994. ISSN 0162-1459. doi: 10.2307/2291017.
50. D. Madigan and J. C. York. Bayesian methods for estimation of the size of a closed population. Biometrika, 84(1):19 -31, 1997. doi: 10.1093/biomet/84.1. 19.
51. D. Madigan, A. E. Raftery, C. Volinsky, and J. Hoeting. Bayesian model averaging. In Proceedings of the AAAI Workshop on Integrating Multiple Learned Models, Portland, OR, pages 77-83, 1996.
52. N. Metropolis and S. Ulam. The Monte Carlo method. Journal of American Statistical Association, 44:335-341, 1949.
53. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21(6):1087, 1953. ISSN 00219606. doi: 10.1063/1.1699114.
54. E. Olivetti, S. Veeramachaneni, and E. Nowakowska. Bayesian hypothesis testing for pattern discrimination in brain decoding. Pattern Recognition, 45(6):2075-2084, June 2012. ISSN 0031- 3203. doi: 10.1016/j.patcog.2011.04.025.
55. E. S. Pearson. Bayes' theorem, examined in the light of experimental sampling. Biometrika, 17 (3/4):388-442, 1925. ISSN 0006-3444.
56. W. D. Penny, K. E. Stephan, A. Mechelli, and K. J. Friston. Comparing dynamic causal models. NeuroImage, 22(3):1157-1172, 2004. (Pubitemid 38829205)
57. M. A. Pitt and I. J. Myung. When a good fit can be bad. Trends in Cognitive Sciences, 6(10), 2002.
58. C. P. Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation. Springer, 2007. ISBN 9780387715988.
59. J. Schelldorfer, P. Bǔhlmann, and S. V. De Geer. Estimation for high-dimensional linear mixedeffects models using l1-penalization. Scandinavian Journal of Statistics, 38(2):197-214, 2011. ISSN 1467-9469. doi: 10.1111/j.1467-9469.2011.00740.x. URL http://onlinelibrary. wiley.com/doi/10. 1111/j.1467-9469.2011.00740.x/abstract.
60. A. Schurger, F. Pereira, A. Treisman, and J. D. Cohen. Reproducibility distinguishes conscious from nonconscious neural representations. Science, 327(5961):97-99, Jan. 2010. doi: 10.1126/ science.1180029.
61. R. Sitaram, N. Weiskopf, A. Caria, R. Veit, M. Erb, and N. Birbaumer. fMRI brain-computer interfaces: A tutorial on methods and applications. Signal Processing Magazine, IEEE, 25(1): 95-106, 2008. ISSN 1053-5888. doi: 10.1109/MSP.2008.4408446.
62. J. G. Skellam. A probability distribution derived from the binomial distribution by regarding the probability of success as variable between the sets of trials. Journal of the Royal Statistical Society. Series B (Methodological), 10(2):257-261, 1948. ISSN 0035-9246.
63. C. Stein. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the Third Berkeley symposium on mathematical statistics and probability, volume 1, page 197-206, 1956.
64. K. E. Stephan, W. D. Penny, J. Daunizeau, R. J. Moran, and K. J. Friston. Bayesian model selection for group studies. NeuroImage, 46(4):1004-1017, July 2009. ISSN 1095-9572. doi: 10.1016/j. neuroimage.2009.03.025.
65. D. R. Velez, B. C. White, A. A. Motsinger, W. S. Bush, M. D. Ritchie, S. M. Williams, and J. H. Moore. A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction. Genetic Epidemiology, 31(4):306-315, May 2007. ISSN 0741- 0395. doi: 10.1002/gepi.20211. (Pubitemid 46642095)
66. U. Wickenberg-Bolin, H. Goransson, M. Fryknas, M. Gustafsson, and A. Isaksson. Improved variance estimation of classification performance via reduction of bias caused by small sample size. BMC Bioinformatics, 7(1):127, 2006. ISSN 1471-2105. doi: 10.1186/1471-2105-7-127.
67. I. A. Wood, P. M. Visscher, and K. L. Mengersen. Classification based upon gene expression data: bias and precision of error rates. Bioinformatics, 23(11):1363-1370, June 2007. doi: 10.1093/bioinformatics/btm117. (Pubitemid 47050518)
68. D. Zhang and W. S. Lee. Learning classifiers without negative examples: A reduction approach. In Third International Conference on Digital Information Management, 2008. ICDIM 2008, pages 638-643, 2008. doi: 10.1109/ICDIM.2008. 4746761.