Critical assessment of automated flow cytometry data analysis techniques

Nature Methods

Volumn 10, Issue 3, 2013, Pages 228-238

  Aghaeepour, Nima ^a   Finak, Greg ^b   Hoos, Holger ^c   Mosmann, Tim R ^d   Brinkman, Ryan ^a   Gottardo, Raphael ^b   Scheuermann, Richard H ^e   Dougall, David ^f,g   Khodabakhshi, Alireza Hadj ^h   Mah, Phillip ^c   Obermoser, Gerlinde ⁱ   Spidlen, Josef ^a   Taylor, Ian ^j   Wuensch, Sherry A ^d   Bramson, Jonathan ^k   Eaves, Connie ^l   Weng, Andrew P ^l   Fortuno III, Edgardo S ^m   Ho, Kevin ^m   Kollmann, Tobias R ^m   Rogers, Wade ⁿ   De Rosa, Stephen ^o   Dalai, Bakul ^p   Azad, Ariful ^q   Pothen, Alex ^q   Brandes, Aaron ^r   Bretschneider, Hannes ^s   Bruggner, Robert ^t   Finck, Rachel ^t   Jia, Robin ^t   Zimmerman, Noah ^t   Linderman, Michael ^t   Dill, David ^t   Nolan, Gary ^t   Chan, Cliburn ^u   Khettabi, Faysal El ^a   O'Neill, Kieran ^a   Chikina, Maria ^v   Ge, Yongchao ^v   Sealfon, Stuart ^v   Sugár, István ^v   Gupta, Arvind ^c   Shooshtari, Parisa ^c   Zare, Habil ^c   De Jager, Philip L ^r,w,x   Jiang, Mike ^b   Keilwagen, Jens ^y   Maisog, Jose M ^z   Luta, George ^z   Barbo, Andrea A ^z   Májek, Peter ^aa   Vilček, Jozef ^aa   Manninen, Tapio ^ab   Huttunen, Heikki ^ab   Ruusuvuori, Pekka ^ab   Nykter, Matti ^ab   McLachlan, Geoffrey J ^ac   Wang, Kui ^ac   Naim, Iftekhar ^ad   Sharma, Gaurav ^ad   Nikolic, Radina ^ae   Pyne, Saumyadipta ^r,af   Qian, Yu ^f   Qiu, Peng ^ag   Quinn, John ^j   Roth, Andrew ^ah   Meyer, Pablo ^ai   Stolovitzky, Gustavo ^ai   Saez Rodriguez, Julio ^aj   Norel, Raquel ^ai   Bhattacharjee, Madhuchhanda ^af,ak   Biehl, Michael ^al   Bucher, Philipp ^am   Bunte, Kerstin ^an   Di Camillo, Barbara ^ao   Sambo, Francesco ^ap   Sanavia, Tiziana ^ao   Trifoglio, Emanuele ^ao   Toffolo, Gianna ^ao   Slavica Dimitrieva, S D ^am   Dreos, Rene ^am   Ambrosini, Giovanna ^am   Grau, Jan ^ap   Grosse, Ivo ^ap   Posch, Stefan ^ap   Guex, Nicolas ^aq   Kursa, Miron ^ar   Rudnicki, Witold ^ar   Liu, Bo ^as   Maienschein Cline, Mark ^at   Schneider, Petra ^au   Seifert, Michael ^av   Strickert, Marc ^aw   Vilar, Jose M G ^ax,ay   more..

^a BRITISH COLUMBIA CANCER AGENCY   (Canada)

^b Fred Hutchinson Cancer Research Center   (United States)

^c UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^d UNIVERSITY OF ROCHESTER SCHOOL OF MEDICINE AND DENTISTRY   (United States)

^e J CRAIG VENTER INSTITUTE   (United States)

^f UNIVERSITY OF TEXAS SOUTHWESTERN MEDICAL CENTER   (United States)

^g UNIVERSITY OF TEXAS SOUTHWESTERN MEDICAL CENTER   (United States)

^h CANADA'S MICHAEL SMITH GENOME SCIENCES CENTRE   (Canada)

ⁱ BAYLOR RESEARCH INSTITUTE   (United States)

^j Treestar Inc   (United States)

^k MCMASTER UNIVERSITY   (Canada)

^l BRITISH COLUMBIA CANCER AGENCY   (Canada)

^m UNIVERSITY OF BRITISH COLUMBIA   (Canada)

ⁿ UNIVERSITY OF PENNSYLVANIA   (United States)

^o UNIVERSITY OF WASHINGTON   (United States)

^p VANCOUVER GENERAL HOSPITAL   (Canada)

^q Purdue University   (United States)

^r BROAD INSTITUTE OF MIT AND HARVARD   (United States)

^s UNIVERSITY OF TORONTO   (Canada)

^t STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^u DUKE UNIVERSITY MEDICAL CENTER   (United States)

^v MOUNT SINAI SCHOOL OF MEDICINE   (United States)

^w BRIGHAM AND WOMEN S HOSPITAL   (United States)

^x HARVARD MEDICAL SCHOOL   (United States)

^y LEIBNIZ INSTITUTE OF PLANT GENETICS AND CROP PLANT RESEARCH IPK   (Germany)

^z GEORGETOWN UNIVERSITY MEDICAL CENTER   (United States)

^aa Metabol KLINIK Sro   (Slovakia)

^ab TAMPERE UNIVERSITY OF TECHNOLOGY   (Finland)

^ac UNIVERSITY OF QUEENSLAND   (Australia)

^ad University of Rochester   (United States)

^ae BRITISH COLUMBIA INSTITUTE OF TECHNOLOGY   (Canada)

^af UNIVERSITY OF HYDERABAD   (India)

^ag UNIVERSITY OF TEXAS MD ANDERSON CANCER CENTER   (United States)

^ah SIMON FRASER UNIVERSITY   (Canada)

^ai IBM RESEARCH   (United States)

^aj EUROPEAN BIOINFORMATICS INSTITUTE   (United Kingdom)

^ak UNIVERSITY OF PUNE   (India)

^al UNIVERSITY OF GRONINGEN   (Netherlands)

^am EPFL   (Switzerland)

^an BIELEFELD UNIVERSITY   (Germany)

^ao UNIVERSITY OF PADOVA   (Italy)

^ap MARTIN LUTHER UNIVERSITY HALLE WITTENBERG   (Germany)

^aq SWISS INSTITUTE OF BIOINFORMATICS   (Switzerland)

^ar UNIVERSITY OF WARSAW   (Poland)

^as Bldg 142   (United States)

^at UNIVERSITY OF CHICAGO   (United States)

^au UNIVERSITY OF BIRMINGHAM   (United Kingdom)

^av DRESDEN UNIVERSITY OF TECHNOLOGY   (Germany)

^aw PHILIPPS UNIVERSITY MARBURG   (Germany)

^ax BASQUE FOUNDATION FOR SCIENCE   (Spain)

^ay UNIVERSITY OF THE BASQUE COUNTRY UPV EHU   (Spain)

more..

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY; ARTICLE; AUTOMATION; CELL POPULATION; FLOW CYTOMETRY; HUMAN; PRIORITY JOURNAL; PRODUCT RECALL; T LYMPHOCYTE SUBPOPULATION; ALGORITHM; ANIMAL; BIOLOGY; BLOOD; CLUSTER ANALYSIS; COMPUTER PROGRAM; GRAFT VERSUS HOST REACTION; IMAGE PROCESSING; LARGE CELL LYMPHOMA; MONONUCLEAR CELL; PATHOLOGY; PROCEDURES; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; STANDARDS; STATISTICAL ANALYSIS; STATISTICS AND NUMERICAL DATA; VIROLOGY; WEST NILE FEVER; ACUTE GRANULOCYTIC LEUKEMIA; ALLOTYPE; CANCER CLASSIFICATION; GRANULOCYTE; HEMATOPOIETIC STEM CELL TRANSPLANTATION; HUMAN IMMUNODEFICIENCY VIRUS; MATURITY; MONOCYTE; PHENOTYPE; PREDICTION; WEST NILE FLAVIVIRUS; WORKFLOW;

MAMMALIA;

ALGORITHMS; ANIMALS; CLUSTER ANALYSIS; COMPUTATIONAL BIOLOGY; DATA INTERPRETATION, STATISTICAL; FLOW CYTOMETRY; GRAFT VS HOST DISEASE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LEUKOCYTES, MONONUCLEAR; LYMPHOMA, LARGE B-CELL, DIFFUSE; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY; SOFTWARE; WEST NILE FEVER;

GLYCOPROTEIN GP 120; HUMAN IMMUNODEFICIENCY VIRUS VACCINE; INTERLEUKIN 2;

EID: 84874666550     PISSN: 15487091     EISSN: 15487105     Source Type: Journal    
DOI: 10.1038/NMETH.2365     Document Type: Article

References (48)

1. Baumgarth, N. & Roederer, M. A practical approach to multicolor flow cytometry for immunophenotyping. J. Immunol. Methods 243, 77-97 ( 2000).
2. Tanner, S.D. et al. Flow cytometer with mass spectrometer detection for massively multiplexed single-cell biomarker assay. Pure Appl. Chem. 80, 2627-2641 ( 2008).
3. Bendall, S.C, Nolan, G.P., Roederer, M. & Chattopadhyay, P.K. A deep profiler's guide to cytometry. Trends Immunol. 33, 323-332 ( 2012).
4. Newell, E.W., Sigal, N., Bendall, S.C, Nolan, G.P. & Davis, M.M. Cytometry by time-of-flight shows combinatorial cytokine expression and virusspecific cell niches within a continuum of CD8+ T cell phenotypes. Immunity 36, 142-152 ( 2012).
5. Lugli, E., Roederer, M. & Cossarizza, A. Data analysis in flow cytometry: The future just started. Cytometry A 77, 705-713 ( 2010).
6. Quinn, J. et al. A statistical pattern recognition approach for determining cellular viability and lineage phenotype in cultured cells and murine bone marrow. Cytometry A 71, 612-624 ( 2007).
7. Lo, K., Brinkman, R.R. & Gottardo, R. Automated gating of flow cytometry data via robust model-based clustering. Cytometry A 73, 321-332 ( 2008).
8. Finak, G., Bashashati, A., Brinkman, R. & Gottardo, R. Merging mixture components for cell population identification in flow cytometry. Adv. Bioinformatics 2009, 247646 ( 2009).
9. Pyne, S. et al. Automated high-dimensional flow cytometric data analysis. Proc. Natl. Acad. Sci. USA 106, 8519-8524 ( 2009).
10. Naumann, U., Luta, G. & Wand, M.P. The curvHDR method for gating flow cytometry samples. BMC Bioinformatics 11, 44 ( 2010).
11. Suchard, M.A. et al. Understanding GPU programming for statistical computation: Studies in massively parallel massive mixtures. J. Comput. Graph. Stat. 19, 419-438 ( 2010).
12. Zare, H., Shooshtari, P., Gupta, A. & Brinkman, R.R. Data reduction for spectral clustering to analyze high throughput flow cytometry data. BMC Bioinformatics 11, 403 ( 2010).
13. Qian, Y. et al. Elucidation of seventeen human peripheral blood B-cell subsets and quantification of the tetanus response using a density-based method for the automated identification of cell populations in multidimensional flow cytometry data. Cytometry B Clin. Cytom. 78 ( suppl. 1), S69-S82 ( 2010).
14. Sugar, I.P. & Sealfon, S.C. Misty Mountain clustering: Application to fast unsupervised flow cytometry gating. BMC Bioinformatics 11, 502 ( 2010).
15. Aghaeepour, N., Nikolic, R., Hoos, H.H. & Brinkman, R.R. Rapid cell population identification in flow cytometry data. Cytometry A 79, 6-13 ( 2011).
16. Ge, Y. & Sealfon, S.C. flowPeaks: A fast unsupervised clustering for flow cytometry data via K-means and density peak finding. Bioinformatics 28, 2052-2058 ( 2012).
17. Aghaeepour, N. et al. Early immunologic correlates of HIV protection can be identified from computational analysis of complex multivariate T-cell flow cytometry assays. Bioinformatics 28, 1009-1016 ( 2012).
18. Zare, H. et al. Automated analysis of multidimensional flow cytometry data improves diagnostic accuracy between mantle cell lymphoma and small lymphocytic lymphoma. Am. J. Clin. Pathol. 137, 75-85 ( 2012).
19. Costa, E.S. et al. Automated pattern-guided principal component analysis vs expert-based immunophenotypic classification of B-cell chronic lymphoproliferative disorders: A step forward in the standardization of clinical immunophenotyping. Leukemia 24, 1927-1933 ( 2010).
20. Roederer, M., Nozzi, J.L. & Nason, M.C. SPICE: Exploration and analysis of post-cytometric complex multivariate datasets. Cytometry A 79, 167-174 ( 2011).
21. Azad, A., Pyne, S. & Pothen, A. Matching phosphorylation response patterns of antigen-receptor-stimulated T cells via flow cytometry. BMC Bioinformatics 13, S10 ( 2012).
22. Bendall, S.C. et al. Single-cell mass cytometry of differential immune and drug responses across a human hematopoietic continuum. Science 332, 687-696 ( 2011).
23. Qiu, P. et al. Extracting a cellular hierarchy from high-dimensional cytometry data with SPADE. Nat. Biotechnol. 29, 886-891 ( 2011).
24. Aghaeepour, N. et al. RchyOptimyx: Cellular hierarchy optimization for flow cytometry. Cytometry A 81, 1022-1030 ( 2012).
25. Chan, C. et al. Statistical mixture modeling for cell subtype identification in flow cytometry. Cytometry A 73, 693-701 ( 2008).
26. 2 discrepancy framework to mine high-throughput screening data for targeted drug discovery: Application to AIDS antiviral activity data of the National Cancer Institute. ( Data Mining for Biomedical Informatics workshop, SIAM Conf. Data Mining 2006).
27. Naim, I. et al. Swift: Scalable weighted iterative sampling for flow cytometry clustering, in Acoustics Speech and Signal Processing 509-512 ( IEEE, 2010).
28. Hahne, F. et al. flowCore: A Bioconductor package for high throughput flow cytometry. BMC Bioinformatics 10, 106 ( 2009).
29. Yang, P., Yang, Y.H., Zhou, B.B. & Zomaya, A.Y. A review of ensemble methods in bioinformatics. Current Bioinformatics 5, 296-308 ( 2010).
30. Maecker, H.T. et al. Standardization of cytokine flow cytometry assays. BMC Immunol. 6, 13 ( 2005).
31. Prill, R.J. et al. Towards a rigorous assessment of systems biology models: The DREAM3 challenges. PLoS ONE 5, e9202 ( 2010).
32. Stolovitzky, G., Prill, R.J. & Califano, A. Lessons from the DREAM2 Challenges. Ann. NY Acad. Sci. 1158, 159-195 ( 2009).
33. Meyer, P. et al. Verification of systems biology research in the age of collaborative competition. Nat Biotechnol. 29, 811-815 ( 2011).
34. Califano, A., Kellis, M. & Stolovitzky, G. Preface: RECOMB systems biology, regulatory genomics, and DREAM 2011 special issue. J. Comput. Biol. 19, 101 ( 2012).
35. Bain, B.J. Blood Cells: A Practical Guide 4th edn., Ch. 9, 398-468 (Wiley-Blackwell, 2006).
36. Maddox, A.M. et al. Philadelphia chromosome-positive adult acute leukemia with monosomy of chromosome number seven: A subgroup with poor response to therapy. Leuk. Res. 7, 509-522 ( 1983).
37. Tecimer, C., Loy, B.A. & Martin, A.W. Acute myeloblastic leukemia (MO) with an unusual chromosomal abnormality: Translocation (1;14) (pl3;q32). Cancer Genet. Cytogenet. 111, 175-177 ( 1999).
38. Peters, J.M. & Ansari, M.Q. Multiparameter flow cytometry in the diagnosis and management of acute leukemia. Arch. Pathol. Lab. Med. 135, 44-54 ( 2011).
39. Hornik, K. & Böhm, W. Hard and soft Euclidean consensus partitions, in Data Analysis, Machine Learning and Applications (eds. Preisach, C., Burkhardt, H., Schmidt-Thieme, L. & Decker, R.) 147-154 ( Springer, 2008).
40. Hornik, K. A clue for cluster ensembles. J. Stat Softw. 14, 1-25 ( 2005).
41. Gentleman, R. & Temple Lang, D. Statistical analyses and reproducible research. J. Comput Graph. Statist. 16, 1-23 ( 2007).
42. Lee, J.A. et al. MIFlowCyt: The minimum information about a flow cytometry experiment. Cytometry A 73, 926-930 ( 2008).
43. Brinkman, R.R. et al. High-content flow cytometry and temporal data analysis for defining a cellular signature of graft-versus-host disease. Biol. Blood Marrow Transplant. 6, 691-700 ( 2007).
44. Aghaeepour, N, Khodabakhshi, A.H. & Brinkman, R.R. Clustering Theory Workshop ( Neural Information Processing Systems, 2009).
45. Hesterberg, T., Moore, D.S., Monaghan, S., Clipson, A. & Epstein, R. Bootstrap methods and permutation tests, in Introduction to the Practice of Statistics 5th edn. (eds. Moore, D.S., McCabe, G.P. & Craig, B.) 16, 1-70 (W.H. Freeman, 2005).
46. Dym, CL., Wood, W.H. & Scott, M.J. Rank ordering engineering designs: Pairwise comparison charts and Borda counts. Res. Eng. Des. 13, 236-242 ( 2002).
47. Maecker, H.T. et al. Standardization of cytokine flow cytometry assays. BMC Immunol. 6, 13 ( 2005).
48. Maecker, H.T., McCoy, J.P. & Nussenblatt, R. Standardizing immunophenotyping for the human immunology project. Nat Rev. Immunol. 12, 191-200 ( 2012).