Machine vision methods for analyzing social interactions

Journal of Experimental Biology

Volumn 220, Issue 1, 2017, Pages 25-34

  Robie, Alice A ^a   Seagraves, Kelly M ^a   Egnor, S E Roian ^a   Branson, Kristin ^a  

^a HOWARD HUGHES MEDICAL INSTITUTE   (United States)

Author keywords

Animal behavior; Computer vision; Machine learning; Social behavior

Indexed keywords

ALGORITHM; ANIMAL; ANIMAL BEHAVIOR; DEVICES; EQUIPMENT DESIGN; MACHINE LEARNING; PROCEDURES; SOCIAL BEHAVIOR; VIDEORECORDING;

ALGORITHMS; ANIMALS; BEHAVIOR, ANIMAL; EQUIPMENT DESIGN; MACHINE LEARNING; SOCIAL BEHAVIOR; VIDEO RECORDING;

EID: 85010460271     PISSN: 00220949     EISSN: None     Source Type: Journal    
DOI: 10.1242/jeb.142281     Document Type: Review

References (69)

1. Adamo, S. A. and Hoy, R. R. (1995). Agonistic behaviour in male and female field crickets, Gryllus bimaculatus, and how behavioural context influences its expression. Anim. Behav. 49, 1491-1501.
2. Agrawal, S., Safarik, S. and Dickinson, M. (2014). The relative roles of vision and chemosensation in mate recognition of Drosophila melanogaster. J. Exp. Biol. 217, 2796-2805.
3. Ahrens, M. B., Orger, M. B., Robson, D. N., Li, J. M. and Keller, P. J. (2013). Whole-brain functional imaging at cellular resolution using light-sheet microscopy. Nat. Methods 10, 413-420.
4. Altmann, J. (1974). Observational study of behavior: Sampling methods. Behaviour 49, 227-267.
5. Anderson, P. W. (1972). More is different. Science 177, 393-396.
6. Ardekani, R., Biyani, A., Dalton, J. E., Saltz, J. B., Arbeitman, M. N., Tower, J., Nuzhdin, S. and Tavare, S. (2013). Three-dimensional tracking and behaviour monitoring of multiple fruit flies. J. R. Soc. Interface 10, 20120547.
7. Berclaz, J., Fleuret, F., Turetken, E. and Fua, P. (2011). Multiple object tracking using K-shortest paths optimization. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1806-1819.
8. Berman, G. J., Choi, D. M., Bialek, W. and Shaevitz, J. W. (2014). Mapping the stereotyped behaviour of freely moving fruit flies. J. R. Soc. Interface 11, 20140672.
9. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. New York: Springer.
10. Branson, K. and Belongie, S. (2005). Tracking multiple mouse contours (without too many samples). IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, Vol. 1, pp. 1039-1046.
11. Branson, K., Robie, A. A., Bender, J., Perona, P. and Dickinson, M. H. (2009). High-throughput ethomics in large groups of Drosophila. Nat. Methods 6, 451-457.
12. Brent, L. J. N., MacLarnon, A., Platt, M. L. and Semple, S. (2013). Seasonal changes in the structure of rhesus macaque social networks. Behav. Ecol. Sociobiol. 67, 349-359.
13. Certel, S. J. and Kravitz, E. A. (2012). Scoring and analyzing aggression in Drosophila. Cold Spring Harb. Protoc. 2012, 319-325.
14. Cheney, D. L. and Seyfarth, R. M. (1999). Recognition of other individuals' social relationships by female baboons. Anim. Behav. 58, 67-75.
15. Clutton-Brock, T. (2007). Sexual selection in males and females. Science 318, 1882-1885.
16. Coen, P., Clemens, J.,Weinstein, A. J., Pacheco, D. A., Deng, Y. and Murthy, M. (2014). Dynamic sensory cues shape song structure in Drosophila. Nature 507, 233-237.
17. Dalal, N. and Triggs, B. (2005). Histograms of oriented gradients for human detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, Vol. 1, pp. 886-893.
18. Dankert, H., Wang, L., Hoopfer, E. D., Anderson, D. J. and Perona, P. (2009). Automated monitoring and analysis of social behavior in Drosophila. Nat.Methods 6, 297-303.
19. Darwin, C. (1888). The Descent of Man, and Selection in Relation to Sex. London: J. Murray.
20. de Chaumont, F., Coura, R. D.-S., Serreau, P., Cressant, A., Chabout, J., Granon, S. and Olivo-Marin, J.-C. (2012). Computerized video analysis of social interactions in mice. Nat. Methods 9, 410-417.
21. Dow, M. A. and von Schilcher, F. (1975). Aggression and mating success in Drosophila melanogaster. Nature 254, 511-512.
22. Fiaschi, L., Diego, F., Gregor, K., Schiegg, M., Koethe, U., Zlatic, M. and Hamprecht, F. (2014). Tracking indistinguishable translucent objects over time using weakly supervised structured learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736-2743.
23. Freund, J., Brandmaier, A. M., Lewejohann, L., Kirste, I., Kritzler, M., Kruger, A., Sachser, N., Lindenberger, U. and Kempermann, G. (2013). Emergence of individuality in genetically identical mice. Science 340, 756-759.
24. Freund, J., Brandmaier, A. M., Lewejohann, L., Kirste, I., Kritzler, M., Kruger, A., Sachser, N., Lindenberger, U. and Kempermann, G. (2015). Association between exploratory activity and social individuality in genetically identical mice living in the same enriched environment. Neuroscience 309, 140-152.
25. Gershow, M., Berck, M., Mathew, D., Luo, L., Kane, E. A., Carlson, J. R. and Samuel, A. D. T. (2012). Controlling airborne cues to study small animal navigation. Nat. Methods 9, 290-296.
26. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2016). Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 38, 142-158.
27. Hamilton, W. D. (1964). The genetical evolution of social behaviour. I. J. Theor. Biol. 7, 1-16.
28. Hong, W., Kennedy, A., Burgos-Artizzu, X. P., Zelikowsky, M., Navonne, S. G., Perona, P. and Anderson, D. J. (2015). Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning. Proc. Natl. Acad. Sci. USA 112, E5351-E5360.
29. Hoopfer, E. D., Jung, Y., Inagaki, H. K., Rubin, G. M. and Anderson, D. J. (2015). P1 interneurons promote a persistent internal state that enhances inter-male aggression in Drosophila. Elife 4, e11346.
30. Hoyer, S. C., Eckart, A., Herrel, A., Zars, T., Fischer, S. A., Hardie, S. L. and Heisenberg, M. (2008). Octopamine in male aggression of Drosophila. Curr. Biol. 18, 159-167.
31. Insel, T. R. (2010). The challenge of translation in social neuroscience: A review of oxytocin, vasopressin, and affiliative behavior. Neuron 65, 768-779.
32. Jhuang, H., Garrote, E., Mutch, J., Yu, X., Khilnani, V., Poggio, T., Steele, A. D. and Serre, T. (2010). Automated home-cage behavioural phenotyping of mice. Nat. Commun. 1, 68.
33. Kabra, M., Robie, A. A., Rivera-Alba, M., Branson, S. and Branson, K. (2013). JAABA: Interactive machine learning for automatic annotation of animal behavior. Nat. Methods 10, 64-67.
34. Kabra, M., Robie, A. and Branson, K. (2015). Understanding Classifier Errors by Examining Influential Neighbors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3917-3925.
35. Kim, Y., Venkataraju, K. U., Pradhan, K., Mende, C., Taranda, J., Turaga, S. C., Arganda-Carreras, I., Ng, L., Hawrylycz, M. J., Rockland, K. S. et al. (2015). Mapping social behavior-induced brain activation at cellular resolution in the mouse. Cell Rep. 10, 292-305.
36. Kravitz, E. A. and Fernandez, M. L. (2015). Aggression in Drosophila. Behav. Neurosci. 129, 549-563.
37. Lenz, P., Geiger, A. and Urtasun, R. (2015). FollowMe: Efficient online min-cost flow tracking with bounded memory and computation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4364-4372.
38. Libersat, F., Murray, J. A. and Hoy, R. R. (1994). Frequency as a releaser in the courtship song of two crickets, Gryllus bimaculatus (de Geer) and Teleogryllus oceanicus: A neuroethological analysis. J. Comp. Physiol. A 174, 485-494.
39. Mersch, D. P., Crespi, A. and Keller, L. (2013). Tracking individuals shows spatial fidelity is a key regulator of ant social organization. Science 340, 1090-1093.
40. Michelsen, A., Andersen, B. B., Kirchner, W. H. and Lindauer, M. (1989). Honeybees can be recruited by a mechanical model of a dancing bee. Naturwissenschaften 76, 277-280.
41. Narins, P. M., Grabul, D. S., Soma, K. K., Gaucher, P. and Hö dl, W. (2005). Crossmodal integration in a dart-poison frog. Proc. Natl. Acad. Sci. USA 102, 2425-2429.
42. Neunuebel, J. P., Taylor, A. L., Arthur, B. J. and Egnor, S. E. R. (2015). Female mice ultrasonically interact with males during courtship displays. Elife 4, e06203.
43. Noldus, L. P. J. J., Spink, A. J. and Tegelenbosch, R. A. J. (2001). EthoVision: A versatile video tracking system for automation of behavioral experiments. Behav. Res. Methods Instrum. Comput. 33, 398-414.
44. Ohayon, S., Avni, O., Taylor, A. L., Perona, P. and Egnor, S. E. R. (2013). Automated multi-day tracking of marked mice for the analysis of social behaviour. J. Neurosci. Methods 219, 10-19.
45. Pavlou, H. J. and Goodwin, S. F. (2013). Courtship behavior in Drosophila melanogaster: Towards a 'courtship connectome'. Curr. Opin. Neurobiol. 23, 76-83.
46. Perez-Escudero, A., Vicente-Page, J., Hinz, R. C., Arganda, S. and de Polavieja, G. G. (2014). idTracker: Tracking individuals in a group by automatic identification of unmarked animals. Nat. Methods 11, 743-748.
47. Perony, N., Tessone, C. J., Konig, B. and Schweitzer, F. (2012). How random is social behaviour? Disentangling social complexity through the study of a wild house mouse population. PLoS Comput. Biol. 8, e1002786.
48. Petru, M., Spinka, M., Charvatova, V. and Lhota, S. (2009). Revisiting play elements and self-handicapping in play: A comparative ethogram of five OldWorld monkey species. J. Comp. Psychol. 123, 250-263.
49. Piccardi, M. (2004). Background subtraction techniques: A review. IEEE International Conference on Systems, Man and Cybernetics, Vol. 4, pp. 3099-3104.
50. Pirsiavash, H., Ramanan, D. and Fowlkes, C. C. (2011). Globally-optimal greedy algorithms for tracking a variable number of objects. IEEE Conference on Computer Vision and Pattern Recognition, pp. 1201-1208.
51. Ramdya, P., Schaffter, T., Floreano, D. and Benton, R. (2012). Fluorescence behavioral imaging (FBI) tracks identity in heterogeneous groups of Drosophila. PLoS ONE 7, e48381.
52. Robie, A. A., Straw, A. D. and Dickinson, M. H. (2010). Object preference by walking fruit flies, Drosophila melanogaster, is mediated by vision and graviperception. J. Exp. Biol. 213, 2494-2506.
53. Saka, E., Goodrich, C., Harlan, P., Madras, B. K. and Graybiel, A. M. (2004). Repetitive behaviors in monkeys are linked to specific striatal activation patterns. J. Neurosci. 24, 7557-7565.
54. Schiegg, M., Hanslovsky, P., Kausler, B., Hufnagel, L. and Hamprecht, F. (2013). Conservation tracking. Proceedings of the IEEE International Conference on Computer Vision, pp. 2928-2935.
55. Schneider, J., Dickinson, M. H. and Levine, J. D. (2012). Social structures depend on innate determinants and chemosensory processing in Drosophila. Proc. Natl. Acad. Sci. USA 109 Suppl. 2, 17174-17179.
56. Seagraves, K. M., Arthur, B. J. and Egnor, S. E. R. (2016). Evidence for an audience effect in mice: Male social partners alter the male vocal response to female cues. J. Exp. Biol. 219, 1437-1448.
57. Seeds, A. M., Ravbar, P., Chung, P., Hampel, S., Midgley, F. M., Jr, Mensh, B. D. and Simpson, J. H. (2014). A suppression hierarchy among competing motor programs drives sequential grooming in Drosophila. Elife 3, e02951.
58. Shannon, C. E. (1949). Communication in the presence of noise. Proc. Inst. Radio Eng. 37, 10-21.
59. Shemesh, Y., Sztainberg, Y., Forkosh, O., Shlapobersky, T., Chen, A. and Schneidman, E. (2013). High-order social interactions in groups of mice. Elife 2, e00759.
60. Simon, J. C. and Dickinson, M. H. (2010). A new chamber for studying the behavior of Drosophila. PLoS ONE 5, e8793.
61. Sokolowski, M. B. (2010). Social interactions in "simple" model systems. Neuron 65, 780-794.
62. Stoddard, P. K., Beecher, M. D., Horning, C. L. and Campbell, S. E. (1991). Recognition of individual neighbors by song in the song sparrow, a species with song repertoires. Behav. Ecol. Sociobiol. 29, 211-215.
63. Swierczek, N. A., Giles, A. C., Rankin, C. H. and Kerr, R. A. (2011). Highthroughput behavioral analysis in C. elegans. Nat. Methods 8, 592-598.
64. Tchernichovski, O. and Golani, I. (1995). A phase plane representation of rat exploratory behavior. J. Neurosci. Methods 62, 21-27.
65. Templeton, C. N., Greene, E. and Davis, K. (2005). Allometry of alarm calls: Blackcapped chickadees encode information about predator size. Science 308, 1934-1937.
66. Vogelstein, J. T., Park, Y., Ohyama, T., Kerr, R. A., Truman, J. W., Priebe, C. E. and Zlatic, M. (2014). Discovery of brainwide neural-behavioral maps via multiscale unsupervised structure learning. Science 344, 386-392.
67. Weissbrod, A., Shapiro, A., Vasserman, G., Edry, L., Dayan, M., Yitzhaky, A., Hertzberg, L., Feinerman, O. and Kimchi, T. (2013). Automated long-term tracking and social behavioural phenotyping of animal colonies within a seminatural environment. Nat. Commun. 4, 2018.
68. Yamamoto, D. and Koganezawa, M. (2013). Genes and circuits of courtship behaviour in Drosophila males. Nat. Rev. Neurosci. 14, 681-692.
69. York, R. A., Patil, C., Hulsey, C. D., Streelman, J. T. and Fernald, R. D. (2015). Evolution of bower building in Lake Malawi cichlid fish: Phylogeny, morphology, and behavior. Front. Ecol. Evol. 3, 18.