Multi-camera real-time three-dimensional tracking of multiple flying animals

Journal of the Royal Society Interface

Volumn 8, Issue 56, 2011, Pages 395-409

  Straw, Andrew D ^a   Branson, Kristin ^a   Neumann, Titus R ^a   Dickinson, Michael H ^a  

^a California Institute of Technology HHMI   (United States)

Author keywords

Animal behaviour; Birds; Computer vision; Flight; Insects; Manoeuvring

Indexed keywords

ALGORITHMS; ANIMALS; BEHAVIORAL RESEARCH; CAMERAS; COMPUTER VISION; NAVIGATION; SENSORY FEEDBACK; THREE DIMENSIONAL; TRACKING (POSITION); VIRTUAL REALITY;

ANIMAL BEHAVIOUR; ANIMAL MOVEMENT; AUTOMATED TRACKING; BODY ORIENTATION; CAMERA SYSTEMS; DATA ASSOCIATION ALGORITHMS; DIMENSIONAL POSITION; DROSOPHILA MELANOGASTER; FLIGHT; FLIGHT SPEED; FRAMES PER SECONDS; GENETIC MANIPULATIONS; GIGABIT NETWORK; HIGH CONTRAST; INSECTS; LOW CONTRAST; MANOEUVRING; MULTI-CAMERAS; MULTI-TARGET TRACKING; NEAREST NEIGHBOUR; PENTIUM 4; REAL TIME; REAL-TIME THREE-DIMENSIONAL TRACKING; TRACKING SYSTEM; VISUAL CONTRAST;

TARGET TRACKING;

ANIMAL BEHAVIOR; BIRD; CAMERA; COMPUTER; COMPUTER GRAPHICS; CONTROLLED STUDY; DROSOPHILA MELANOGASTER; FILTER; FLY; FLYING; GENETIC ALGORITHM; NEUROBIOLOGY; NONHUMAN; PUBLICATION; REVIEW; THREE DIMENSIONAL IMAGING; VELOCITY; VISUAL FIELD;

ANIMALIA; AVES; DROSOPHILA MELANOGASTER; HEXAPODA;

EID: 79251514605     PISSN: 17425689     EISSN: 17425662     Source Type: Journal    
DOI: 10.1098/rsif.2010.0230     Document Type: Review

References (64)

1. Srinivasan, M. V., Zhang, S. W., Lehrer, M. & Collett, T. S. 1996 Honeybee navigation en route to the goal: visual flight control and odometry. J. Exp. Biol. 199, 237-244.
2. Schilstra, C. & Hateren, J. H. 1999 Blowfly flight and optic flow. I. Thorax kinematics and flight dynamics. J. Exp. Biol. 202, 1481-1490.
3. Tammero, L. F. & Dickinson, M. H. 2002 The influence of visual landscape on the free flight behavior of the fruit fly Drosophila melanogaster. J. Exp. Biol. 205, 327-343.
4. Kern, R., van Hateren, J. H., Michaelis, C., Lindemann, J. P. & Egelhaaf, M. 2005 Function of a fly motion-sensitive neuron matches eye movements during free flight. PLoS Biol. 3, 1130-1138. (doi:10.1371/journal. pbio. 0030171)
5. Land, M. F. & Collett, T. S. 1974 Chasing behavior of houseflies (Fannia canicularis): a description and analysis. J. Comp. Physiol. 89, 331-357. (doi:10.1007/BF00695351)
6. Buelthoff, H., Poggio, T. & Wehrhahn, C. 1980 3-D analysis of the flight trajectories of flies (Drosophila melanogaster). Z. Nat. 35c, 811-815.
7. Wehrhahn, C., Poggio, T. & Bülthoff, H. 1982 Tracking and chasing in houseflies (musca). Biol. Cybernet. 45, 123-130. (doi:10.1007/ BF00335239)
8. Wagner, H. 1986 Flight performance and visual control of the flight of the free-flying housefly (Musca domestica). II. Pursuit of targets. Phil. Trans. R. Soc. Lond. B 312, 553-579. (doi:10.1098/rstb.1986.0018)
9. Mizutani, A., Chahl, J. S. & Srinivasan, M. V. 2003 Insect behaviour: motion camouflage in dragonflies. Nature 423, 604. (doi:10.1038/423604a)
10. Frye, M. A., Tarsitano, M. & Dickinson, M. H. 2003 Odor localization requires visual feedback during free flight in Drosophila melanogaster. J. Exp. Biol. 206, 843-855. (doi:10.1242/jeb.00175)
11. Budick, S. A. & Dickinson, M. H. 2006 Free-flight responses of Drosophila melanogaster to attractive odors. J. Exp. Biol. 209, 3001-3017. (doi:10.1242/jeb.02305)
12. David, C. T. 1978 Relationship between body angle and flight speed in free-flying Drosophila. Physiol. Entomol. 3, 191-195. (doi:10.1111/j.1365-3032. 1978.tb00148.x)
13. Fry, S. N., Sayaman, R. & Dickinson, M. H. 2003 The aerodynamics of free-flight maneuvers in Drosophila. Science 300, 495-498. (doi:10.1126/science. 1081944)
14. Collett, T. S. & Land, M. F. 1975 Visual control of flight behaviour in the hoverfly Syritta pipiens L. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 99, 1-66. (doi:10.1007/BF01464710)
15. Collett, T. S. & Land, M. F. 1978 How hoverflies compute interception courses. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 125, 191-204. (doi:10.1007/BF00656597)
16. Wehrhahn, C. 1979 Sex-specific differences in the chasing behaviour of houseflies (musca). Biol. Cybernet. 32, 239-241. (doi:10.1007/BF00337647)
17. Dahmen, H.-J. & Zeil, J. 1984 Recording and reconstructing three-dimensional trajectories: a versatile method for the field biologist. Proc. R. Soc. Lond. B 222, 107-113. (doi:10.1098/rspb.1984.0051)
18. Srinivasan, M. V., Zhang, S. W., Chahl, J. S., Barth, E. & Venkatesh, S. 2000 How honeybees make grazing landings on flat surfaces. Biol. Cybernet. 83, 171-183. (doi:10.1007/s004220000162)
19. Hedrick, T. L. & Biewener, A. A. 2007 Low speed maneuvering flight of the rose-breasted cockatoo (Eolophus roseicapillus). I. Kinematic and neuromuscular control of turning. J. Exp. Biol. 210, 1897-1911. (doi:10.1242/jeb.002055)
20. Tian, X., Diaz, J. I., Middleton, K., Galvao, R., Israeli, E., Roemer, A., Sullivan, A., Song, A., Swartz, S. & Breuer, K. 2006 Direct measurements of the kinematics and dynamics of bat flight. Bioinspiration & Biomimetics 1, S10-S18. (doi:10.1088/1748-3182/1/4/S02)
21. Khan, Z., Balch, T. & Dellaert, F. 2005 MCMC-based particle filtering for tracking a variable number of interacting targets. IEEE Trans. Pattern Anal. Mach. Intell. 27, 2005.
22. Khan, Z., Balch, T. & Dellaert, F. 2006 MCMC data association and sparse factorization updating for real time multitarget tracking with merged and multiple measurements. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1960-1972. (doi:10.1109/TPAMI.2006.247)
23. Branson, K., Robie, A., Bender, J., Perona, P. & Dickinson, M. H. 2009 High-throughput ethomics in large groups of Drosophila. Nat. Methods 6, 451-457. (doi:10.1038/nmeth.1328)
24. Klein, D. J. 2008 Coordinated control and estimation for multi-agent systems: theory and practice, ch. 6. Tracking Multiple Fish Robots Using Underwater Cameras, PhD thesis, University of Washington.
25. Qu, W., Schonfeld, D. & Mohamed, M. 2007 Distributed Bayesian multiple-target tracking in crowded environments using multiple collaborative cameras. EURASIP J. Adv. Signal Process. Article ID 38373. (doi:10.1155/2007/ 38373)
26. Qu, W., Schonfeld, D. & Mohamed, M. 2007 Real-time interactively distributed multi-object tracking using a magnetic-inertia potential model. IEEE Trans. Multimedia (TMM) 9, 511-519. (doi:10.1109/TMM.2006.886266)
27. Ballerini, M. et al. 2008 Empirical investigation of starling flocks: a benchmark study in collective animal behaviour. Anim. Behav. 76, 201-215. (doi:10.1016/j.anbehav.2008.02.004)
28. Cavagna, A., Cimarelli, A., Giardina, I., Orlandi, A., Parisi, G., Procaccini, A., Santagati, R. & Stefanini, F. 2008 New statistical tools for analyzing the structure of animal groups. Math. Biosci. 214, 32-37. (doi:10.1016/j.mbs.2008.05.006)
29. Cavagna, A., Giardina, I., Orlandi, A., Parisi, G. & Procaccini, A. 2008 The starflag handbook on collective animal behaviour: 2. Three-dimensional analysis. Anim. Behav. 76, 237-248. (doi:10.1016/j.anbehav.2008.02.003)
30. Cavagna, A., Giardina, I., Orlandi, A., Parisi, G., Procaccini, A., Viale, M. & Zdravkovic, V. 2008 The starflag handbook on collective animal behaviour: 1. Empirical methods. Anim. Behav. 76, 217-236. (doi:10.1016/j. anbehav. 2008.02.002)
31. Wu, H., Zhao, Q., Zou, D. & Chen, Y. 2009 Acquiring 3d motion trajectories of large numbers of swarming animals. IEEE Int. Conf. on Computer Vision (ICCV) Workshop on Video Oriented Object and Event Classification.
32. Zou, D., Zhao, Q., Wu, H. & Chen, Y. 2009 Reconstructing 3d motion trajectories of particle swarms by global correspondence selection. In Int. Conf. on Computer Vision (ICCV 09). Workshop on Video Oriented Object and Event Classification, pp. 1578-1585.
33. Bomphrey, R. J., Walker, S. M. & Taylor, G. K. 2009 The typical flight performance of blowflies: measuring the normal performance envelope of Calliphora vicina using a novel corner-cube arena. PLoS ONE 4, e7852. (doi:10.1371/journal.pone.0007852)
34. Marden, J. H., Wolf, M. R. & Weber, K. E. 1997 Aerial performance of Drosophila melanogaster from populations selected for upwind flight ability. J. Exp. Biol. 200, 2747-2755.
35. Fry, S. N., Muller, P., Baumann, H. J., Straw, A. D., Bichsel, M. & Robert, D. 2004 Context-dependent stimulus presentation to freely moving animals in 3d. J. Neurosci. Methods 135, 149-157. (doi:10.1016/j.jneumeth.2003.12.012) (Pubitemid 38333135)
36. Fry, S. N., Rohrseitz, N., Straw, A. D. & Dickinson, M. H. 2008 Trackfly-virtual reality for a behavioral system analysis in free-flying fruit flies. J. Neurosci. Methods 171, 110-117. (doi:10.1016/j.jneumeth.2008.02.016)
37. Grover, D., Tower, J. & Tavare, S. 2008 O fly, where art thou? J. R. Soc. Interface 5, 1181-1191. (doi:10.1098/rsif.2007.1333)
38. Maimon, G., Straw, A. D. & Dickinson, M. H. 2008 A simple vision-based algorithm for decision making in flying Drosophila. Curr. Biol. 18, 464-470. (doi:10.1016/j.cub.2008.03.050)
39. Piccardi, M. 2004 Background subtraction techniques: a review. In Proc. of IEEE SMC 2004 Int. Conf. on Systems, Man and Cybernetics. The Hague, The Netherlands, October 2004.
40. Bar-Shalom, Y. & Fortmann, T. E. 1988 Tracking and data association. New York, NY: Academic Press.
41. Abdel-Aziz, Y. I. & Karara, H. M. 1971 Direct linear transformation from comparator coordinates into object space coordinates. Proc. American Society of Photogrammetry Symp. on Close-Range Photogrammetry, Urbana, IL, 26-29 January 1971, pp. 1-18. Falls Church, VA: American Society of Photogrammetry.
42. Svoboda, T., Martinec, D. & Pajdla, T. 2005 A convenient multi-camera self-calibration for virtual environments. PRESENCE: Teleoperators and Virtual Environments 14, 407-422. (doi:10.1162/105474605774785325) (Pubitemid 41365734)
43. Sturm, P. & Triggs, B. 1996 A factorization based algorithm for multi-image projective structure and motion. In Proc. 4th Eur. Conf. on Computer Vision, April 1996, Cambridge, UK, pp. 709-720. Springer.
44. Hartley, R. I. & Zisserman, A. 2003 Multiple view geometry in computer vision, 2nd edn. Cambridge, UK: Cambridge University Press.
45. Bouguet, J. 2010 Camera calibration toolbox. See http:// www.vision.caltech.edu/bouguetj/calib-doc.
46. Prescott, B. & McLean, G. F. 1997 Line-based correction of radial lens distortion. Graph. Models Image Process. 59, 39-47. (doi:10.1006/gmip.1996. 0407)
47. Straw, A. D. & Dickinson, M. H. 2009 Motmot, an open-source toolkit for realtime video acquisition and analysis. Source Code Biol. Med. 4, 5, 1-20. (doi:10.1186/1751-0473-4-5)
48. Correll, K., Barendt, N. & Branicky, M. 2005 Design considerations for software-only implementations of the IEEE 1588 precision time protocol. Proc. Conf. on IEEE-1588 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems, NIST and IEEE, Winterthur, Switzerland, 10-12 October 1975.
49. Kennedy, J. S. 1939 Visual responses of flying mosquitoes. Proc. Zool. Soc. Lond. 109, 221-242.
50. Neumann, T. R. 2002 Modeling insect compound eyes: space-variant spherical vision. In Proc. of the 2nd Int. Workshop on Biologically Motivated Computer Vision (eds H. H. Bülthoff, S.-W. Lee, T. Poggio & C. Wallraven), LNCS 2525, pp. 360-367. Berlin, Germany: Springer.
51. Dickson, W. B., Straw, A. D. & Dickinson, M. H. 2008 Integrative model of drosophila flight. AIAA J. 46, 2150-2165. (doi:10.2514/1.29862)
52. Strauss, R., Schuster, S. & Götz, K. G. 1997 Processing of artificial visual feedback in the walking fruit fly Drosophila melanogaster. J. Exp. Biol. 200, 1281-1296.
53. Straw, A. D. 2008 Vision egg: an open-source library for realtime visual stimulus generation. Front. Neuroinformatics 2, 1-10. (doi:10.3389/neuro.11.004. 2008)
54. Stamps, J., Buechner, M., Alexander, K., Davis, J. & Zuniga, N. 2005 Genotypic differences in space use and movement patterns in Drosophila melanogaster. Anim. Behav. 70, 609-618. (doi:10.1016/j.anbehav.2004.11.018)
55. Dvorak, D., Srinivasan, M. V. & French, A. S. 1980 The contrast sensitivity of fly movement-detecting neurons. Vision Res. 20, 397-407. (doi:10.1016/0042-6989(80)90030-9)
56. Srinivasan, M. V. & Dvorak, D. R. 1980 Spatial processing of visual information in the movement-detecting pathway of the fly: characteristics and functional significance. J. Comp. Physiol. A 140, 1-23. (doi:10.1007/BF00613743)
57. Egelhaaf, M. & Borst, A. 1989 Transient and steady-state response properties of movement detectors. J. Opt. Soc. Am. A 6, 116-127. (doi:10.1364/JOSAA.6.000116)
58. Hassenstein, B. & Reichardt, W. 1956 Systemtheoretische analyse der zeit-, reihenfolgen- und vorzeichenauswertung bei der bewegungsperzeption des rüsselkafers Chlorophanus. Z. Nat. 11b, 513-524.
59. David, C. T. 1982 Compensation for height in the control of groundspeed by drosophila in a new, barbers pole wind-tunnel. J. Comp. Physiol. 147, 485-493. (doi:10.1007/BF00612014)
60. Srinivasan, M. V., Lehrer, M., Kirchner, W. H. & Zhang, S. W. 1991 Range perception through apparent image speed in freely flying honeybees. Vis. Neurosci. 6, 519-535. (doi:10.1017/S095252380000136X)
61. Srinivasan, M. V., Zhang, S. W. & Chandrashekara, K. 1993 Evidence for two distinct movement detecting mechanisms inc insect vision. Naturwissenschaften 80, 38-41. (doi:10.1007/BF01139758) (Pubitemid 23034492)
62. Si, A., Srinivasan, M. V. & Zhang, S. 2003 Honeybee navigation: properties of the visually driven 'odometer'. J. Exp. Biol. 206, 1265-1273. (doi:10.1242/jeb.00236)
63. Baird, E., Srinivasan, M. V., Zhang, S. & Cowling, A. 2005 Visual control of flight speed in honeybees. J. Exp. Biol. 208, 3895-3905. (doi:10.1242/jeb.01818)
64. Harris, R. A., O'Carroll, D. C. & Laughlin, S. B. 2000 Contrast gain reduction in fly motion adaptation. Neuron 28, 595-606. (doi:10.1016/S0896- 6273(00)00136-7)