Music via motion: Transdomain mapping of motion and sound for interactive performances

Proceedings of the IEEE

Volumn 92, Issue 4, 2004, Pages 645-655

  Ng, Kia C ^a  

^a UNIVERSITY OF LEEDS   (United Kingdom)

Author keywords

Augmented reality; Computer vision; Gesture interface; Multimedia systems; Music; Sensors; Virtual reality

Indexed keywords

COMPUTER MUSIC; COMPUTER VISION; CORRELATION METHODS; GESTURE RECOGNITION; INTERFACES (COMPUTER); MAPPING; MOTION CONTROL; MULTIMEDIA SYSTEMS; SENSORS; SERVERS; VIRTUAL REALITY;

GESTURE ANALYSIS; GESTURE INTERFACE; MUSIC; VIDEO-BASED TRACKING;

MUSICAL INSTRUMENTS;

EID: 21244432705     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2004.825885     Document Type: Conference Paper

References (78)

1. K. C. Ng, "Music via motion," presented at the XIII CIM 2000 - Colloq. Musical Informatics, L'Aquila, Italy, 2000.
2. _, "Music via Motion (MvM): virtual and augmented performances," in Proc. Electronic Imaging: The Visual Arts and Beyond (EVA2002), pp. 1-9.
3. B. Schoner, C. Cooper, and N. Gershenfeld, "Cluster-weighted sampling for synthesis and cross-synthesis of violin family instruments," in Proc. Int. Computer Music Conf. (ICMC 2000), pp. 376-379.
4. Y. Nagashima, "Composition of 'visional legend'," presented at the Int. Workshop on Human Supervision and Control in Engineering and Music, Kassel, Germany, 2001.
5. J. A. Paradiso, K.-Y. Hsiao, J. Strickon, and P. Rice, "New sensor and music systems for large interactive surfaces," in Proc. Int. Computer Music Conf. (ICMC 2000), pp. 277-280.
6. H. Livingston, "Paradigms for the new string instrument: digital and materials technology," Int. J. Music Technol.: Organized Sound, vol. 5, no. 3, pp. 135-147, 2000.
7. A. Hunt and R. Kirk, "Mapping strategies for musical performance," in Trends in Gestural Control of Music, M. Wanderley and M. Battier, Eds. Paris, France: IRCAM - Centre Pompidou, 2000.
8. Int. Conf. New Interfaces for Musical Expression (NIME2002), Dublin, Ireland.
9. Int. Conf. Music and Gesture, Norwich, U.K., 2003.
10. E. Fléty, "AtoMIC pro: a multiple sensor acquisition device," in Proc. Int. Conf. New Interfaces for Musical Expression, 2002, pp. 96-101.
11. M. M. Wanderley and M. Battier, Eds., Trends in Gestural Control of Music. Paris, France: IRCAM - Centre Pompidou, 2000.
12. A. Mulder, "The I-cube system: moving toward sensor technology for artists," presented at the 6th Symp. Electronic Arts (ISEA 95), Montreal, QB, Canada, 1995.
13. ICMA/EMF Working Group on Interactive Systems and Instrument Design in Music (ISIDM) [Online]. Available: www.igmusic.org
14. A Real-Time Video to MIDI Macintosh Software. BigEye. [Online]. Available: http://www.steim.nl/bigeye.html
15. D. Rokeby, "The construction of experience: interface as content," in Digital Illusion: Entertaining the Future With High Technology, C.Clark Dodsworth Jr., Ed. New York: ACM, 1997.
16. _, (1995) Lecture for 'Info art'. Kwangju Biennale, Kwangju, Korea. [Online]. Available: http://www.interlog.com/drokeby/install.html
17. _, Body language: ACM SIGGRAPH art show, Atlanta, GA, 1988.
18. _, Transforming mirrors: Subjectivity and control in interactive media. [Online]. Available: http://www.interlog.com/~drokeby/mirrorsintro.html
19. T. Winkler, "Creating interactive dance with the Very Nervous System," presented at the 1997 Connecticut College Symp. Art and Technology. New London, CT, 1997.
20. _, Composing Interactive Music: Techniques and Ideas Using Max. Cambridge, MA: MIT Press, 1998.
21. Proc. AIMI Int. Workshop Kansei: The Technology of Emotion, A. Camurri, Ed., Genova, Italy, 1998.
22. A. Camurri, S. Hashimoto, M. Ricchetti, A. Ricci, K. Suzuki, R. Trocca, and G. Volpe, "EyeWeb: toward gesture and affect recognition in interactive dance and music systems," Comput. Music J., vol. 24, no. 1, pp. 57-69, 2000.
23. A. Camurri, P. Coletta, M. Peri, M. Ricchetti, A. Ricci, R. Trocca, and G. Volpe, "A real-time platform for interactive performance," presented at the Int. Computer Music Conf. (ICMC 2000), Berlin, Germany.
24. M. J. Lyons, M. Haehnel, and N. Tetsutani, "The mouthesizer: a facial gesture musical interface," in Conf. Abstract SIGGRAPH 2001, p. 230.
25. M. J. Lyons and N. Tetsutani, "Facing the music: a facial action controlled musical interface," in Proc. Conf. Human Factors in Computing Systems, 2001, pp. 309-310.
26. W. Siegel and J. Jacobsen, "The challenges of interactive dance, an overview and case study," Comput. Music J., vol. 22, no. 4, pp. 29-43, 1998.
27. _, "Composing for the digital dance interface," in Proc. Int. Computer Music Conf. (ICMC 1999), pp. 276-277.
28. W. Siegel, "Two compositions for interactive dance," in Proc. Int. Computer Music Conf. (ICMC 1999), pp. 56-59.
29. D. Rubine and P. McAvinney, "The videoharp." in Proc. Int. Computer Music Conf. (ICMC 1988), pp. 49-55.
30. A. Sato, T. Harada, S. Hashimoto, and S. Ohteru, "Singing and playing in musical virtual space," in Proc. Int. Computer Music Conf. (ICMC 1991), pp. 289-292.
31. H. Sawada, S. Ohkura, and S. Hashimoto, "Gesture analysis using 3D acceleration sensor for music control," in Proc. Int. Computer Music Conf. (ICMC 1995), pp. 257-260.
32. D. Keane and P. Gross, "The MIDI baton," in Proc. Int. Computer Music Conf. (ICMC 1989), pp. 151-154.
33. E. J. Paulos, "Personal tele-embodiment," Ph.D. dissertation, Univ. California, Berkeley, CA, 2001.
34. J. Ryan, "Effort and expression," in Proc. Int. Computer Music Conf. (ICMC 1992), pp. 414-416.
35. S. J. Morris and J. A. Paradiso, "Shoe-integrated sensor system for wireless gait analysis and real-time feedback," in Proc. 2nd Joint IEEE Engineering in Medicine and Biology Soc. and Biomedical Engineering Soc. Conf., 2002, pp. 2468-2469.
36. J. Paradiso and E. Hu, "Expressive footwear for computer-augmented dance performance," presented at the 1st Int. Symp. Wearable Computers, Cambridge, MA, 1997.
37. S. Pardue and J. A. Paradiso, "Musical navigatrics: new musical interactions with passive magnetic tags," in Proc. New Interfaces for Musical Expression 2002 Conf. (NIME-02), pp. 168-169.
38. [381 K. Suzuki, K. Tabe, and S. Hashimoto, "A mobile robot platform for music and dance performance," in Proc. Int. Computer Music Conf. 2000, pp. 539-542.
39. J. Paradiso, "Electronic music interfaces: new ways to play," IEEE Spectr., vol. 34, pp. 18-30. Dec. 1997.
40. M. M. Wanderley and P. Depalle, "Gestural control of computer-generated sound," Proc. IEEE (Special Issue on Engineering and Music: Supervisory Control and Auditory Communication), vol. 92, pp. 632-644, Apr. 2004.
41. A. Hunt, "Radical user interfaces for realtime musical control," D.Phil, dissertation. University of York, York, U.K., 1999.
42. A. Hunt and R. Kirk, "Radical user interfaces for real-time control," in Proc. 25th Euromicro Conf. (EUROMICRO '99), vol. 2, pp. 2006-2012.
43. T. Marrin-Nakra, "Searching for meaning in gestural data: interpretive feature extraction and signal processing for affective and expressive content," in Trends in Gestural Control of Music, M. Wanderley and M. Battier, Eds. Paris, France: IRCAM - Centre Pompidou, 2000.
44. _, "Inside the conductor's jacket: Analysis, interpretation and musical synthesis of expressive gesture," Ph.D. dissertation. Media Lab, Massachusetts Inst. Technol., Cambridge, MA, 2000.
45. T. Marrin and J. Paradiso, "The digital baton: a versatile performance instrument," presented at the Int. Computer Music Conf. '97, Thessaloniki, Greece.
46. M. M. Wanderley, "Gestural control of music," presented at the Int. Workshop on Human Supervision and Control in Engineering and Music, Kassel, Germany, 2001.
47. _, "Performer-instrument interaction: Applications to gestural control of sound synthesis," Ph.D. dissertation, University of Paris, Paris, France, 2001.
48. K. Warwick, "Linking human neural processes directly with technology - the future," in Proc. Int. Conf. Artificial Neural Nets and Genetic Algorithms, 1999, pp. 8-13.
49. D. Wenn, R, J. Mitchell. and M. Gabb, "Toward the control room of the future," in Proc. IEE Conf. People in Control (PIC 2001), pp. 79-85.
50. A. Hunt. M. Wanderley, and R. Kirk, "Toward a model for instrumental mapping in expert musical interaction," in Proc. Int. Computer Music Conf. (ICMC 2000), pp. 209-212.
51. S. Maruyama, "How is a symphony started by the orchestra? An empirical study for interpersonal coordination between the professional orchestral conductor and player," presented at the Int. Workshop on Human Supervision and Control in Engineering and Music, Kassel, Germany, 2001.
52. H. Morita, S. Hashimoto, and S. Ohteru, "A computer music system that follows a human conductor," IEEE Computer, vol. 24, pp. 44-53, July 1991.
53. Y. Raja, S. McKenna, and S. Gong, "Segmentation and tracking using color mixture models," in Proc. Asian Conf, Computer Vision (ACCV), vol. 1, 1998, pp. 607-614.
54. M. S. Drew, J. Wei, and Z.-N. Li, "Illumination-invariant color object recognition via compressed chromaticity histograms of normalized images." in Proc. 6th Int. Conf. Computer Vision, 1998, pp. 533-540.
55. D. Chai and K. N. Ngan, "Locating facial region of a head-and-shoulders color image," in Proc. 3rd Int. Conf. Automatic Face and Gesture Recognition, 1998, pp. 124-129.
56. J. L. Crowley and J. M. Bedrune, "Integration and control of reactive visual processes," in Proc. 3rd Eur. Conf. Computer Vision, vol. 2, 1994, pp. 47-58.
57. V. E. Devin and D. C. Hogg, "Reactive memories: An interactive talking-head," School Comput., Univ. Leeds, Leeds, U.K., Rep. 2001.09, 2001.
58. K. Sobottka and I. Pitas, "Face localization and feature extraction based on shape and color information," in Proc. IEEE Int. Conf. Image Proceeding, 1996, pp. 483-486.
59. K. C. Ng, I. Symonds, J. Scott, J. Cook, A. Bohn, and R. Sage, "Music via motion: an interactive multimedia installation with video and sensor," presented at the MAXIS: Festival of Sound and Experimental Music, Sheffield, U.K., 2002.
60. M. Das, D. M. Howard, and S. L. Smith, "Motion curves in music: the statistical analysis of MIDI data," in Proc. 25th Euromicro Conf. (EUROMICRO '99), vol. 2, pp. 2013-2019.
61. N. Johnson, "Learning object behavior models," Ph.D. dissertation. School Comput. Stud., Univ. Leeds, Leeds, U.K., 1998.
62. N. Johnson, A. Galata, and D. Hogg, "The acquisition and use of interaction behavior models." in Proc. IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition, 1998, pp. 866-871.
63. R. J. Mitchell, J. M. Bishop, D. A. Keating, and K. Dautenhahn, "Cybernetic approaches to artificial life," Kunstliche Intelligenz (Special Edition on Artificial Life), vol. 1, pp. 5-11, 2000.
64. N. I. Badler, R. Bindiganavale, J, Rourne, J. Allbeck, J. Shi, and M. Palmer, "Real time virtual humans," presented at the 4th Int. Conf. Digital Media Futures, National Museum of Photography, Film and Television, Bradford, U.K., 1999.
65. P. Volino and N. Magnenat-Thalmann, "3D fashion design and the virtual catwalk," presented at the 4th Int. Conf. Digital Media Futures, National Museum of Photography, Film and Television, Bradford, U.K., 1999.
66. V. J. Vincent. Mandala virtual reality system. VIVID Group. [Online]. Available: http://www.vividgroup.com
67. R. J. Neagle, K. C. Ng, and R. A. Ruddle, "Notation and 3D animation of dance movement," in Proc. Int. Computer Music Conf. (ICMC2002), pp. 459-462.
68. K. C. Ng, V. Sequeira, S. Butterfield, D. C. Hogg, and J. G. M. Gonçalves, "An integrated multi-sensory system for photo-realistic 3d scene reconstruction," in Proc. ISPRS Int. Symp. Real-Time Imaging and Dynamic Analysis, 1998, pp. 356-363.
69. K. C. Ng, V. Sequeira, E. Bovisio, N. Johnson, D. Cooper, J. G. M. Gonçalves, and D. C. Hogg, "Playing on a holo-stage: toward the interaction between real and virtual performers," in Digital Creativity: A Reader, C.Colin Beardon and L.Lone Malmborg, Eds. Lisse, Switzerland: Swets & Zeitlinger, 2002.
70. V. Sequeira, K. C. Ng. E. Wolfart, J. G. M. Gonçalves, and D. C. Hogg, "Automated reconstruction of 3d models from real environment," ISPRS J. Photogrammetry Remote Sensing, vol. 54, no. 1. pp. 1-22, 1999.
71. G. Johannsen, "Human supervision and control in engineering and music," presented at the Int. Workshop Human Supervision and Control in Engineering and Music, Kassel, Germany, 2001.
72. Parasite: Event for invaded and involuntary body, Stelarc. (1997). [Online]. Available: http://www.stelarc.va.com.au/parasite/index.htm
73. A. Mulder, "Design of virtual three-dimensional instruments for sound control," Ph.D. dissertation. Simon Fraser Univ., Burnaby, BC, Canada, 1998.
74. F. Babiloni, F. Cincotti, L. Lazzarini, J. del R. Millán, J. Mouriño, M. Varsta, J. Heikkonen, L. Bianchi, and M. G. Marciani, "Linear classification of low-resolution EEC patterns produced by imagined hand movements," IEEE Trans. Rehab. Eng. (Special Issue on Brain - Computer Interfaces), vol. 8, pp. 186-188, June 2000.
75. L. Tarabella and B. Graziano, "Giving expression to multimedia performance," presented at the ACM Multimedia Workshop, Marnia Del Rey, CA, 2000.
76. M. Feldmeier, M. Malinowski, and J. A. Paradiso, "Large group musical interaction using disposable wireless motion sensors," in Proc. Int. Computer Music Conf. 2002, pp. 83-87.
77. T. Anderson, "Using music performance software with flexible control interfaces for live performance by severely disabled musicians," in Proc. 25th Euromicro Conf. (EUROMICRO '99). vol. 2, pp. 2020-2029.
78. L. M. Taylor and J. M. Bishop, "Computer mediated communication use by the deaf," in Proc. World Organization Systems and Cybernetics, 1999, pp. 185-189.