Temporal interaction between an artificial orchestra conductor and human musicians

Computers in Entertainment

Volumn 6, Issue 4, 2008, Pages

  Reidsma, Dennis ^a   Nijholt, Anton ^a   Bos, Pieter ^a  

^a UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Continuous interaction; Embodied agents; Virtual conductor

Indexed keywords

CONTINUOUS INTERACTION; EMBODIED AGENTS; VIRTUAL CONDUCTOR; VIRTUAL HUMANS; VIRTUAL ORCHESTRAS;

VIRTUAL REALITY;

EID: 67549113794     PISSN: 15443574     EISSN: 15443981     Source Type: Journal    
DOI: 10.1145/1461999.1462005     Document Type: Article

References (50)

1. ALONSO, M., DAVID, B., AND RICHARD, G. 2004. Tempo and beat estimation of musical signals. In Proceedings International Conference on Music Information Retrieval. 158-163.
2. BAILENSON, J. AND YEE, N. 2005. Digital chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological Science 16, 10 (Oct.), 814-819.
3. BOS, P. 2007. Interacting with a virtual conductor. M. S. thesis, Human Media Interaction, University of Twente, NL.
4. BROWN, J. 1991. Calculation of a constant q spectral transform. J. Acoustical Society of America 89, 1, 425-434.
5. CARSE, A. 1935. Orchestral Conducting. Augener Ltd.
6. CROWN, C. L. 1991. Coordinated interpersonal timing of vision and voice as a function of interpersonal attraction. J. Language Social Psychology 10, 1, 29-46.
7. DANNENBERG, R. AND HU, N. 2003. Polyphonic audio matching for score following and intelligent audio editors. In Proceedings of the Interactional Computer Music Conference. International Computer Music Association, 27-33.
8. DIPAOLA, S. AND ARYA, A. 2006. Emotional remapping of music to facial animation. In Proceedings of the ACM SIGGRAPH Symposium on Videogames (Sandbox '06). ACM, New York, 143-149.
9. DIXON, S. 2003. On the analysis of musical expression in audio signals. In Storage and Retrieval for Media Databases, SPIE vol. 5021. 122-132.
10. DIXON, S. 2005. Live tracking of musical performances using on-line time warping. In Proceedings of the 8th International Conference on Digital Audio Effects. 92-97.
11. FUELBERTH, R. J. V. 2004. The effect of various left-hand conducting gestures on perceptions of anticipated vocal tension in singers. Int. J. Research Choral Singing 2, 1 (Jan.), 27-38.
12. GALKIN, E. W. 1988. A History off Orchestral Conducting: In Theory and Practice. Stuyvesant, New York.
13. GOTO, M. 2001. An audio-based real-time beat tracking system for music with or without drum-sounds. J. New Music Research 3, 2 (June), 159-171.
14. GOUYON, F. and DIXON, S. 2005. A review of automatic rhythm description systems. Computer Music J. 29, 1 (Feb.), 34-54.
15. GOUYON, F., KLAPURI, A., DIXON, S., ALONSO, M., TZANETAKIS, G., UHLE, C., AND CANO, P. 2006. An experimental comparison of audio tempo induction algorithms. IEEE Trans. Speech Audio Processing 14, 5 (Sept.), 1832-1844.
16. ILMONEN, T. AND TAKALA, T. 1999. Conductor following with artificial neural networks. In Proceedings of the International Computer Music Conference (ICMC, Beijing), 367-370.
17. JANSEN, L., VAN DIJK, B., AND RETRA, J. 2006. Musical multimodal child computer interaction. In Proceedings of the 2006 Conference on Interaction Design and Children (IDC). ACM, New York, 163-164.
18. KLAPURI, A., ERONEN, A., AND ASTOLA, J. 2006. Analysis of the meter of acoustic musical signals. IEEE Trans. Speech Audio Processing 14, 1 (Jan.), 342-355.
19. KOLESNIK, P. AND WANDERLEY, M. 2004. Recognition, analysis and performance with expressive conducting gestures. In Proceedings of the International Computer Music Conference (ICMC).
20. LEE, E., GRÜLL, I., KIEL, H., AND BORCHERS, J. 2006. CONGA: A framework for adaptive conducting gesture analysis. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME, IRCAM, Centre Pompidou, Paris), 260-265.
21. LEE, M., GARNETT, G., AND WESSEL, D. 1992. An adaptive conductor follower. In Proceedings of the International Computer Music Conference. International Computer Music Association, 454-455.
22. MANCINI, M., BRESIN, R., AND PELACHAUD, C. 2007. A virtual head driven by music expressivity. IEEE Trans. Audio, Speech, and Language Processing 15, 6, 1833-1841.
23. MICHALOWSKI, M. P., SABANOVIC, S., AND KOZIMA, H. 2007. A dancing robot for rhythmic social interaction. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), ACM, New York, 89-96.
24. MURPHY, D., ANDERSEN, T. H., AND JENSEN, K. 2003. Conducting audio files via computer vision. In Proceedings of the GW03. 529-540.
25. NAGAOKA, C., KOMORI, M., AND YOSHIKAWA, S. 2007. Embodied synchrony in conversation. In Conversational Informatics: An Engineering Approach, L. Nishida (ed.), Wiley, 331-351.
26. NAKRA, T. M. 2000. Inside the conductor's jacket. Ph.D. dissertation, MIT Media Laboratory. Cambridge, MA.
27. NIJHOLT, A., REIDSMA, D., VAN WELBERGEN, H., OP DEN AKKER, H., AND RUTTKAY, Z. M. 2008. Mutually coordinated anticipatory multimodal interaction. In Proceedings of the COST. Springer, Berlin.
28. PARDO, B. AND BIRMINGHAM, W. P. 2005. Modeling form for on-line following of musical performances. In AAAI. 1018-1023.
29. POGGI, I. 2002. The lexicon of the conductor's face. In Language, Vision and Music. John Benjamins, 271-284.
30. PRAUSNITZ, F. 1983. Score and Podium: A Complete Guide to Conducting. W. W. Norton, New York.
31. RAMSEYER, F. AND TSCHACHER, W. 2008. Synchrony in dyadic psychotherapy sessions. In Simultaneity: Temporal Structures and Observer Perspectives. World Scientific, Singapore, Ch. 18.
32. RAPHAEL, C. 2004. A hybrid graphical model for aligning polyphonic audio with musical scores. In Proceedings of the International Conference on Music Information Retrieval.
33. REIDSMA, D., VAN WELBERGEN, H., POPPE, R., BOS, P., AND NIJHOLT, A. 2006. Towards bidirectional dancing interaction. In Lecture Notes in Computer Science 4161, R. Harper et al. (eds.) Springer, Berlin, 1-12.
34. ROBINS, B., DAUTENHAHN, K., NEHANIV, C. L., MIRZA, N A., FRANCOIS, D., AND OLSSON, L. 2005. Sustaining interaction dynamics and engagement in dyadic child-robot interaction kinesics: Lessons learnt from an exploratory study. In Proceedings of the 14th IEEE International Workshop on Robot and Human Interactive Communication (ROMAN).
35. rd ed.), Schirmer.
36. RUTTKAY, Z. M., HUANG, A., AND ELIËNS, A. 2003. The conductor: Gestures for embodied agents with logic programming. In Proceedings of the 2nd Hungarian Computer Graphics Conference. 9-16.
37. SCHEIRER, E. D. 1998. Tempo and beat analysis of acoustic musical signals. J. Acoustical Society America 103, 1 (Jan.), 558-601.
38. SCHÖGLER, B. 1998. Music as a tool in communications research. Nordic J. Music Therapy 7, 1, 40-49.
39. SCHWARZ, D., ORIO, N., AND SCHNELL, N. 2004. Robust polyphonic midi score following with hidden markov models. In Proceedings of the International Computer Music Conference (ICMC).
40. SEGEN, J. 1999. Visual conductor. In Proceedings of the ACM SIGGRAPH '99 Conference: Conference abstracts and applications. ACM, New York, 180.
41. SEPPÄNEN, J., ERONEN, A., AND HIIPAKKA, J. 2006. Joint beat and tatum tracking from music signals. In Proceedings of the 7th International Conference on Music Information Retrieval, R. Dannenberg et al. (eds.), 23-28.
42. SHIRATORI, T., NAKAZAWA, A., AND IKEUCHI, K. 2006. Dancing-to-music character animation. Computer Graphics Forum 25, 3 (Sept.), 449-458.
43. SKADSEM, J. A. 1997. Effect of conductor verbalization, dynamic markings, conductor gesture, and choir dynamic level on singers' dynamic responses. J. Research in Music Education 45, 4, 509-520.
44. SUZUKI, N., TAKEUCHI, Y., ISHII, K., AND OKADA, M. 2003. Effects of echoic mimicry using hummed sounds on human-computer interaction. Speech Communication 40, 4, 559-573.
45. TANAKA, H. AND SUZUKI, H. 2004. Dance interaction with qrio: A case study for non-boring interaction by using an entrainment ensemble model. In Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication (ROMAN), IEEE, 419-424.
46. TER MAAT, M., EBBERS, R. M., REIDSMA, D., AND NIJHOLT, A. 2008. Beyond the beat: Modeling intentions in a virtual conductor. In Proceedings of the INTETAIN. ACM, New York.
47. TOMIDA, T., ISHIHARA, A., Ueki, A., TOMARI, Y., FUKUSHIMA, K., AND INAKAGE, M. 2007. MiXer: The communication entertainment content by using "entrainment phenomenon" and "bio-feedback". In Advances in Computer Entertainment Technology, M. Inakage et al. (eds.), ACM, New York, 286-287.
48. WANG, T.-S., ZHENG, N.-N., Li, Y., Xu, Y.-Q., AND SHUM, H.-Y. 2003. Learning kernel-based HMMs for dynamic sequence synthesis. Graphical Models 65, 4, 206-221.
49. WEINBERG, G. AND DRISCOLL, S. 2006. Robot-human interaction with an anthropomorphic percussionist. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, New York, 1229-1232.
50. YIN, J., WANG, Y., AND HSU, D. 2005. Digital violin tutor: An integrated system for beginning violin learners. In Proceedings of the 13th Annual ACM International Conference on Multimedia. ACM, New York, 976-985.