Using affective and behavioural sensors to explore aspects of collaborative music making

International Journal of Human Computer Studies

Volumn 82, Issue , 2015, Pages 31-47

  Morgan, Evan ^a   Gunes, Hatice ^a   Bryan Kinns, Nick ^a  

^a QUEEN MARY UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

Affect; Collaboration; Creativity; Improvisation; Music; Psychophysiology

Indexed keywords

HARDWARE; PSYCHOPHYSIOLOGY;

AFFECT; COLLABORATION; CREATIVITY; IMPROVISATION; MUSIC;

PHYSIOLOGY;

EID: 84930677099     PISSN: 10715819     EISSN: 10959300     Source Type: Journal    
DOI: 10.1016/j.ijhcs.2015.05.002     Document Type: Article

References (77)

1. T.M. Amabile Social psychology of creativity a consensual assessment technique J. Personal. Soc. Psychol. 43 5 1982 997 1013 (http://content.apa.org/journals/psp/43/5/997)
2. T.M. Amabile, S.G. Barsade, J.S. Mueller, and B.M. Staw Affect and creativity at work Adm. Sci. Q. 50 September (3) 2005 367 403 (http://asq.sagepub.com/content/50/3/367.short)
3. M. Argyle The Psychology of Interpersonal Behaviour 1978 Penguin Books UK
4. R. Baayen, D. Davidson, and D. Bates Mixed-effects modeling with crossed random effects for subjects and items J Mem. Lang. 59 November (4) 2008 390 412 URL (http://linkinghub.elsevier.com/retrieve/pii/S0749596X07001398)
5. Barsade, S., 2002. The ripple effect: emotional contagion and its influence on group behavior. Adm. Sci. Q. (December), 644-675.
6. M. Benedek, and C. Kaernbach A continuous measure of phasic electrodermal activity J. Neurosci. Methods 190 June (1) 2010 80 91 URL (http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2892750&tool=pmcentrez&rendertype=abstract)
7. Bevilacqua, F., Fels, S., Jensenius, A.R., Lyons, M.J., Schnell, N., Tanaka, A., 2013. SIG NIME: music, technology, and human-computer interaction. In: CHI '13 Extended Abstracts on Human Factors in Computing Systems - CHI EA '13. ACM Press, New York, NY, USA, p. 2529. URL (http://dl.acm.org/citation.cfm?doid=2468356.2468822).
8. M. Biasutti, and L. Frezza Dimensions of music improvisation Creat. Res. J. 21 May (2-3) 2009 232 242
9. Bränzel, A., Holz, C., Hoffmann, D., Schmidt, D., Knaust, M., Lühne, P., Meusel, R., Richter, S., Baudisch, P., 2013. GravitySpace. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13. ACM Press, New York, NY, USA, p. 725. URL (http://dl.acm.org/citation.cfm?doid=2470654.2470757).
10. N. Bryan-Kinns Mutual engagement and collocation with shared representations Int. J. Human-Comput. Stud. 71 January (1) 2013 76 90 URL (http://linkinghub.elsevier.com/retrieve/pii/S1071581912000328)
11. N. Bryan-Kinns, and F. Hamilton Identifying mutual engagement Behav. Inf. Technol. 31 February (2) 2012 101 125
12. T. Budzynski, H. Budzynski, J. Evans, and A. Abarbanel Introduction to Quantitative EEG and Neurofeedback: Advanced Theory and Applications 2nd ed. 2008 Academic Press Massachusetts, US
13. Carlile, J., Hartmann, B., 2005. OROBORO: a collaborative controller with interpersonal haptic feedback. In: New Interfaces for Musical Expression (NIME05), pp. 250-251.
14. Castellano, G., Villalba, S.D., Camurri, A., 2007. Recognising human emotions from body movement and gesture dynamics. In: Affective Computing and Intelligent Interaction (ACII07). Springer-Verlag, Lisbon, Portugal, pp. 71-82. URL (http://www.springerlink.com/index/aj246lvw0635g238.pdf).
15. M. Chaouachi, and C. Frasson Exploring the relationship between learner EEG mental engagement and affect V. Aleven, J. Kay, J. Mostow, Intelligent Tutoring Systems 2010 Springer Berlin, Heidelberg 291 293
16. T.L. Chartrand, and J.L., Lakin The antecedents and consequences of human behavioral mimicry Annu. Rev. Psychol 64 September 2013 285 308 http://dx.doi.org/10.1146/annurev-psych-113011-143754. URL (http://www.ncbi.nlm.nih.gov/pubmed/23020640)
17. M. Csikszentmihalyi, and K. Rathunde The Measurement of Flow in Everyday Life: Toward a Theory of Emergent Motivation vol. 40 1993 University of Nebraska Press Lincoln, NE, US pp. 57-97
18. F. Cummins Gaze and blinking in dyadic conversation a study in coordinated behaviour among individuals Lang. Cogn. Process. 27 December (10) 2012 1525 1549
19. C.K.W. De Dreu, M. Baas, and B.A. Nijstad Hedonic tone and activation level in the mood-creativity link toward a dual pathway to creativity model J. Person. Soc. Psychol. 94 May (5) 2008 739 756 URL (http://www.ncbi.nlm.nih.gov/pubmed/18444736)
20. O. de Manzano, T. Theorell, L. Harmat, and F. Ullén The psychophysiology of flow during piano playing Emotion (Washington D.C.) 10 June (3) 2010 301 311 URL (http://www.ncbi.nlm.nih.gov/pubmed/20515220)
21. A. Delorme, and S. Makeig EEGLAB an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis J. Neurosci. Methods 134 March (1) 2004 9 21 URL (http://www.ncbi.nlm.nih.gov/pubmed/15102499)
22. A. Dietrich, and R. Kanso A review of EEG, ERP, and neuroimaging studies of creativity and insight Psychol. Bull. 136 October (5) 2010 822 848 URL (http://www.ncbi.nlm.nih.gov/pubmed/20804237)
23. S. Dixon Automatic extraction of tempo and beat from expressive performances J. New Music Res. 30 March (1) 2001 39 58
24. Edwards, L., 2013. Ford S-MAX Concept will Measure Your Heart Rate from the Driver's Seat. Online (September). URL (http://www.pocket-lint.com/news/123510-ford-s-max-concept-will-measure-your-heart-rate-from-the-driver-s-seat)(accessed 25.09.13).
25. H. Egermann, M.T. Pearce, G.A. Wiggins, and S. McAdams Probabilistic models of expectation violation predict psychophysiological emotional responses to live concert music Cogn. Affect. Behav. Neurosci. 13 September (3) 2013 533 553
26. Fencott, R., Bryan-Kinns, N., 2010. Hey Man, you re invading my personal space! Privacy and awareness in collaborative music. In: New Interfaces for Musical Expression (NIME10), pp. 198-203.
27. Gabora, L., 2002. Cognitive mechanisms underlying the creative process. In: Proceedings of the Fourth Conference on Creativity & Cognition - C&C '02. ACM Press, New York, NY, USA, pp. 126-133. URL (http://portal.acm.org/citation.cfm?doid=581710.581730).
28. Glowinski, D., Gnecco, G., Piana, S., Camurri, A., 2013. Expressive non-verbal interaction in string quartet. In: Human Association Conference on Affective Computing and Intelligent Interaction (ACII'13). IEEE Computer Society, Geneva, Switzerland, pp. 233-238.
29. H. Gunes, and B. Schuller Categorical and dimensional affect analysis in continuous input current trends and future directions Image Vis. Comput. 31 July (2) 2013 120 136 URL (http://linkinghub.elsevier.com/retrieve/pii/S0262885612001084)
30. Hadjakos, A., Großhauser, T., Goebl, W., 2013. Motion analysis of music ensembles with the kinect. In: International Conference on New Interfaces for Musical Expression (NIME), KAIST, Daejeon, Korea, pp. 106-110.
31. Healey, P.G.T., Leach, J., Bryan-kinns, N., 2005. Inter-play: understanding group music improvisation as a form of everyday interaction. In: Proceedings of Less is More: Simple Computing in an Age of Complexity.
32. B.A. Hennessey, T.M. Amabile, and J.S. Mueller Consensual assessment M. Runco, S. Pritzker, Encyclopedia of Creativity vol. 1 2011 Academic Press San Diego 253 260
33. Jamison, K.R., Dec. 1996. Touched with Fire: Manic-Depressive Illness and the Artistic Temperament. Simon & Schuster Ltd., New York.
34. P.E. Keller, D. Fabian, R. Timmers, and E. Schubert Ensemble performance interpersonal alignment of musical expression Expr. Music Perform.: Empir. Approaches Across Styles Cult. 2014 260 282
35. A. Kendon Some functions of gaze-direction in social interaction Acta Psychol. 26 January 1967 22 63 (http://linkinghub.elsevier.com/retrieve/pii/0001691867900054)
36. Kendon, A., 2010. Spacing and orientation in co-present interaction. In: Lecture Notes in Computer Science, vol. 5967, pp. 1-15.
37. Kenny, B.J., Gellrich, M., 2002. Improvisation. In: Parncutt, R., McPherson, G. (Eds.), The Science and Psychology of Music Performance: Creative Strategies for Teaching and Learning. Oxford University Press, USA, pp. 117-134 (Chapter 8).
38. J. Kim, and E. André Emotion recognition based on physiological changes in music listening IEEE Trans. Pattern Anal. Mach. Intell. 30 December (12) 2008 2067 2083 URL (http://www.ncbi.nlm.nih.gov/pubmed/18988943)
39. K.H. Kim, S.W. Bang, and S.R. Kim Emotion recognition system using short-term monitoring of physiological signals Med. Biol. Eng. Comput. 42 May (3) 2004 419 427
40. Knapp, R.B., Jaimovich, J., Coghlan, N., Sep. 2009. Measurement of motion and emotion during musical performance. In: 2009 Third International Conference on Affective Computing and Intelligent Interaction and Workshops. pp. 1-5. URL (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5349469).
41. Kuhara, Y., Kobayashi, D., 2011. Kinetic particles synthesizer using multi-touch screen interface of mobile devices. In: New Interfaces for Musical Expression (NIME11), pp. 136-137 (June).
42. Kuyken, B., Verstichel, W., Bossuyt, F., Vanfleteren, J., Demey, M., Leman, M., 2008. The HOP sensor: wireless motion sensor. In: New Interfaces for Musical Expression (NIME08), pp. 229-232.
43. Limb, C.J., Braun, A.R., 2008. Neural substrates of spontaneous musical performance: an FMRI study of jazz improvisation. PloS One 3(January 2). URL (http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2244806&tool=pmcentrez&rendertype=abstract).
44. L.-O. Lundqvist, F. Carlsson, P. Hilmersson, and P.N. Juslin Emotional responses to music experience, expression, and physiology Psychol. Music 37 August (1) 2008 61 90 URL (http://pom.sagepub.com/cgi/doi/10.1177/0305735607086048)
45. R.L. Mandryk, and M.S. Atkins A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies Int. J. Human-Comput. Stud. 65 Apr. (4) 2007 329 347 URL (http://linkinghub.elsevier.com/retrieve/pii/S1071581906001881)
46. R.L. Mandryk, K.M. Inkpen, and T.W. Calvert Using psychophysiological techniques to measure user experience with entertainment technologies Behav. Inf. Technol. 25 March (2) 2006 141 158
47. A. Mazur, E. Rosa, M. Faupel, J. Heller, R. Leen, and B. Thurman Physiological aspects of communication via mutual gaze Am. J. Sociol. 86 July (1) 1980 50 74 URL (http://www.ncbi.nlm.nih.gov/pubmed/7435770)
48. Mealla, S., Väljamäe, A., Bosi, M., Jordà, S., 2011. Listening to your brain: implicit interaction in collaborative music performances. In: The International Conference on New Interfaces for Musical Expression (NIME), Oslo, Norway, pp. 149-154 (June). URL (http://mtg.upf.es/system/files/publications/listening-to-your-brain-camera-ready.pdf).
49. Mitchell, T., Madgwick, S., Heap, I., 2012. Musical interaction with hand posture and orientation: a toolbox of gestural control mechanisms. In: New Interfaces for Musical Expression (NIME12).
50. I. Monson Saying Something: Jazz Improvisation and Interaction. Chicago Studies in Ethnomusicology 1996 University of Chicago Press Chicago, IL URL (http://books.google.co.uk/books?id=Pus8-ccYWlMC)
51. Morgan, E.L., Gunes, H., Harris, D., 2013. Gesturing at architecture: experiences and issues with new forms of interaction. In: Proceedings of CHI'13 Workshop on Experiencing Interactivity in Public Spaces (EIPS), Paris, France, pp. 117-121. URL (http://www.cs.tut.fi/ihte/EIPS-workshop-CHI13/papers.shtml).
52. L. Nowicki, W. Prinz, M. Grosjean, B.H. Repp, and P.E. Keller Mutual adaptive timing in interpersonal action coordination Psychomusicol.: Music Mind Brain 23 1 2013 6 20
53. Oertel, C., Wlodarczak, M., Edlund, J., Wagner, P., Gustafson, J., 2012. Gaze patterns in turn-taking. In: Proceedings of Interspeech, vol. 29 (February).
54. M. Pantic, R. Cowie, F. D'Errico, D. Heylen, M. Mehu, C. Pelachaud, I. Poggi, M. Schroeder, and A. Vinciarelli Social signal processing the research agenda Vis. Anal. Humans 23 2011 511 538
55. M. Patel, S. Lal, D. Kavanagh, and P. Rossiter Applying neural network analysis on heart rate variability data to assess driver fatigue Expert Syst. Appl. 38 June (6) 2011 7235 7242 URL (http://linkinghub.elsevier.com/retrieve/pii/S0957417410013916)
56. A. Pentland Socially aware, computation and communication Computer 38 March (3) 2005 33 40 URL (http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1413116)
57. Price, J., 2011. Plug in to movies that know how you're feeling. Online (March). URL (http://www.newscientist.com/blogs/culturelab/2011/03/plug-in-for-emotional-response-cinema.html)(accessed 10.12.12).
58. Pronin, E., 2009. The introspection illusion. In: Advances in Experimental Social Psychology, vol. 41. Elsevier Inc., USA, pp. 1-67 (Chapter 1).
59. N. Ravaja, T. Saari, M. Salminen, J. Laarni, and K. Kallinen Phasic emotional reactions to video game events a psychophysiological investigation Media Psychol. 8 November (4) 2006 343 367
60. R.G. Reed, A.K. Randall, J.H. Post, and E.a. Butler Partner influence and in-phase versus anti-phase physiological linkage in romantic couples Int. J. Psychophysiol.: Off. J. Int. Organ. Psychophysiol. 88 June (3) 2013 309 316 URL (http://www.ncbi.nlm.nih.gov/pubmed/22922526)
61. D. Reidsma, M. Radha, and A. Nijholt Mediated interactions and musical expression - a survey N. Lee, Digital Da Vinci: Computers in Music 2014 Springer New York, NY, USA 79 98
62. H. Sacks, E. Schegloff, and G. Jefferson A simplest systematics for the organization of turn-taking for conversation Language 50 4 1974 696 735
63. E. Sariyanidi, H. Gunes, and A. Cavallaro Automatic analysis of facial affect a survey of registration, representation and recognition IEEE Trans. Pattern Anal. Mach. Intell. 8828 AUGUST 2013 2014 1 URL (http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6940284)
64. R.K. Sawyer Group Creativity: Music, Theater, Collaboration 2003 Psychology Press London, UK
65. R.K. Sawyer Music and conversation D. Miell, R. MacDonald, D.J. Hargreaves, Musical Communication 2005 Oxford University Press USA (July) (Chapter 3)
66. L.A. Schmidt, and L.J. Trainor Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions Cogn. Emot. 15 Jul. (4) 2001 487 500
67. F. Schneider, W. Grodd, U. Weiss, U. Klose, K.R. Mayer, T Nägele, and R.C. Gur Functional MRI reveals left amygdala activation during emotion Psychiatry Res.: Neuroimaging 76 December (2-3) 1997 75 82
68. M.F. Schober, and N. Spiro Jazz improvisers' shared understanding a case study Front. Psychol. 5 August 2014 808 URL (http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=4126153&tool=pmcentrez&rendertype=abstract)
69. F.a. Seddon Modes of communication during jazz improvisation Br. J. Music Educ. 22 Mar. (1) 2005 47 61 URL (http://www.journals.cambridge.org/abstract-S0265051704005984)
70. D.K. Simonton Creativity, task complexity, and intuitive versus analytical problem solving Psychol. Rep. 37 October (2) 1975 351 354 URL (http://www.ncbi.nlm.nih.gov/pubmed/1187912)
71. Vera, B., Chew, E., Healey, P., 2013. A study of ensemble performance under restricted line of sight. In: Proceedings of the International Conference on Music Information Retrieval, Curitiba, Brazil. URL (http://www.ppgia.pucpr.br/ismir2013/wp-content/uploads/2013/09/171-Paper.pdf).
72. A. Vinciarelli, M. Pantic, and H. Bourlard Social signal processing survey of an emerging domain Image Vis. Comput. 27 November (12) 2009 1743 1759 URL (http://linkinghub.elsevier.com/retrieve/pii/S0262885608002485)
73. A. Vinciarelli, M. Pantic, D. Heylen, C. Pelachaud, I. Poggi, F. D'Errico, and M. Schroeder Bridging the gap between social animal and unsocial machine a survey of social signal processing IEEE Trans. Affect. Comput. 3 January (1) 2012 69 87 URL (http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5989788)
74. M. Walter, B. Eilebrecht, T. Wartzek, and S. Leonhardt The smart car seat personalized monitoring of vital signs in automotive applications Person. Ubiquitous Comput. 15 January (7) 2011 707 715
75. B.T. West, K.B. Welch, and A.T. Galecki Linear Mixed Models: A Practical Guide Using Statistical Software 2007 Chapman and Hall/CRC London, UK
76. G.B. Wilson, and R.A.R. MacDonald The sign of silence negotiating musical identities in an improvising ensemble Psychol. Music 40 August (5) 2012 558 573
77. H. Zhou, T. Stone, H. Hu, and N. Harris Use of multiple wearable inertial sensors in upper limb motion tracking Med. Eng. Phys. 30 Jan. (1) 2008 123 133 URL (http://www.ncbi.nlm.nih.gov/pubmed/17251049)