Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction

Visual Computer

Volumn 27, Issue 1, 2011, Pages 57-66

  Hulusić, Vedad ^a   Debattista, Kurt ^a   Aggarwal, Vibhor ^a   Chalmers, Alan ^a  

^a UNIVERSITY OF WARWICK   (United Kingdom)

Author keywords

Cross modal; High fidelity rendering; Perception

Indexed keywords

ANIMATION; COMPUTER GRAPHICS; SENSORY PERCEPTION;

CROSS-MODAL; CROSS-MODAL INTERACTION; ENTERTAINMENT INDUSTRY; GRAPHICS ALGORITHMS; GRAPHICS HARDWARE; HIGH-FIDELITY RENDERINGS; INTERACTIVE ENVIRONMENTS; PERFORMANCE REQUIREMENTS;

INDUSTRY;

EID: 84856646619     PISSN: 01782789     EISSN: None     Source Type: Journal    
DOI: 10.1007/s00371-010-0514-2     Document Type: Article

References (54)

1. Alais, D., Morrone, C., Burr, D.: Separate attentional resources for vision and audition. Proc. Biol. Sci. 273(1592), 1339-1345 . doi:10.1098/rspb.2005.3420
2. Allport, D.A., Antonis, B., Reynolds, P.: On the division of attention: a disproof of the single channel hypothesis. Q. J. Exp. Psychol. 24(2), 225-235 (1972)
3. Arif, M., Ohtaki, Y., Nagatomi, R.H.I.: Estimation of the effect of cadence on gait stability in young and elderly people using approximate entropy technique. Meas. Sci. Rev. 4, 29-40 (2004)
4. Bonnel, A.M., Hafter, E.R.: Divided attention between simultaneous auditory and visual signals. Percept. Psychophys. 60(2), 179-190 (1998)
5. Bonneel, N., Suied, C., Viaud-Delmon, I., Drettakis, G.: Bimodal perception of audio-visual material properties for virtual environments. ACM Trans. Appl. Percept. (2009). http://www-sop. inria.fr/reves/Basilic/2009/BSVD09
6. Bonneel, N., Suied, C., Viaud-Delmon, I., Drettakis, G.: Bimodal perception of audio-visual material properties for virtual environments. ACM Trans. Appl. Percept. 7(1), 1-16 (2010). http://doi.acm.org/10.1145/1658349. 1658350
7. Burr, D., Alais, D.: Combining visual and auditory information. Prog. Brain Res. 155, 243-258 (2006). doi:10.1016/S0079-6123(06)55014-9
8. Cater, K., Chalmers, A., Ward, G.: Detail to attention: exploiting visual tasks for selective rendering. In: EGRW '03: Proceedings of the 14th EurographicsWorkshop on Rendering Techniques, pp. 270-280. Eurographics Association, Leuven, Belgium (2003)
9. Choe, C.S., Welch, R.B., Gilford, R.M., Juola, J.F.: The "ventriloquist effect": visual dominance or response bias? Percept. Psychophys. 18(1), 55-60 (1975)
10. Dumont, R., Pellacini, F., Ferwerda, J.A.: Perceptually-driven decision theory for interactive realistic rendering. ACM Trans. Graph. 22(2), 152-181 (2003). http://doi.acm.org/10.1145/636886.636888
11. Duncan, J., Martens, S., Ward, R.: Restricted attentional capacity within but not between sensory modalities. Nature 387(6635), 808-810 (1997)
12. Funkhouser, T.A., Séquin, C.H.: Adaptive display algorithm for interactive frame rates during visualization of complex virtual environments. In: SIGGRAPH '93: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, pp. 247-254. ACM, New York (1993). http://doi.acm.org/10.1145/166117.166149
13. Gebhard, J.W., Mowbray, G.H.: On discriminating the rate of visual flicker and auditory flutter. Am. J. Psychol. 72, 521-529 (1959)
14. Getzmann, S.: The effect of brief auditory stimuli on visual apparent motion. Perception 36(7), 1089-1103 (2007)
15. Grelaud, D., Bonneel, N.,Wimmer, M., Asselot, M., Drettakis, G.: Efficient and practical audio-visual rendering for games using crossmodal perception. In: Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (2009). http://www-sop.inria.fr/reves/Basilic/2009/GBWAD09
16. Howard, I.P., Templeton,W.B.: Human Spatial Orientation.Wiley, London (1966)
17. Hulusic, V., Aranha, M., Chalmers, A.: The influence of crossmodal interaction on perceived rendering quality thresholds. In: Skala, V. (ed.) WSCG 2008 Full Papers Proceedings, pp. 41-48
18. Hulusic, V., Czanner, G., Debattista, K., Sikudova, E., Dubla, P., Chalmers, A.: Investigation of the beat rate effect on frame rate for animated content. In: Hauser, H. (ed.) Spring Conference on Computer Graphics 2009, pp. 167-174. ACM SIGGRAPH Press, New York (2009)
19. Hulusic, V., Debattista, K., Aggarwal, V., Chalmers, A.: Exploiting audio-visual cross-modal interaction to reduce computational requirements in interactive environments. In: Proceedings of the IEEE conference on Games and VirtualWorlds for Serious Applications. IEEE Computer Society, Los Alamitos (2010)
20. Humphreys, G.W., Bruce, V.: Visual Cognition: Computational, Experimental and Neuropsychological Perspectives. Lawrence Erlbaum Associates Ltd, East Sussex, BN3 2FA, UK (1989)
21. James, W.: The Principles of Psychology. Holt, New York (1890)
22. Kajiya, J.T.: The rendering equation. SIGGRAPH Comput. Graph. 20(4), 143-150 (1986)
23. Kamitani, Y., Shimojo, S.: Sound-induced visual "rabbit". J. Vis. 1(3), 478-478 (2001). http://journalofvision.org/1/3/478
24. Kayser, C., Petkov, C.I., Lippert, M., Logothetis, N.K.: Mechanisms for allocating auditory attention: an auditory saliency map. Curr. Biol. 15(21), 1943-1947 (2005). doi:10.1016/j.cub.2005.09. 040
25. Keetels, M., Vroomen, J.: Tactile-visual temporal ventriloquism: no effect of spatial disparity. Percept. Psychophys. 70(5), 765-771
26. Kelly, M.C., Tew, A.I.: The continuity illusion in virtual auditory space. In: Proc. of AES 112th Convention. Munich, Germany (2002)
27. Larsen, A., McIlhagga, W., Baert, J., Bundesen, C.: Seeing or hearing? Perceptual independence, modality confusions, and crossmodal congruity effects with focused and divided attention. Percept. Psychophys. 65(4), 568-574 (2003)
28. Lecuyer, A., Burkhardt, J.M., Henaff, J.M., Donikian, S.: Camera motions improve the sensation of walking in virtual environments. In: VR '06: Proceedings of the IEEE conference on Virtual Reality, pp. 11-18. IEEE Computer Society, Washington (2006). doi:10.1109/VR.2006.31
29. Mack, A., Rock, I.: Inattentional Blindness. MIT Press, Cambridge (1998)
30. Massaro, D.W., Warner, D.S.: Dividing attention between auditory and visual perception. Percept. Psychophys. 21(6), 569-574
31. Mastoropoulou, G.: The effect of audio on the visual perception of high-fidelity animated 3d computer graphics. PhD in Computer science, University of Bristol (2006)
32. Mastoropoulou, G., Chalmers, A.: The effect of music on the perception of display rate and duration of animated sequences: an experimental study. In: TPCG '04: Proceedings of the Theory and Practice of Computer Graphics 2004 (TPCG'04), pp. 128-134. IEEE Computer Society, Washington (2004). doi:10.1109/TPCG.2004.34
33. Mastoropoulou, G., Debattista, K., Chalmers, A., Troscianko, T.: Auditory bias of visual attention for perceptually-guided selective rendering of animations. In: GRAPHITE '05: Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, pp. 363-369. ACM Press, New York (2005). http://doi.acm.org/10.1145/ 1101389.1101462
34. Mastoropoulou, G., Debattista, K., Chalmers, A., Troscianko, T.: The influence of sound effects on the perceived smoothness of rendered animations. In: APGV '05: Proceedings of the 2nd Symposium on Applied Perception in Graphics and Visualization, pp. 9-15. ACM, New York (2005). http://doi.acm.org/10.1145/ 1080402. 1080404
35. Mcgurk, H.,Macdonald, J.: Hearing lips and seeing voices. Nature 264(5588), 746-748 (1976). doi:10.1038/264746a0
36. Moeck, T., Bonneel, N., Tsingos, N., Drettakis, G., Viaud-Delmon, I., Alloza, D.: Progressive perceptual audio rendering of complex scenes. In: I3D '07: Proceedings of the 2007 Symposium on Interactive 3D Graphics and Games, pp. 189-196. ACM, New York (2007). http://doi.acm.org/10.1145/1230100.1230133
37. Moore, B.C.: An Introduction to the Psychology of Hearing, 2nd edn. Academic Press, San Diego (1982)
38. Morein-Zamir, S., Soto-Faraco, S., Kingstone, A.: Auditory capture of vision: examining temporal ventriloquism. Brain Res. Cogn. Brain Res. 17(1), 154-163 (2003)
39. Parke, R., Chew, E., Kyriakakis, C.: Quantitative and visual analysis of the impact of music on perceived emotion of film. Comput. Entertain. 5(3), 5 (2007). http://doi.acm.org/10.1145/1316511. 1316516
40. Ramic-Brkic, B., Chalmers, A., Boulanger, K., Pattanaik, S., Covington, J.: Cross-modal affects of smell on the real-time rendering of grass. In: Hauser, H. (ed.) Spring Conference on Computer Graphics 2009, pp. 175-179. ACM SIGGRAPH Press, New York (2009)
41. Recanzone, G.H.: Auditory influences on visual temporal rate perception. J. Neurophysiol. 89, 1078-1093 (2003)
42. Sekuler, R., Sekuler, A.B., Lau, R.: Sound alters visual motion perception. Nature 385(6614), 308 (1997). http://dx.doi.org/10. 1038/385308a0
43. Shams, L., Kamitani, Y., Shimojo, S.: What you see is what you hear. Nature 408, 788+ (2000)
44. Shams, L., Kamitani, Y., Shimojo, S.: Visual illusion induced by sound. Brain Res. Cogn. Brain Res. 14(1), 147-152 (2002)
45. Shams, L., Kamitani, Y., Shimojo, S.: Modulations of visual perception by sound. In: Calvert, G.A., Spence, C., Stein, B.E. (eds.) The Handbook of Multisensory Processes, pp. 27-33 (2004)
46. Shipley, T.: Auditory flutter-driving of visual flicker. Science 145, 1328-1330 (1964)
47. Simons, D., Chabris, C.: Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception 28, 1059-1074
48. Suied, C., Bonneel, N., Viaud-Delmon, I.: Integration of auditory and visual information in the recognition of realistic objects. Exp. Brain Res. (2009)
49. Tsingos, N., Gallo, E., Drettakis, G.: Perceptual audio rendering of complex virtual environments. ACM Trans. Graph. 23(3), 249-258 (2004). http://doi.acm.org/10.1145/1015706.1015710
50. Vroomen, J., de Gelder, B.: Perceptual effects of cross-modal stimulation: ventriloquism and the freezing phenomenon. In: Calvert, G.A., Spence, C., Stein, B.E. (eds.) The Handbook of Multisensory Processes, pp. 140-150 (2004)
51. Vroomen, J., Bertelson, P., Gelder, B.D.: A visual influence in the discrimination of auditory location (1998)
52. Wada, Y., Kitagawa, N., Noguchi, K.: Audio-visual integration in temporal perception. Int. J. Psychophysiol. 50(1-2), 117-124
53. Welch, R.B., Warren, D.H.: Immediate perceptual response to intersensory discrepancy. Psychol. Bull. 88(3), 638-667 (1980)
54. Yee, H., Pattanaik, S., Greenberg, D.P.: Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments. ACM Trans. Graph. 20(1), 39-65 (2001). http://doi.acm. org/10.1145/383745.383748. http://pdiff.sourceforge.net/ypg01. Pdf