Real-time photorealistic virtualized reality interface for remote mobile robot control

International Journal of Robotics Research

Volumn 30, Issue 3, 2011, Pages 384-404

  Kelly, Alonzo ^a   Chan, Nicholas ^a   Herman, Herman ^a   Huber, Daniel ^a   Meyers, Robert ^a   Rander, Pete ^a   Warner, Randy ^a   Ziglar, Jason ^a   Capstick, Erin ^b  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b DCS CORPORATION   (United States)

Author keywords

compression; human interface; latency compensation; mobile robot; teleoperation; Virtual reality

Indexed keywords

COMPRESSION; DISPLAY QUALITY; ENVIRONMENT MODELS; HIGH QUALITY; HIGH-SPEED; HUMAN INTERFACE; LATENCY COMPENSATION; LIDAR DATA; LINE-OF-SIGHT; LIVE VIDEO; MOBILE ROBOT CONTROL; PHOTO-REALISTIC; PHOTOREALISTIC RENDERING; REALITY INTERFACE; SPEED REQUIREMENT; TELE-OPERATIONS; USER EXPERIENCE; VIDEO COMPRESSION; VIDEO GAME; VIDEO QUALITY; WIRELESS LINK; WIRELESS VIDEO;

IMAGE COMPRESSION; MOBILE ROBOTS; OPTICAL RADAR; REMOTE CONTROL;

VIRTUAL REALITY;

EID: 79952150239     PISSN: 02783649     EISSN: 17413176     Source Type: Journal    
DOI: 10.1177/0278364910383724     Document Type: Conference Paper

References (60)

1. Anderson, D., Howard, T., Apfelbaum, D., Herman, H. and Kelly, A. (2008). Coordinated control and range imaging for mobile manipulation. Proceedings of the International Symposium on Experimental Robotics (ISER).
2. Anderson, D. and Kelly, A. (2005). Experimental characterization of commercial flash lidar devices. Proceedingsof the International Conference on Sensing Technologies, Palmerston North, New Zealand.
3. Bejczy, A., Kim, W. and Venema, S. (1990). The phantom robot: predictive displays for teleoperation with time delay. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
4. Buss, M., Peer, A., Schaub, T., Stefanov, N. and Unterhinninghofen, U. (2008). Multi-modal multi-user telepresence and teleaction system. Intelligent Robots and Systems, 4137-4138.
5. Chatila, R., Lacroix, S., Simion, T. and Herrb, M. (1995). Planetary exploration by a mobile robot: mission teleprogramming and autonomous navigation. Autonomous Robots 2 (4). 333-344.
6. Chen, S.E. and Williams, L. (1993). View interpolation for image synthesis. Proceedings of SIGGRAPH 93, pp. 279-288.
7. Daily, M. et al. (1988). Autonomous cross country navigation with the ALV. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 718-726.
8. Debevec, P., Taylor, C. and Malik, J. (1996). Modeling and rendering architecture from photographs. Proceedings of SIGGRAPH, pp. 11-20.
9. Dima, C., Vandapel, N. and Hebert, M. (2004). Classifier fusion for outdoor obstacle detection. Proceedings of the International Conference on Robotics and Automation (ICRA), pp. 665-671.
10. Edwards, L., Sims, M., Kunz, C., Lees, D. and Bowman, J. (2005). Photo-realistic terrain modeling and visualization for mars exploration rover science operations. IEEE Systems, Man, and Cybernetics, Waikoloa, Hawaii.
11. El-Hakim, S.F., Boulanger, P., Blais, F. and Beraldin, J.A. (1997). A system for indoor 3D mapping and virtual environments. Videometrics V, SPIE Volume 3174. pp. 21-35.
12. Ferland, F., Pomerleau, F., Le Dinh, C. and Michaud, F. (2009). Egocentric and exocentric teleoperation interface using real-time, 3D video projection. Proceedings of the 4th ACM/IEEE Inter-national Conference on Human Robot Interaction, pp. 37-44.
13. Fong, T., Pangels, H. and Wettergreen, D. (1995). Operator interfaces and network-based participation for Dante II. Proceedings of the SAE 25th International Conference on Environmental Systems, 11(1): 77-85
14. Fong, T., Thorpe, C. and Baur, C. (2001). Advanced interfaces for vehicle teleoperation: collaborative control, sensor fusion displays, and remote driving tools. Autonomous Robots, 77-85. (Pubitemid 32650940)
15. Früh, C. and Zakhor, A. (2004). An automated method for large-scale, ground-based city model acquisition. International Journal of Computer Vision, 60 (1). 5-24.
16. Funda, J., Lindsay, T. and Paul, R. (1992). Teleprogramming: toward delay-invariant remote manipulation. Presence: Teleoperation. Virtual Environonments, 1 (1). 29-44.
17. Hart, S.G. and Staveland, L.E. (1987). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research (Vol. Human mental workload). Meshkati, P. H. (eds). Amsterdam: Elsevier.
18. Hine, B., Hontalas, P., Fong, T., Piguet, L., Nygren, E. and Kline, A. (1995). VEVI: A Virtual Environment Teleoperations Interface for planetary exploration. Proceedings of the SAE 25th International Conference on Environmental Systems, San Diego, CA.
19. Hine, B., Stocker, C. and Sims, M. (1994). The application of telepresence and virtual reality to subsea exploration. Proceedings of the 2nd Workshop on Mobile Robots for Subsea Environments (Proc. ROV'94), Monterey, CA.
20. Ho, N. and Jarvis, R. (2007). Large scale 3D environmental modelling for stereoscopic walk-through visualisation. Proceedings of 3DTV07, pp. 1-4.
21. Hoiem, A., Efros, A. and Hebert, M. (2005). Geometric context from a single image. Proceedings of the International Conference on Computer Vision (ICCV), pp. 654-661.
22. Howard, T., Green C., and Kelly A. Receding Horizon Model-Predictive Control for Mobile Robot Navigation of Intricate Paths, FSR 09.
23. Hu, S.Y. and Neumann, U. (2003). Approaches to large-scale urban modeling. IEEE Computer Graphics and Applications (CG&A), 23 (6). 62-69.
24. Huber, D., Herman, H., Kelly, A., Rander, P. and Warner, R. (2009). Real-time photorealistic visualization of 3D environments for enhanced teleoperation of vehicles. Proceedings of the 2nd International Conference on 3D Digital Imaging and Modeling, Kyoto, Japan.
25. Jakel, L., Krotkov, E., Perschbacher, M., Pippine, J. and Sullivan, C. (2006). The DARPA LAGR program: goals, challenges, methodology, and phase I results. Journal of Field Robotics, 23, 945-973.
26. Johnson, J., Alberts, J., Berkemeier, M., and Edwards, J. (2008). Manipulator autonomy for EOD robots. Proceedings of the 26th Army Science Conference.
27. Kanade, T., Rander, P. and Narayanan, P.J. (1997). Virtualized reality: constructing virtual worlds from real scenes. IEEE MultiMedia, 4 (1). 34-47. (Pubitemid 127583991)
28. Kelly, A. and Stentz, A. (1998). Rough terrain autonomous mobility, part 2: an active vision, predictive control approach. Autonomous Robots, 5, 163-198. (Pubitemid 128514283)
29. Kelly, A., Stentz, A., Amidi, O., Bode, M., Bradley, D., Diaz-Calderon, A. (2006). Toward reliable off-road autonomous vehicles operating in challenging environments. The International Journal of Robotics Research, 25 (5-6). 449-483. (Pubitemid 43735584)
30. Kim, W.S. (1993). Advanced teleoperation, graphics aids, and application to time delay environments. Proceedings of the 1st Industrial Virtual Reality Show and Conference (IVR '93), Makuhari Meese, Japan, pp. 23-25.
31. Kim, W.S. (1996). Virtual reality calibration and preview / predictive displays for telerobotics. Presence: Teleoperators and Virtual Environments, 5 (2). 173-190.
32. Lacase, A., Murphy, K. and DelGiorno, M. (2002). Autonomous mobility for the Demo III experimental unmanned vehicles. Proceedings of the Assoc. for Unmanned Vehicle Systems Int. Conf. on Unmanned Vehicles (AUVSI 02).
33. Lewis, R.A. and Johnston, A.R. (1977). A scanning laser rangefinder for a robotic vehicle. Proceedsings of the 5th International Joint Conference on Artificial Intelligence.
34. Lloyd, J.E., Beis, J.S., Pai, D.K. and Lowe, D.G. (1997). Modelbased telerobotics with vision. Proceedings of the International Conference on Robotics and Automation (ICRA), pp. 1297-1304.
35. Matusik, W., Beuhler, C., Raskar, R., Gortler, L. and McMillan, L. (2000). Image-based visual hulls. Proceedings of SIGGRAPH, pp. 369-374. (Pubitemid 32394001)
36. McMillan, L. and Bishop, G. (1995). Plenoptic modeling: an image-based rendering system. Proceedings of ACM SIGGRAPH, pp. 39-46.
37. Messuri, D.A. and Klein, C.A. (1985). Automatic body regulation for maintaining stability of a legged vehicle during terrain locomotion. IEEE Journal of Robotics and Automation, 1 (3). 132-141.
38. Milgram, P., Yin, S. and Grodski, J. (1997). An augmented reality based teleoperation interface for unstructured environments. Proceedings of the American Nuclear Society (ANS) 7th Topical Meeting on Robotics and Remote Systems, pp. 966-973.
39. Milgram, P., Zhai, S., Drascic, D. and Grodski, J. (1993). Applications of augmented reality for human-robot communication. Proceedings of the International Conference on Intelligent Robots and Systems (IROS), Yokohama, Japan, pp. 1467-1472.
40. Möller, T., Kraft, H., Frey, J., Albrech, M. and Lange, R. (2005). Robust 3D measurement with pmd sensors. Proceedings of the First Range Imaging Research Dat at ETH Zurich.
41. Mordohai, P., Frahm, J.M. and Akbarzadeh, A. (2007). Real-time video-based reconstruction of urban environments. Proceedings of the ISPRS Workshop on 3D Virtual Reconstruction and Visualization of Complex Architectures.
42. Nielsen, C.W., Goodrich, M.A. and Ricks, B. (2007). Ecological interfaces for improving mobile robot teleoperation. IEEE Transactions on Robotics and Automation, 23 (5). 927-941. (Pubitemid 47577969)
43. Rander, P. (1998). A multi-camera method for 3D digitization of dynamic, real-world events. Ph.D. Thesis, Pittsburgh, PA, Robotics Institute, Carnegie Mellon University.
44. Ricks, B., Nielsen, C. and Goodrich, M. (2004). Ecological displays for robot interaction: a new perspective. IEEE/RSJ Proceedings of the International Conference on Intelligent Robots and Systems (IROS), pp. 2855-2860.
45. Rohlf, J. and Helman, J. (1994). IRIS Performer: a high performance multiprocessing toolkit for real-time 3D graphics. Proceedings of SIGGRAPH, pp. 381-394.
46. Ross, B., Bares, J., Stager, D., Jacker, L. and Perschbacher, M. (2008). An advanced teleoperation testbed. Field and Service Robotics. 42: 397-304.
47. Ryde, J. and Hu, H. (2008). 3D laser range scanner with hemispherical field of view for robot navigation. Proceedings of the 2008 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Xi'an, China.
48. Schneider, M. and Klein, R. (2008). Enhancing textured digital elevation models using photographs. Proceedings of the Intl. Symp. on 3D Data Processing, Visualization, and Transmission (3DPVT), pp. 261-268.
49. Se, S. and Jasiobedzki, P. (2008). Stereo-vision based 3D modeling and localization for unmanned vehicles. International Journal of Intelligent Control and Systems, 13 (1). 47-58.
50. Segal, M., Korobkin, C., van Widenfelt, R., Foran, J. and Haeberli, P. (1992). Fast shadows and lighting effects using texture mapping. Proceedings of SIGGRAPH, pp. 249-252. (Pubitemid 23614119)
51. Sheridan, T. (1986). Human supervisory control of robot systems. Proceedings of the IEEE International Conference on Robotics and Automation.
52. Sheridan, T. (1993). Space teleoperation through time delay: review and prognosis. IEEE Trans. Robotics Automation, 9, 592-606.
53. Smallman, H. and St. John, M. (2005). Naive realism: misplaced faith in realistic displays. Ergonomics in Design, 13: 6-13. (Pubitemid 44756577)
54. Stamos, I. and Allen, P.K. (2002). Geometry and texture recovery of scenes of large scale. Computer Vision and Image Understanding (CVIU), 88 (2). 94-118.
55. Stewart, B., Ko, J. and Konolige, K. (2003). The revisiting problem in mobile robot map building: a hierarchical bayesian approach. Proceedings of the Conference on Uncertainty in Artificial Intelligence.
56. Theunissen, E., Koeners, G., Roefs, F., Rademaker, R., Jinkins, R. and Etherington, T. (2005). Guidance, situation awareness and integrity monitoring with an SVS+EVS. Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit, San Francisco, CA.
57. Wellington, C., Courville, A. and Stentz, A. (2006). A generative model of terrain for autonomous navigation in vegetation. Proceedings of the Intl. Journal of Robotics Research (IJRR), 25 (1). 1287-1304. (Pubitemid 44745897)
58. Wellington, C. and Stentz, A. (2004). Online adaptive rough-terrain navigation in vegetation. Proceedings of the IEEE Int. Conf. on Robotics and Automation (ICRA).
59. Westover, L. (1990). Footprint evaluation for volume rendering. in computer graphics. Proceedings of SIGGRAPH, pp. 367-376.
60. Williams, L. (1978). Casting curved shadows on curved surfaces. Proceedings of ACM SIGGRAPH, pp. 270-274.