Benchmarking and standardization of intelligent robotic systems

2009 International Conference on Advanced Robotics, ICAR 2009

Volumn , Issue , 2009, Pages

  Madhavan, Raj ^a,b   Lakaemper, Rolf ^c   Kalmár Nagy, Tamás ^d  

^a OAK RIDGE NATIONAL LABORATORY   (United States)

^b NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

^c Temple University   (United States)

^d Department of Electrical and Computer Engineering   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPLICATION AREA; DISASTER SCENARIO; FIRST RESPONDERS; INTELLIGENT ROBOTIC SYSTEMS; MOBILE ROBOTIC SYSTEMS; PERFORMANCE METRICS; REPETITIVE TASK; ROBOTIC MAPPING; ROBOTIC TECHNOLOGIES; SEARCH AND RESCUE; STANDARDIZED METHODS;

ROBOTICS; STANDARDIZATION;

ROBOTS;

EID: 70449344636     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (50)

1. AS-4 Unmanned Systems Committee. http://www.sae.org/servlets/works/ committeeHome.do?comtID=TEAAS4.
2. Autonomy Levels for Unmanned Systems. http://www.isd.mel.nist.gov/ projects/autonomy-levels/.
3. CCC/CRA Roadmapping for Robotics. http://www.us-robotics.us/.
4. Committee E54 on Homeland Security Applications. http://www.astm.org/ COMMIT/COMMITTEE/E54.htm/.
5. Defining a Standard Benchmark for Mobile Service Robots. http://wiki.robot-standards.org/index.php/Benchmarks.
6. EURON Benchmarking Inititative. http://www.euron.org/activities/ benchmarks/;http://www.robot.uji.es/EURON/en/index.htm/.
7. IEEE 802 LAN/MAN Standards Committee. http://www.ieee802.org/.
8. IEEE RAS Standing Committee on Standards Activities. http://www.ieee-ras. org/industrial/standards/.
9. Jacobs Map Analysis Toolkit. http://robotics.iu-bremen.de/datasets/ MapEvaluation/.
10. Joint Architecture for Unmanned Systems. http://www.jauswg.org/.
11. OpenSLAM. http://www.openslam.org/.
12. Radish: The Robotics Data Set Repository. http://radish.sourceforge.net/.
13. RAWSEEDS: Robotics Advancement through Web-publishing of Sensorial and Elaborated Extensive Data Sets. http://www.rawseeds.org/.
14. Reference Datasets. http://www.isd.mel.nist.gov/projects/USAR/2007/ refdatasets.htm/.
15. Response Robot Evaluation Exercise (#5). http://www.isd.mel.nist.gov/ US&R-Robot-Standards/disaster-city/eventint%ro5.htm/.
16. Robocuprescue. http://www.robocuprescue.org.
17. Robot Standards and Reference Architectures. http://www.robot-standards. eu/.
18. Robotics Domain Task Force. http://robotics.omg.org/.
19. Robotics Industries Association. http://www.robotics.org/.
20. The European Robotics Platoform. http://cordis.europa.eu/ist/europ/.
21. The European Robotics Platoform: Strategic Research Agenda. ftp://ftp.cordis.europa.eu/pub/ist/docs/europ/rob-plat-2.pdf/.
22. The FP7 ICT Work Programme. http://cordis.europa.eu/fp7/ict/programme/.
23. The Mobile Robot Programming Toolkit. http://babel.isa.uma.es/mrpt/.
24. Urban Search and RescueRobot Competitions. http://www.isd.mel.nist.gov/ projects/USAR/competitions.htm/.
25. Workshop on 'Robotic Standardization', IEEE/RSJ International Conference on Intelligent Robots and Systems. http://staff.aist.go.jp/t.kotoku/conf/ iros2006ws/.
26. Performance Metrics for Intelligent Systems (PerMIS) Workshop, 2000-2008. http://www.isd.mel.nist.gov/PerMIS/.
27. Robotics in Manufacturing Technology Roadmap. Technical report, Energetics Incorporated, November 2006.
28. Rising Above The Gathering Storm: Energizing and Employing America for a Brighter Economic Future. National Academies Press, 2007.
29. S. Abdallah, D. Asmar, and J. Zelek. Towards Benchmarks for Vision SLAM Algorithms. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 3834-3839, May 2006.
30. R. Madhavan A.P. del Pobil and E. Messina (Organizers). In Performance Evaluation and Benchmarking for Intelligent Robots and Systems, November 2007.
31. R. Madhavan A.P. del Pobil and F. Bonsignorio (Organizers). In Performance Evaluation and Benchmarking for Intelligent Robots and Systems, September 2008.
32. J. Baltes. A Benchmark Suite for Mobile Robots. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1101-1106, 2000.
33. G. Bekey, R. Ambrose, V. Kumar, D. Lavery, A. Sanderson, B. Wilcox, J. Yuh, and Y. Zheng. Robotics: State of the Art and Future Challenges. Imperial College Press, 2008.
34. G. Bekey, R. Ambrose, V. Kumar, A. Sanderson, and B. Wilcox. WTEC Panel Report on International Assessment of Research and Development in Robotics. Technical report, National Technical Information Service (NTIS), January 2006. Report No. PB2007-102141.
35. F. Bonsignorio, J. Hallam, and A.P. del Pobil (Organizers). Experimental Methodology and Benchmarking in Robotics Research. In Workshop Proceedings, Robotics: Science and Systems (RSS) Conference, Zurich, Switzerland, June 2008. http://www.heronrobots.com/EuronGEMSig/GEMSIGRSS08Program.html/.
36. A. Bose. The OMG Robotics DTF: Motives, Challenges & Efforts. In Workshop on 'Measures and Procedures for the Evaluation of Robot Architectures and Middleware', IEEE/RSJ International Conference on Intelligent Robots and Systems. http://wiki.robot-standards.org/images/1/1a/Omg-robotics.pdf/.
37. H. Christensen. EURON - The European Robotics Network. IEEE Robotics & Automation Magazine, 12(2):10-13, June 2005.
38. R. Dillmann. KA 1.10 Benchmarks for Robotics Research, April 2004. http://www.cas.kth.se/euron/eurondeliverables/ka1-10-benchmarking.pdf/.
39. D. Feil-Seifer, K. Skinner, and M. Matarić. Benchmarks for Evaluating Socially Assistive Robotics. Interaction Studies: Psychological Benchmarks of Human-Robot Inteaction, 8(3), October 2007.
40. W. Gates. A Robot in Every Home. In Scientific American, December 2006.
41. A. Jacoff, B. Weiss, and E. Messina. Evolution of Metrics and Performance for USAR Competitions. In Proceedings of the Performance Metrics for Intelligent Systems (PerMIS) Workshop, NIST Special Publication 1017, September 2003.
42. R. Lakaemper and N. Adluru. Improving Sparse Laser Scan Alignment with Virtual Scans. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2915-2921, 2008.
43. R. Madhavan, C. Scrapper, and A. Kleiner (eds.). Characterizing Mobile Robot Localization and Mapping. Autonomous Robots, 2009 (to appear).
44. R. Madhavan, E. Tunstel, and E. Messina, editors. Performance Evaluation and Benchmarking of Intelligent Systems. Springer, September 2009 (to appear).
45. E. Messina. Performance Standards for Urban Search and Rescue Robots. ASTM Standardization News, August 2006.
46. E. Messina, A. Jacoff, J. Scholtz, C. Schlenoff, H.-M. Huang, A. Lytle, and J. Blitch. Statement Of Requirements For Urban Search And Rescue Robot Performance Standards, 2005. http://www.isd.mel.nist.gov/US&R-Robot- Standards/Requirements%20Report%%20(prelim).pdf.
47. K. Nilsson. Personal Communication.
48. The President's Council of Advisors on Science and Technology (PCAST). Leadership Under Challenge: Information Technology R&D in a Competitive World. Technical report, August 2007. http://www.ostp.gov/pdf/nitrd-review.pdf.
49. E. Prassler and K. Nilsson. 1,001 Robot Architectures for 1,001 Robots. IEEE Robotics & Automation Magazine, 16(1):113, March 2009.
50. I. Varsadan, A. Birk, and M. Pfingsthorn. Determining Map Quality through an Image Similarity Metric. In Lecture Notes in Artificial Intelligence (LNAI): RoboCup 2008: Robot WorldCup XII, 2008.