Methodology for evaluating static six-degree-of-freedom (6DoF) perception systems

Performance Metrics for Intelligent Systems (PerMIS) Workshop

Volumn , Issue , 2010, Pages 290-297

  Chang, Tommy ^a   Hong, Tsai ^a   Falco, Joe ^a   Shneier, Michael ^a   Shah, Mili ^b   Eastman, Roger ^b  

^a NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

^b LOYOLA UNIVERSITY MARYLAND   (United States)

Author keywords

6DoF metrology; ground truth; laser tracker; performance evaluation

Indexed keywords

ACCURACY AND PRECISION; GROUND TRUTH; LASER TRACKER; PERCEPTION SYSTEMS; PERFORMANCE EVALUATION; POSE DETERMINATION; SIX-DEGREE-OF-FREEDOM; SYSTEM CALIBRATION; SYSTEM'S PERFORMANCE; VISION BASED;

END EFFECTORS; INTELLIGENT SYSTEMS;

LASERS;

EID: 84871365885     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2377576.2377629     Document Type: Conference Paper

References (23)

1. D. Anderson, H. Herman, and A. Kelly. Experimental characterization of commercial flash ladar devices. In International Conference of Sensing and Technology, November 2005.
2. S. Bouix, M. Martin-Fernandez, L. Ungar, M. N. M.-S. Koo, R. W. McCarley, and M. E. Shenton. On evaluating brain tissue classifiers without a ground truth. NeuroImage, 36:1207-1224, 2007.
3. T. Chang, T. Hong, M. Shneier, G. Holguin, J. Park, and R. D. Eastman. Dynamic 6dof metrology for evaluating a visual servoing system. In PerMIS '08: Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems, pages 173-180, New York, NY, USA, 2008. ACM.
4. C. E. Erdem, A. M. Tekalp, and B. Sankur. Metrics for performance evaluation of video object segmentation and tracking without ground-truth. International Conference On Image Processing, 2001.
5. J. Fitzpatrick and J. West. A blinded evaluation and comparison of image registration methods. In IEEE Workshop on Empirical Evaluation Methods in Computer Vision, 1998.
6. ISO 9283:1998. Manipulating industrial robots - Performance criteria and related test methods. Second edition. ISO, Geneva, Switzerland.
7. J. Kybic. Fast no ground truth image registration accuracy evaluation: Comparison of bootstrap and hessian approaches. In ISBI, pages 792-795, 2008.
8. A. M. Lytle, S. Szabo, G. Cheok, K. Saidi, and R. Norcross. Performance evaluation of a 3d imaging system for vehicle safety. In Unmanned Systems Technology IX. Proceedings of the SPIE, Volume 6561, 2007.
9. C. B. Madsen. A comparative study of the robustness of two pose estimation techniques. Machine Vision and Applications, 9(5-6):291-303, 1997.
10. R. Schestowitz, C. J. Twining, T. Cootes, V. Petrovic, B. Crum, and C. J. Taylor. Non-rigid registration assessment without ground truth. In Medical Image Understanding and Analysis, volume 2, pages 151-155, 2006.
11. S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR '06: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 519-528, Washington, DC, USA, 2006. IEEE Computer Society.
12. M. Shah, T. Chang, and T. Hong. Mathematical metrology for evaluation a 6dof visual servoing system. In PerMIS '09: Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems, 2009.
13. M. I. Shah. Six degree of freedom point correspondences. Technical Report, Loyola University Maryland, TR2009-02., 2009.
14. F. Smit and R. van Liere. A framework for performance evaluation of model-based optical trackers. EGVE Symposium, 6, 2008.
15. A. C. Tamhane and D. D. Dunlop. Statistics and Data Analysis from Elementary to Intermediate. Prentice Hall, Upper Saddle River, NJ 07458, 2000.
16. N. A. Thacker, A. F. Clark, J. L. Barron, J. R. Beveridge, P. Courtney, W. R. Crum, V. Ramesh, and C. Clark. Performance characterization in computer vision: A guide to best practices. Computer Vision and Image Understanding, 109(3):305-334, 2008.
17. M. Tonko and H.-H. Nagel. Model-based stereo-tracking of non-polyhedral objects for automatic disassembly experiments. International Journal of Computer Vision, 37(1):99-118, 2000.
18. R. van Liere and A. van Rhijn. An experimental comparison of three optical trackers for model based pose determination in virtual reality. In S. Coquillart and M. Göbel, editors, 10th Eurographics Symposium on Virtual Environments, pages 25-34, Grenoble, France, 2004. Eurographics Association.
19. F. Vikstén, P.-E. Forssén, B. Johansson, and A. Moe. Comparison of local image descriptors for full 6 degree-of-freedom pose estimation. In IEEE International Conference on Robotics and Automation, Kobe, Japan, May 2009. IEEE, IEEE Robotics and Automation Society.
20. J. B. West and J. M. Fitzpatrick. The distribution of target registration error in rigid-body, point-based registration. In IPMI '99: Proceedings of the 16th International Conference on Information Processing in Medical Imaging, pages 460-465, London, UK, 1999. Springer-Verlag.
21. J. B. West, J. M. Fitzpatrick, M. Y. Wang, B. M. Dawant, C. R. M. Jr., R. M. Kessler, and R. J. Maciunas. Retrospective intermodality registration techniques for images of the head: Surface-based versus volume-based. IEEE Trans. Med. Imaging, 18(2):144-150, 1999.
22. Q. Yang, R. M. Steele, D. Nister, and C. Jaynes. Learning the probability of correspondences without ground truth. In ICCV '05: Proceedings of the Tenth IEEE International Conference on Computer Vision, pages 1140-1147, Washington, DC, USA, 2005. IEEE Computer Society.
23. C. Ye and J. Borenstein. Characterization of a 2-d laser scanner for mobile robot obstacle negotiation. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 2512-2518, 2002.