Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set

IEEE Robotics and Automation Magazine

Volumn 22, Issue 3, 2015, Pages 36-52

  Calli, Berk ^a   Walsman, Aaron ^b   Singh, Arjun ^c   Srinivasa, Siddhartha ^b   Abbeel, Pieter ^c   Dollar, Aaron M ^a  

^a YALE UNIVERSITY   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ROBOTICS;

CARNEGIE MELLON UNIVERSITY; DIFFERENT SHAPES; GEOMETRIC MODELS; LITERATURE SURVEY; MECHANICAL DESIGN; RESEARCH GROUPS; ROBOTIC MANIPULATION; SOFTWARE PLATFORMS;

BENCHMARKING;

EID: 84941919395     PISSN: 10709932     EISSN: None     Source Type: Journal    
DOI: 10.1109/MRA.2015.2448951     Document Type: Article

References (66)

1. A. Singh, J. Sha, K. S. Narayan, T. Achim, and P. Abbeel, "BigBIRD: A large-scale 3D database of object instances," in Proc. Int. Conf. Robotics Automation, 2014, pp. 509-516.
2. B. Li, Y. Lu, C. Li, A. Godil, T. Schreck, M. Aono, Q. Chen, N. K. Chowdhury, B. Fang, T. Furuya, H. Johan, R. Kosaka, H. Koyanagi, R. Ohbuchi, and A. Tatsuma. (2014). SHREC'14 track: Large scale comprehensive 3D shape retrieval. [Online]. Available: http://www.itl.nist.gov/iad/vug/sharp/contest/2014/Generic3D/
3. C. Goldfeder, M. Ciocarlie, H. Dang, and P. K. Allen, "The Columbia grasp database," in Proc. IEEE Int. Conf. Robotics Automation, 2009, pp. 1710-1716.
4. (2014). Sketchup. Available: [Online]. Available: http://www.sketchup.com/
5. S. Chitta, I. Sucan, and S. Cousins, "MoveIt!," IEEE Robot. Automat. Mag., vol. 19, pp. 18-19, Mar. 2012.
6. Y. S. Choi, T. Deyle, T. Chen, J. D. Glass, and C. C. Kemp, "A list of household objects for robotic retrieval prioritized by people with ALS," in Proc. IEEE Int. Conf. Rehabilitation Robotics, 2009, pp. 510-517.
7. Amazon Picking Challenge. [Online]. Available: http://amazonpickingchallenge. org/
8. J. Adams, K. Hodges, J. Kujawa, and C. Metcalf, "Test-retest reliability of the Southampton hand assessment procedure," Int. J. Rehab. Res., vol. 32, p. S18, Aug. 2009.
9. S. Kalsi-Ryan, A. Curt, M. Verrier, and M. G. Fehlings, "Development of the graded redefined assessment of strength, sensibility and prehension (GRASSP): Reviewing measurement specific to the upper limb in tetraplegia," J. Neurosurg Spine, vol. 1, pp. 65-76, Sept. 2012.
10. N. Yozbatiran, L. Der-Yeghiaian, and S. C. Cramer, "A standardized approach to performing the action research arm test," Neurorehabil Neural Repair, vol. 22, no. 1, pp. 78-90, 2008.
11. E. D. Sears and K. C. Chung, "Validity and responsiveness of the Jebsen-Taylor hand function test," J. Hand Surg. Amer., vol. 35, pp. 30-37, Jan. 2010.
12. A. P. del Pobil, "Benchmarks in robotics research," in Proc. IEEE Int. Conf. Robotics System, 2006.
13. R. Madhavan, R. Lakaemper, and T. Kalmar-Nagy, "Benchmarking and standardization of intelligent robotic systems," in Proc. Int. Conf. Advanced Robotics, 2009, pp. 1-7.
14. I. Iossifidis, G. Lawitzky, S. Knoop, and R. Zöllner, "Towards benchmarking of domestic robotic assistants," in Advances in Human-Robot Interaction, vol. 14. Berlin, Heidelberg, Germany: Springer, 2005, pp. 403-414.
15. R. Detry, O. Kroemer, and D. Kragic, "International workshop on autonomous grasping and manipulation: An open challenge," in Proc. ICRA, 2014.
16. (2006). AIM@SHAPE Shape Repository v4.0. [Online]. Available: http://shapes.aim-atshape.net/viewmodels.php
17. A. Richtsfeld. (2012). The object segmentation database (OSD). [Online]. Available: http://www.acin.tuwien.ac.at/?id=289
18. P. Shilane, P. Min, M. Kazhdan, and T. Funkhouser, "The Princeton shape benchmark," in Proc. Shape Modeling Applications, 2004, pp. 167-178.
19. A. Kasper, Z. Xue, and R. Dillmann, "The KIT object models database: An object model database for object recognition, localization and manipulation in service robotics," Int. J. Robot. Res., vol. 31, no. 8, pp. 927-934, 2012.
20. S. Ulbrich, D. Kappler, T. Asfour, N. Vahrenkamp, A. Bierbaum, M. Przybylski, and R. Dillmann, "The OpenGRASP benchmarking suite: An environment for the comparative analysis of grasping and dexterous manipulation," in Proc. IEEE/RSJ Int. Conf. Intelligent Robots Systems, 2011, pp. 1761-1767.
21. B. Li, A. Godil, M. Aono, X. Bai, T. Furuya, L. Li, R. López-Sastre, H. Johan, R. Ohbuchi, C. Redondo-Cabrera, A. Tatsuma, T. Yanagimachi, and S. Zhang, (2012). SHREC'12 track: Generic 3D shape retrieval. [Online]. Available: http://www.itl.nist.gov/iad/vug/sharp/contest/2012/Generic3D/
22. G. Kootstra, M. Popovic, J. A. Jorgensen, D. Kragic, H. G. Petersen, and N. Kruger, "VisGraB: A benchmark for vision-based grasping," Paladyn, vol. 3, no. 2, pp. 54-62, 2012.
23. A. Tatsuma, H. Koyanagi, and M. Aono, "A large-scale Shape Benchmark for 3D object retrieval: Toyohashi shape benchmark," in Proc Asia-Pacific Signal Information Processing Association Annu. Summit Conf., 2012, pp. 1-10.
24. R. B. Rusu. (2011). NIST and willow garage: Solutions in perception challenge. [Online]. Available: http://www.nist.gov/el/isd/challenge-022511.cfm and http://www.willowgarage.com/blog/2011/02/28/nistand-willow-garagesolutions-perception-challenge
25. H. Dutagaci, A. Godil, P. Daras, A. Axenopoulos, G. Litos, S. Manolopoulou, K. Goto, T. Yanagimachi, Y. Kurita, S. Kawamura, T. Furuya, and R. Ohbuchi. (2011). SHREC'11 Track: Generic Shape Retrieval. [Online]. Available: http://www.itl.nist.gov/iad/vug/sharp/contest/2011/NonRigid/
26. A. Janoch, S. Karayev, Y. Jia, J. T. Barron, M. Fritz, K. Saenko, and T. Darrell, "A category-level 3-D object dataset: Putting the Kinect to work," in Proc. IEEE Int. Conf. Computer Vision Workshops, 2011, pp. 1168-1174.
27. K. Lai, L. Bo, X. Ren, and D. Fox, "A large-scale hierarchical multi-view RGB-D object dataset," in Proc. IEEE Int. Conf. Robotics Automation, 2011, pp. 1817-1824.
28. Z. Lian, A. Godil, T. Fabry, T. Furuya, J. Hermans, R. Ohbuchi, C. Shu, D. Smeets, P. Suetens, D. Vandermeulen, and S. Wuhrer. (2010). SHREC'10 track: Non-rigid 3D shape retrieval. [Online]. Available: http://tosca.cs.technion.ac. il/book/shrec-robustness2010.html
29. R. Wessel, M. B. Ina, and R. Klein, "A 3D shape benchmark for retrieval and automatic classification of architectural data," in Proc. Eurographics Workshop 3D Object Retrieval, 2009, pp. 53-56.
30. N. Iyer, S. Jayanti, and K. Ramani, "An engineering shape benchmark for 3D models," in Proc. IDETC/CIE ASME Int. Design Engineering Tech. Conf. Computers Information Engineering Conf., 2005, pp. 501-509.
31. R. Fang, A. Godil, X. Li, and A. Wagan, "A new shape benchmark for 3D object retrieval," in Proc. 4th Int. Symp. Advances Visual Computing, Berlin, Heidelberg, Germany, 2008, pp. 381-392.
32. (2005). McGill 3D Shape Benchmark. [Online]. Available: http://www. cim.mcgill.ca/~shape/benchMark/
33. N. Kapadia, V. Zivanovic, M. Verrier, and M. R. Popovic, "Toronto rehabilitation institute-hand function test: Assessment of gross motor function in individuals with spinal cord injury," Top. Spinal Cord Inj. Rehabil., vol. 18, no. 2, pp. 167-186, 2012.
34. R. W. Sumner and J. Popovic, "Deformation transfer for triangle meshes," ACM Trans. Graphics, vol. 3, no. 3, pp. 399-405, Aug. 2014.
35. (2003). NTU 3D Model Benchmark. [Online]. Available: http://3d.csie. ntu.edu.tw/
36. (2014). The ITI database. [Online]. Available: http://vcl.iti.gr/3dobjectretrieval/
37. (2014). Model Bank Library. [Online]. Available: http://digimation. com/3d-libraries/model-bank-library/
38. T. Wisspeintner, T. van der Zan, L. Iocchi, and S. Schiffer, "RoboCup@ Home: Results in benchmarking domestic service robots," in RoboCup 2009: Robot Soccer World Cup XIII, vol. 5949. Berlin, Heidelberg, Germany: Springer, 2010, pp. 390-401.
39. G. Kragten, A. C. Kool, and J. Herder, "Ability to hold grasped objects by underactuated hands: Performance prediction and experiments," in Proc. IEEE Int. Conf. Robotics Automation, 2009, pp. 2493-2498.
40. G. A. Kragten, C. Meijneke, and J. L. Herder, "A proposal for benchmark tests for underactuated or compliant hands," Mech. Sci., vol. 1, no. 1, pp. 13-18, 2010.
41. V. Mathiowetz, G. Volland, N. Kashman, and K. Wever, "Adult norms for the box and blocks test of manual dexterity," Amer. J. Occup. Ther., vol. 39, no. 6, pp. 386-391, 1985.
42. V. Mathiowetz, K. Weber, N. Kashman, and G. Volland, "Adult norms for the nine hole peg test of finger dexterity," Occup. Ther. J. Res., vol. 5, no. 1, pp. 24-38, 1985.
43. Salia Rehab. (2015, Aug. 7). Action Research Arm Test (ARAT). [Online]. Available: http://saliarehab.com/
44. (2015, Aug. 7). GRASSP. [Online]. Available: http://grassptest.com
45. (2015, Aug. 7). Grasp and Release Test. [Online]. Available: http://www.rehabmeasures. org/Lists/RehabMeasures/DispForm.aspx?ID=1053
46. L. Resnik, L. Adams, M. Borgia, J. Delikat, R. Disla, C. Ebner, and L. S. Walters, "Development and evaluation of the activities measure for upper limb amputees," Arch. Phy. Med. Rehab., vol. 94, no. 3, pp. 488-494, 2013.
47. M. Popovic and C. Contway, "Rehabilitation engineering laboratory hand function test for functional electrical stimulation assisted grasping," in Proc. 8th Annu. Conf. Int. Functional Electrical Stimulation Society, 2003, pp. 231-234.
48. H. Burger, F. Franchignoni, A. W. Heinemann, S. Kotnik, and A. Giordano, "Validation of the orthotics and prosthetics user survey upper extremity functional status module in people with unilateral upper limb amputation," J. Rehab. Med., vol. 40, no. 5, pp. 393-399, 2008.
49. M. R. Popovic, T. Keller, I. P. I. Papas, V. Dietz, and M. Morari, "Surfacestimulation technology for grasping and walking neuroprostheses," IEEE Eng. Med. Biol. Mag., vol. 20, no. 1, pp. 82-93, 2001.
50. J. Harris, J. Eng, W. Miller, and A. S. Dawson, "A self-administered graded repetitive arm supplementary program (GRASP) improves arm function during inpatient stroke rehabilitation: A multi-site randomized controlled trial," Stroke, vol. 40, no. 6, pp. 2123-2131, 2009.
51. K. A. Stubblefield, L. A. Milller, R. D. Lipschutz, M. E. Phillips, C. W. Heckathorne, and T. A. Kuiken, "Occupational therapy outcomes with targeted hyper-reinnervation nerve transfer surgery: Two case studies," in Proc. MyoElectric Controls/Powered Prosthetics Symp., 2005, pp. 169-177.
52. K. Matheus and A. M. Dollar, "Benchmarking grasping and manipulation: Properties of the objects of daily living," in Proc. IEEE/RSJ Int. Conf.Intelligent Robots Systems, 2010, pp. 5020-5027.
53. (2015, Aug. 7). The Dash-"Disabilities of the Arm, Shoulder and Hand". [Online]. Available: http://dash.iwh.on.ca/system/files/dash-questionnaire-2010.pdf
54. M. Kazhdan, M. Bolitho, and H. Hoppe, "Poisson Surface Reconstruction," presented at the Eurographics Symp. Geometry Processing, 2006.
55. (2015, Aug. 7). [Online]. Available: http://rll.eecs.berkeley.edu/ycb/
56. M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, E. Berger, R. Wheeler, and A. Y. Ng, "ROS: An open-source robot operating system," in Proc. ICRA Workshop Open Source Software, 2009, vol. 3, p. 5.
57. R. Diankov and J. Kuffner, "OpenRAVE: A planning architecture for autonomous robotics," Robotics Institute, Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-RI-TR-08-34, 2008.
58. N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa, "CHOMP: Gradient optimization techniques for efficient motion planning," in Proc. IEEE Int. Conf. Robotics Automation, 2009, vols. 1-7, pp. 4030-4035.
59. D. Berenson, S. S. Srinivasa, D. Ferguson, and J. J. Kuffner, "Manipulation planning on constraint manifolds," in Proc. IEEE Int. Conf. Robotics Automation, 2009, pp. 625-632.
60. M. Garland and P. S. Heckbert, "Surface simplification using quadric error metrics," presented at the Proc. 24th Annu. Conf. Computer Graphics and Interactive Techniques, 1997, pp. 209-216.
61. S. S. Srinivasa, D. Berenson, M. Cakmak, A. Collet, M. R. Dogar, A. D. Dragan, R. A. Knepper, T. D. Niemueller, K. Strabala, and J. M. Vandeweghe, "Herb 2.0: Lessons learned from developing a mobile manipulator for the home," Proc. IEEE, vol. 100, no. 8, pp. 2410-2428, 2012.
62. I. A. Sucan, M. Moll, and L. E. Kavraki, "The open motion planning library," IEEE Robot. Automat. Mag., vol. 19, pp. 72-82, Dec. 2012.
63. (2015, Aug. 7). YCB Protocols and Benchmarks. [Online]. Available: http://www.ycbbenchmarks.org
64. (2015, Aug. 7). YCB Protocols. [Online]. Available: http://protocols.ycbbenchmarks. org
65. R. R. MA, L. U. Odhner, and A. M. Dollar, "A modular, open-source 3D printed underactuated hands," in Proc. IEEE Int. Conf. Robotics Automation, Karlsruhe, Germany, 2013, pp. 2737-2743.
66. W. Garage. (2009). Overview of the PR2 robot. [Online]. Available: http://www.willowgarage.com/pages/pr2/overview