Good Laboratory Practice for optimization research

Journal of the Operational Research Society

Volumn 67, Issue 4, 2016, Pages 676-689

  Kendall, Graham ^a,b   Bai, Ruibin ^c   Błazewicz, Jacek ^d   De Causmaecker, Patrick ^e   Gendreau, Michel ^f   John, Robert ^a   Li, Jiawei ^a   McCollum, Barry ^g   Pesch, Erwin ^h   Qu, Rong ^a   Sabar, Nasser ^b   Berghe, Greet Vanden ^e   Yee, Angelina ^b  

^a UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM MALAYSIA CAMPUS   (Malaysia)

^c UNIVERSITY OF NOTTINGHAM NINGBO CHINA   (China)

^d POZNAN UNIVERSITY OF TECHNOLOGY   (Poland)

^e UNIVERSITY OF LEUVEN   (Belgium)

^f UNIVERSITÉ DE MONTRÉAL   (Canada)

^g QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

^h UNIVERSITY OF SIEGEN   (Germany)

Author keywords

operations research; optimization; reproducibility

Indexed keywords

OPERATIONS RESEARCH; OPTIMIZATION;

CONCRETE SET; LABORATORY PRACTICES; OPTIMIZATION RESEARCHES; REPRODUCIBILITIES;

CLINICAL RESEARCH;

EID: 84962258400     PISSN: 01605682     EISSN: 14769360     Source Type: Journal    
DOI: 10.1057/jors.2015.77     Document Type: Article

References (92)

1. Adenso-Diaz B and Laguna M (2006). Fine-tuning of algorithms using fractional experimental designs and local search. Operations Research 54(1): 99-114.
2. Ahuja RV and Orlin J (1996). Use of representative operation counts in computational testing of algorithms. INFORMS Journal on Computing 8(3): 318-330.
3. Anon (2013). Announcement: Reducing our irreproducibility. Nature 496(7446): 398.
4. Archetti C and Speranza MG (2008). The split delivery vehicle routing problem: A survey. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challanges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 103-122.
5. Baker M (2012). Independent labs to verify high-profile papers: Nature News, 14 August 2012, Chicago. doi:10.1038/nature.2012.11176.
6. Balinski ML (1978). On the reporting of computational experiments. Mathematical Programming 15(1): 315.
7. Barr RS, Golden BL, Kelly JP, Resende MGC and Stewart Jr WR (1995). Designing and reporting on computational experiments with heuristic methods. Journal of Heuristics 1(1): 9-32.
8. Begley CG and Ellis LM (2012). Drug development: Raise standards for preclinical cancer research. Nature 483(7391): 531-533.
9. Bellmore M and Nemhauser GL (1968). Travelling salesman problem: A survey. Operations Research 16(3): 538-558.
10. Błazewicz J, Domschke W and Pesch E (1996). The job shop scheduling problem: Conventional and new solution techniques. European Journal of Operational Research 93(1): 1-33.
11. Błazewicz J, Ecker KH, Pesch E, Schmidt G and Weglarz J (2007). Handbook on Scheduling. Springer-Verlag: Berlin, Heidelberg.
12. Błazewicz J, Formanowicz P and Kasprzak M (2005). Selected combinatorial problems of computational biology. European Journal of Operational Research 161(3): 585-597.
13. Boylan JE, Goodwin P, Mohammadipour M and Syntetos AA (2015). Reproducibility in forecasting research. International Journal of Forecasting 31(1): 79-90.
14. Braysy O, Dullaert W and Gendreau M (2004). Evolutionary algorithms for the vehicle routing problem with time windows. Journal of Heuristics 10(6): 587-611.
15. Braysy O and Gendreau M (2005a). Vehicle routing problem with time windows, part I: Route construction and local search algorithms. Transportation Science 39(1): 104-118.
16. Braysy O and Gendreau M (2005b). Vehicle routing problem with time windows, part II: Metaheuristics. Transportation Science 39(1): 119-139.
17. Brown L (1999). Technical and Military Imperatives: A Radar History of World War 2. CRC Press: Bristol, UK.
18. Brucker P, Sotskov YN and Werner F (2007). Complexity of shopscheduling problems with fixed number of jobs: A survey. Mathematical Methods of Operations Research 65(3): 461-481.
19. Burkard RE, Deineko VG, Van Dal R, Van der Veen JAA and Woeginger GJ (1998). Well-solvable special cases of the traveling salesman problem: A survey. SIAM Review 40(3): 496-546.
20. Burke EK et al (2013). Hyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society 64(12): 1695-1724.
21. Cattrysse DG and Van Wassenhove LN (1992). A survey of algorithms for the generalized assignment problem. European Journal of Operational Research 60(3): 260-272.
22. Cheng RW, Gen M and Tsujimura Y (1999a). A tutorial survey of jobshop scheduling problems using genetic algorithms, part II: Hybrid genetic search strategies. Computers & Industrial Engineering 36(2): 343-364.
23. Cheng RW, Gen M and Tsujimura Y (1999b). A tutorial survey of jobshop scheduling problems using genetic algorithms: Part II. Hybrid genetic search strategies. Computers & Industrial Engineering 37(1-2): 51-55.
24. Cook WJ (2011). In Pursuit of the Traveling Salesman: Mathematics at the Limits of Computation. Princeton University Press: USA.
25. Cordeau JF, Gendreau M, Laporte G, Potvin JY and Semet F (2002). A guide to vehicle routing heuristics. Journal of the Operational Research Society 53(5): 512-522.
26. Coy SP, Golden BL, Runger GC and Wasil EA (2001). Using experimental design to find effective parameter settings for heuristics. Journal of Heuristics 7(1): 77-97.
27. Crowder H, Dembo RS and Mulvey JM (1979). On reporting computational experiments with mathematical software. ACM Transactions on Mathematical Software 5(2): 193-203.
28. Crowder HP, Dembo RS and Mulvey JM (1978). Reporting computational experiments in mathematical programming. Mathematical Programming 15(1): 316-329.
29. Derrac J, Garcia S, Molina D and Herrera F (2011). A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation 1(1): 3-18.
30. Doerner KF and Schmid V (2010). Survey:Matheuristics for rich vehicle routing problems. In: Blesa MJ, Blum C, Raidl G, Roli A and Sampels M (eds). Hybrid Metaheuristics, Lecture Notes in Computer Science. Vol. 6373, 7th International Workshop on Hybrid Metaheuristics, Vienna, Austria, Springer: Berlin Heidelberg, pp 206-221, Oct 01-02, 2010.
31. Dongarra JJ (1992). Performance of various computers using standard linear equations software. ACM SIGARCH Computer Architecture News 20(3): 22-44.
32. Easley RW, Madden CS and Dunn MG (2000). Conducting marketing science: The role of replication in the research process. Journal of Business Research 48(1): 83-92.
33. Easton K, Nemhauser G and Trick M (2001). The traveling tournament problem description and benchmarks, chap. Principles and Practice of Constraint Programming CP 2001: 7th International Conference, CP 2001 Paphos, Cyprus, 26 November-1 December, 2001 Proceedings. Lecture Notes in Computer Science 2239, Springer, Berlin Heidelberg, pp 580-584.
34. Ernst AT, Jiang H, KrishnamoorthyMand Sier D (2004). Staff scheduling and rostering: A review of applications, methods and models. European Journal of Operational Research 153(1): 3-27.
35. Evanschitzky H and Armstrong JS (2010). Replications of forecasting research. International Journal of Forecasting 26(1): 4-8.
36. Evanschitzky H, Baumgarth C, Hubbard R and Armstrong JS (2007). Replication research's disturbing trend. Journal of Business Research 60(4): 411-415.
37. Fildes R (1979). Quantitative forecasting the state of the art: Extrapolative models. Journal of the Operational Research Society 30(8): 691-710.
38. Fildes R (1985). Quantitative forecasting the state of the art: Econometric models. Journal of the Operational Research Society 30(7): 549-580.
39. Fildes R, Nikolopoulos K, Crone SF and Syntetos AA (2008). Forecasting and operational research: A review. Journal of the Operational Research Society 59(9): 1150-1172.
40. Garcia S, Fernández A, Luengo J and Herrera F (2010). Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences 180(10): 2044-2064.
41. Gass SI and Assad AA (2006). An Annotated Timeline of Operations Research: An Informal History. Springer: New York.
42. Gendreau M, Potvin J-Y, Braysy O, Hasle G and Lokketangen A (2008). Metaheuristics for the vehicle routing problem and its extensions: A categorized bibliography. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challenges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 143-169.
43. Ghobbar AA and Friend CH (2003). Evaluation of forecasting methods for intermittent parts demand in the field of aviation: A predictive model. Computers & Operations Research 30(14): 2097-2114.
44. Golden BL, Assad AA,Wasil WA and Baker E (1986). Experimentation in optimization. European Journal of Operational Research 27(1): 1-16.
45. Golden BL and Stewart WR (1985). The Traveling Salesman Problem. chap. Empirical Analysis of Heuristics, John Wiley & Sons: Chichester, UK, pp 207-249.
46. Greenberg HJ (1990). Computational testing:Why, how and how much. ORSA Journal on Computing 2(1): 94-97.
47. Hellekalek P (1998). Good random number generators are (not so) easy to find. Mathematics and Computers in Simulation 46(5-6): 485-505.
48. Hoffman A, Mannos M, Sokolowsky D and Wiegmann N (1953). Computational experience in solving linear programs. Journal of the Society for Industrial and Applied Mathematics 1(1): 17-33.
49. Hooker JN (1994). Needed: An empirical science of algorithms. Operations Research 42(2): 201-212.
50. Hooker JN (1995). Testing heuristics - We have it all wrong. Journal of Heuristics 1(1): 33-42.
51. Hooker JN (2007). Good and bad futures for constraint programming (and operations research). Constraint Programming Letters 1: 21-32.
52. Hubbard R and Armstrong JS (1994). Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing 11(3): 233-248.
53. Hubbard R and Vetter DE (1996). An empirical comparison of published replication research in accounting, economics, finance, management, and marketing. Journal of Business Research 35(2): 153-164.
54. Ignizio JP (1971). On the establishment of standards for comparing algorithm performance. Interfaces 2(1): 8-11.
55. Ince DC, Hatton L and Graham-Cumming J (2012). The case for open computer programs. Nature 482(7386): 485-488.
56. Ioannidis JA (2005). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association 294(2): 218-228.
57. Jackson RHF, Boggs PT, Nash SG and Powell S (1991). Guidelines for reporting results of computational experiments. Report of the ad hoc committee. Mathematical Programming 49(1-3): 413-425.
58. Jackson RHF and Mulvey JM (1978). A critical review of methods for comparing mathematical programming algorithms and software (1953-1977). Journal of Research of the National Bureau of Standards 83(6): 563-584.
59. Jaillet P and Wagner MR (2008). Online vehicle routing problems: A survey. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challenges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 221-237.
60. Jourdan L, Basseur M and Talbi E-G (2009). Hybridizing exact methods and metaheuristics: A taxonomy. European Journal of Operational Research 199(3): 620-629.
61. Kendall G, Knust S, Ribeiro CC and Urrutia SS (2010). Scheduling in sports: An annotated bibliography. Computers & Operations Research 37(1): 1-19.
62. Kiran AS and Smith ML (1984). Simulation studies in job shop scheduling - 1. A survey. Computers & Industrial Engineering 8(2): 87-93.
63. Kirkpatrick S, Gelatt Jr CD and Vecchi M (1983). Optimization by simulated annealing. Science 220(4598): 671-680.
64. Laporte G (2009). Fifty years of bvehicle routing. Transportation Science 43(4): 408-416.
65. Lee C-Y, Bard J, Pinedo M and Wilhelm WE (1993). Guidelines for reporting computational resuts in IIE transactions. IIE Transactions 25(6): 121-123.
66. Lin EYH (1998). A bibliographical survey on some well-known nonstandard knapsack problems. INFOR 36(4): 274-317.
67. Loiola EM, de Abreu NMM, Boaventura-Netto PO, Hahn P and Querido T (2007). A survey for the quadratic assignment problem. European Journal of Operational Research 176(2): 657-690.
68. Lukasiak P, Błazewicz J and Milostan M (2010). Some operations research methods for analyzing protein sequences and structures. Annals of Operations Research 175(1): 9-35.
69. Lust T and Teghem J (2010). The multiobjective traveling salesman problem: A survey and a new approach. In: Coello CAC, Dhaenens C and Jourdan L (eds). Studies in Computational Intelligence. Vol. 272, Springer-Verlag: Berlin, pp 119-141.
70. Lust T and Teghem J (2012). The multiobjective multidimensional knapsack problem: A survey and a new approach. International Transactions in Operational Research 19(4): 495-520.
71. Marshall E (1983). The murky world of toxicity testing. Science 220(4602): 1130-1132.
72. McCollum B et al (2010). Setting the research agenda in automated timetabling: The second international timetabling competition. INFORMS Journal on Computing 22(1): 120-130.
73. McGeoch CC (1996). Toward an experimental method for algorithm simulation. INFORMS Journal on Computing 8(1): 1-15.
74. Miller HE (1976). Personnel scheduling in public systems - A survey. Socio-Economic Planning Sciences 10(6): 241-249.
75. Miller DM and Williams D (2003). Shrinkage estimators of time series seasonal factors and their effect on forecasting accuracy. International Journal of Forecasting 19(4): 669-684.
76. Pentico DW (2007). Assignment problems: A golden anniversary survey. European Journal of Operational Research 176(2): 774-793.
77. Pisinger D (2007). The quadratic knapsack problem - A survey. Discrete Applied Mathematics 155(5): 623-648.
78. Prinz F, Schlange T and Asadullah K (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery 10(7391): 10-11.
79. Rardin RL and Uzsoy R (2001). Experimental evaluation of heuristic optimization algorithms: A tutorial. Journal of Heuristics 7(3): 261-304.
80. Rasmussen RV and Trick MA (2008). Round robin scheduling - A survey. European Journal of Operational Research 188(3): 617-636.
81. Russell JF (2013). If a job is worth doing, it is worth doing twice. Nature 496(7443): 7.
82. Salkin HM and Kluyver CAD (1975). Knapsack problem - Survey. Naval Research Logistics 22(1): 127-144.
83. Sörensen K (2015). Metaheuristics - The metaphor exposed. International Transactions in Operational Research 22(1): 3-18.
84. Sörensen K and Glover F (2013). Encyclopedia of Operations Research and Management Science. chap. Metaheuristics, 3rd edn. Springer: New York.
85. Taillard ÉD, Waelti P and Zuber J (2008). Few statistical tests for proportions comparison. European Journal of Operational Research 185(3): 1336-1350.
86. Talbi E-G (2009). Metaheuristics: From Design to Implementation. Wiley: Hoboken, NJ.
87. Trapero JR, Kourentzes N and Fildes R (2015). Identification of sales forecasting models. Journal of the Operational Research Society 66(2): 299-307.
88. Vaux DL (2012). Research methods: Know when your numbers are significant. Nature 492(7428): 180-181.
89. Vidal T, Crainic TG, Gendreau M and Prins C (2013). Heuristics for multiattribute vehicle routing problems: A survey and synthesis. European Journal of Operational Research 231(1): 1-21.
90. Vines TH et al (2013). The availability of research data declines rapidly with article age. Current Biology 24(1): 94-97.
91. Wilbaut C, Hanafi S and Salhi S (2008). A survey of effective heuristics and their application to a variety of knapsack problems. IMA Journal of Management Mathematics 19(3): 227-244.
92. Zanakis SH, Evans JR and Vazacopoulos AA (1989). Heuristic methods and applications: A categorized survey. European Journal of Operational Research 43(1): 88-110.