Evaluation of three methods for calculating statistical significance when incorporating a systematic uncertainty into a test of the background-only hypothesis for a Poisson process

Nuclear Instruments and Methods in Physics Research, Section A: Accelerators, Spectrometers, Detectors and Associated Equipment

Volumn 595, Issue 2, 2008, Pages 480-501

  Cousins, Robert D ^a   Linnemann, James T ^b   Tucker, Jordan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

Confidence interval; Hypothesis test; Systematic uncertainties

Indexed keywords

BOOLEAN FUNCTIONS; ERROR ANALYSIS; GAMMA RAYS; HIGH ENERGY PHYSICS; IONIZING RADIATION; POISSON EQUATION; POISSON RATIO; SATELLITE OBSERVATORIES; STATISTICAL METHODS; THREE TERM CONTROL SYSTEMS;

CONFIDENCE INTERVAL; HYPOTHESIS TEST; PARAMETER SPACES; SYSTEMATIC UNCERTAINTIES;

PARAMETER ESTIMATION;

EID: 51649119067     PISSN: 01689002     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.nima.2008.07.086     Document Type: Article

References (50)

1. For a recent review, see R.D. Cousins, Treatment of nuisance parameters in high energy physics, and possible justifications and improvements in the statistics literature, in: Proceedings of PhyStat 05: Statistical Problems in Particle Physics, Astrophysics and Cosmology (Oxford, September 12-15, 2005) 〈http://www.physics.ox.ac.uk/phystat05/proceedings/〉. See also the response following by N. Reid.
2. For a comprehensive discussion of hypothesis testing, see A. Stuart, K. Ord, S. Arnold, Kendall's Advanced Theory of Statistics, vol. 2A, sixth ed., Arnold, London, 1999, and earlier editions by Kendall and Stuart. From the formal correspondence between hypothesis tests and confidence intervals in Chapter 20, all the frequentist hypothesis tests in this paper can be restated in terms of confidence intervals and limits.
3. Berger J.O., and Sellke T. J. Am. Stat. Assoc. 82 (1987) 112
4. Sellke T., Bayarri M.J., and Berger J.O. Am. Stat. 55 (2001) 62
5. J.T. Linnemann, Measures of significance in HEP and astrophysics, in: Proceedings of PhyStat 2003: Statistical Problems in Particle Physics, Astrophysics, and Cosmology, SLAC, Stanford, CA, USA, September 8-11, 2003 [arXiv:physics/0312059] 〈http://www.slac.stanford.edu/econf/C030908/papers/MOBT001.pdf〉.
6. Zhang S.N., and Ramsden D. Exp. Astron. 1 (1990) 145 The figures are missing from the published paper but are in the Ph.D. Thesis of Zhang, who kindly provided them to us.
7. F. James, MINUIT. Function Minimization and Error Analysis 〈http://wwwasdoc.web.cern.ch/wwwasdoc/minuit/minmain.html〉.
8. Li T.-P., and Ma Y.-Q. Astrophys. J. 272 (1983) 317
9. Rolke W.A., and Lopez A.M. Nucl. Instr. and Meth. A 458 (2001) 745
10. Rolke W.A., Lopez A.M., and Conrad J. Nucl. Instr. and Meth. A 551 (2005) 493 [arXiv:physics/0403059]
11. K.S. Cranmer, Frequentist hypothesis testing with background uncertainty, in: Proceedings of PhyStat 2003: Statistical Problems in Particle Physics, Astrophysics, and Cosmology, SLAC, Stanford, CA, USA, September 8-11, 2003 [arXiv:physics/0310108] 〈http://www.slac.stanford.edu/econf/C030908/papers/WEMT004.pdf〉.
12. b (b in his notation) is known by the experimenter when calculating the rms of the Gaussian but unknown when calculating the mean. This does not cause a significant problem for his example of b = 100 in which he used a self-consistent set of numbers. But for the profile likelihood for the more general Gaussian-mean problem, one must use Eqn. 21 in the present paper (which is consistent with the normal equations used by Ref. [10]).
13. J. Heinrich, Review of the Banff challenge on upper limits, in: Talk at PHYSTAT-LHC Workshop on Statistical Issues for LHC Physics, CERN, 27-29 June 2007 〈http://phystat-lhc.web.cern.ch/phystat-lhc/〉.
14. Przyborowski J., and Wilenski H. Biometrika 31 (1940) 313
15. Reid N. Stat. Sci. 10 (1995) 138
16. James F., and Roos M. Nucl. Phys. B 172 (1980) 475
17. Gehrels N. Astrophys. J. 303 (1986) 336
18. Clopper C.J., and Pearson E.S. Biometrika 26 (1934) 404
19. W.H. Press, et al., Numerical Recipes in C, second ed., Cambridge, 1992.
20. ROOT, An object-oriented data analysis framework 〈http://root.cern.ch〉.
21. Majumder K.L., and Bhattacharjee G.P. Appl. Stat. 22 (1973) 409. http://lib.stat.cmu.edu/apstat/63 See also 〈http://lib.stat.cmu.edu/apstat/63〉
22. Cousins R.D. Nucl. Instr. and Meth. A 417 (1998) 391
23. Berger J.O., Liseo B., and Wolpert R.L. Stat. Sci. 14 (1999) 1
24. Box G.E.P. J. R. Stat. Soc. Ser. A (General) 143 (1980) 383
25. Cousins R.D., and Highland V.L. Nucl. Instr. and Meth. A 320 (1992) 331
26. Tegenfeldt F., and Conrad J. Nucl. Instr. and Meth. A 539 (2005) 407 arXiv:physics/0408039
27. A. Stuart, J. Keith Ord, Kendall's Advanced Theory of Statistics, Distribution Theory, vol. 1, sixth ed., Arnold, London, 1994 (see also earlier editions by Kendall and Stuart).
28. L. Demortier, private communication, including the pointer to Ref. [24].
29. N and other aspects of this computer program need further study, in our opinion.).
30. K. Cranmer, B. Mellado, W. Quayle, S.L. Wu, Statistical methods to assess the combined sensitivity of the ATLAS detector to the Higgs Boson in the standard model, ATL-PHYS-2004-034 〈http://cdsweb.cern.ch/record/726480〉; and further comments by Kyle Cranmer, private communication.
31. J. Linnemann, Upper limits and priors, in: Talk at Workshop on Confidence Limits, Fermilab, 2000 〈http://conferences.fnal.gov/cl2k/copies/linnemann1.pdf〉.
32. Alexandreas D.E., et al. Nucl. Instr. and Meth. A 328 (1993) 570
33. Cousins R.D. Am. J. Phys. 63 (1995) 398
34. H.K. Kim (then a graduate student at Michigan State University), private communication to JTL.
35. James F. Comput. Phys. Commun. 20 (1980) 29
36. Wilks S.S. Ann. Math. Stat. 9 (1938) 60
37. Babu G.J., and Feigelson E.D. Astrostatistics (1996), Chapman & Hall, London
38. Fraser D.A.S. J. Am. Stat. Assoc. 86 (1991) 258
39. Fraser D.A.S., Reid N.N., and Wu J. Biometrika 86 (1999) 249
40. Artificial case suggested by an example in Ref. [32].
41. Abe F., et al., CDF Collaboration. Phys. Rev. Lett. 74 (1995) 2626 The example uses a dilepton analysis
42. Abachi S., et al., D0 Collaboration. Phys. Rev. Lett. 74 (1995) 2422
43. Abachi S., et al., D0 Collaboration. Phys. Rev. Lett. 74 (1995) 2632
44. Two artificial examples from Ref. [6].
45. An artificial example with small τ.
46. Aharonian F., et al. Astron. Astrophys. 393 (2002) L37
47. Reynolds P.T., et al. Astrophys. J. 404 (1993) 206
48. Atkins R., et al., Milagro Collaboration. Astrophys. J. 595 (2003) 803
49. In: Abramowitz M., and Stegun I.A. (Eds). Handbook of Mathematical Functions (1968), Dover, New York
50. Rose C., and Smith M.D. Mathematical Statistics with Mathematica (2002), Springer, New York