Sifting data in the real world

Nuclear Instruments and Methods in Physics Research, Section A: Accelerators, Spectrometers, Detectors and Associated Equipment

Volumn 556, Issue 1, 2006, Pages 308-324

  Block, M M ^a  

^a NORTHWESTERN UNIVERSITY   (United States)

Author keywords

Adaptive algorithm; Experimental data; Phenomenology

Indexed keywords

COMPUTER SIMULATION; MATHEMATICAL MODELS; NORMAL DISTRIBUTION; SCATTERING; SPURIOUS SIGNAL NOISE;

ADAPTIVE ALGORITHM; EXPERIMENTAL DATA; PHENOMENOLOGY;

DATA PROCESSING;

EID: 29144525971     PISSN: 01689002     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.nima.2005.10.019     Document Type: Article

References (10)

1. K. Hagiwara et al., Phys. Rev. D 66 (2002) 010001.
2. The χ 2 probability density distribution has ν, the number of degrees of freedom, as its mean value and has a variance equal to 2 ν. To have an intuitive feeling for the goodness-of-fit, i.e., the probability that χ 2 > χ min 2, we note that for the large number of degrees of freedom ν that we are considering in this note, the probability density distribution for χ 2 is well approximated by a Gaussian, with a mean at ν and a width of 2 ν, where 0 < χ 2 < ∞ (n.b., the usual lower limit of - ∞ is truncated here to 0, since by definition χ 2 ≥ 0 ). In this approximation, we have the most probable situation if χ min 2 / ν = 1, which corresponds to a goodness-of-fit probability of 0.5. The chance of having small χ min 2 ∼ 0, corresponding to a goodness-of-fit probability ∼ 1, is exceedingly small. In our computer-generated example of a straight line fit with ν = 103, the fit first can be considered to become poor - say by three standard deviations - when χ min 2 > 146, yielding χ min 2 / ν > 1.41. We found a renormalized χ min 2 / ν = 1.01, indicating a very good fit.
3. In this context, a random distribution means a uniform distribution between a and b, generated by a random number generator that has a flat output between 0 and 1. A normally distributed (Gaussian) distribution means using a Gaussian random number generator that has as its output random numbers y i distributed normally about ȳ, with a probability density 1 / 2 π exp - 1 2 ( (ȳ - y i ) / σ i ) 2, where σ i represents the error (standard deviation) of the point y i.
4. The fact that r χ 2 is greater than 1 is counter-intuitive. Consider the case of generating a Gaussian distribution with unit variance about the value y = 0. If we were to define Δ χ i 2 ≡ (y i - 0 ) 2 = y i 2, with Δ being the cut Δ χ i 2 max, then the truncated differential probability distribution would be P (x ) = 1 / 2 π exp (- x 2 / 2 ) for - Δ ≤ x ≤ + Δ, whose rms value clearly is less than 1 - after all, this distribution is truncated compared to its parent Gaussian distribution. However, that is not what we are doing. What we do is to first make a robust fit to each untruncated event that was Gaussianly generated with unit variance about the mean value zero. For every event we then find the value y 0, its best fit parameter, which, although close to zero with a mean of zero, is non-zero. In order to obtain the truncated event whose width we sample with the next χ 2 fit, we use Δ χ i 2 ≡ (y i - y 0 ) 2. It is the jitter in y 0 about zero that is responsible for the rms width becoming greater than 1. This result is true even if the first fit to the untruncated data were a χ 2 fit.
5. In deriving these equations, we have employed real analytic amplitudes derived using unitarity, analyticity, crossing symmetry, Regge theory and the Froissart bound.
6. Attributed in "Numerical Recipes" [7] to G. E. P. Box in 1953. A very simple example of a robust estimator is to use the median of a discrete distribution rather than the mean to characterize a typical characteristic of the distribution. For example, the "average price" of a home in a luxury resort area, which had a few 25 million dollar homes - a few outliers at very large values of the distribution - could be seriously distorted and essentially meaningless, whereas the median would scarcely be affected.
7. W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vettering, Numerical Recipes, The Art of Scientific Computing, Cambridge University Press, Cambridge, 1986, pp. 289-293. There is also an excellent discussion of modeling of data, including a section on confidence limits by Monte Carlo simulation, in Chapter 14.
8. P.J. Huber Robust Statistics 1981 Wiley New York
9. F. Hampel Robust Statistics: The Approach Based on Influence Functions 1986 Wiley New York
10. P.J. Rousseeuw, A.M. Leroy, Robust Regression and Outlier Detection, Wiley, New York, 1987. Robust regression is also included in the R and S languages for statistical analysis.