Single-pass low-storage arbitrary quantile estimation for massive datasets

Statistics and Computing

Volumn 13, Issue 2, 2003, Pages 91-100

  Liechty, John C ^a   Lin, Dennis K J ^b   McDermott, James P ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

^b Department of Supply Chain ^*  (United States)

Author keywords

Data mining; Large datasets; Low storage; Quantile estimation; Single pass algorithms; Tail quantile

Indexed keywords

EID: 0346463260     PISSN: 09603174     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1023296123228     Document Type: Article

References (14)

1. Bahadur R.R. 1966. A note on quantites in large samples. Annals of Mathematical Statistics 37: 577-580.
2. Breiman L., Gins J., and Stone C. 1979. New Methods for Estimating Tail Probabilities and Extreme Value Distributions. Technology Service Corp. Santa Monica, CA, TSC-PD-A2261.
3. Chao M.T and Lin G.D. 1993. The asympotic distribution of the remedians. Journal of Statistical Planning and Inference 37: 1-11.
4. Dunn C.L. 1991. Precise simulated percentiles in a pinch. The American Statistician 45(3): 207-211.
5. Hurley C. and Modarres R. 1995. Low-Storage quantile estimation. Computational Statistics 10(4): 311-325.
6. Kesidis G. 1999. Bandwidth adjustments using on-line packet-level adjustments. In: SPIE Conference on Performance and Control of Network Systems, Boston, Sept. 19-22.
7. Krutchkoff R.G. 1986. Percentiles by simulation: Reducing time and storage. Journal of Statistical Computation and Simulation 25: 304-305.
8. Manku G.S., Rajagopalan S., and Lindsay B.C. 1998. Approximate medians and other quantiles in one pass and with limited memory. In: Proc. ACM SIGMOD International Conf. on Management of Data June, pp. 426-435.
9. Pearl J. 1981. A space-efficient on-line method of computing quantile estimates. Journal of Algorithms 2: 164-177.
10. Ott W.R. 1995. Environmental Statistics and Data Analysis. Lewis Publishers.
11. Pfanzagl J. 1974. Investigating the quantile of an unknown distribution. Contributions to Applied Statistics, Ziegler WJ. (Ed.), Birkhauser Verlag, Basel, pp. 111-126.
12. Rousseeuw P.J. and Bassett G.W. 1990. The remedian: A robust averaging method for large datasets. Journal of the American Statistical Association 85(409): 97-104.
13. Serfling R.J. 1980. Approximation Theorems of Mathematical Statistics. Wiley, New York.
14. Tierney L. 1983. A space-efficient recursive procedure for estimating a quantile of an unknown distribution. SIAM Journal on Scientific and Statistical Computing 4(4): 706-711.