Outlier detection in near-infrared spectroscopic analysis by using Monte Carlo cross-validation

Science in China, Series B: Chemistry

Volumn 51, Issue 8, 2008, Pages 751-759

  Liu, Zhichao ^a   Cai, Wensheng ^a   Shao, Xueguang ^a  

^a NANKAI UNIVERSITY   (China)

Author keywords

Monte Carlo cross validation; Near infrared spectrum; Outlier detection; Partial least squares(PLS)

Indexed keywords

INFRARED DEVICES; KETONES; MAGNETIC RESONANCE SPECTROSCOPY; MONTE CARLO METHODS; PHILOSOPHICAL ASPECTS; PRESSES (MACHINE TOOLS); PROBABILITY; RANDOM PROCESSES; RISK ASSESSMENT; SPECTRUM ANALYZERS; SPEECH; TESTING;

CROSS-VALIDATION (CV); CROSS-VALIDATION METHODS; DATA SETS; HIGHLY EFFICIENT; LARGE DATA; LEAVE-ONE-OUT (LOO); MONTE CARLO (MC); NEAR INFRA-RED (NIR); OUTLIER DETECTION; PARTIAL LEAST SQUARES (PLS) MODELS; RESIDUAL ERRORS; SUM OF SQUARES (SOS);

SPECTROSCOPIC ANALYSIS;

EID: 47749116004     PISSN: 10069291     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11426-008-0080-x     Document Type: Article

References (49)

1. Siesler H W, Ozaki Y, Kawata S, Heise H M. Near-infrared spectroscopy, principles, instruments, applications. New York: Wiley, 2002. 11-35
2. Massart D L, Vandeginste B G M, et al. Handbook of chemometrics and Qualimetrics: Part A. Amsterdam: Elsevier, 1997. 331-340
3. Pell R J. Multiple outlier detection for multivariate calibration using robust statistical techniques. Chemom Intell Lab Syst, 2000, 52(1): 87-104
4. Chen D, Shao X G, Hu B, Su Q D. Simultaneous wavelength selection and outlier detection in multivariate regression of near-infrared spectra. Anal Sci, 2005, 21(2): 161-166
5. Karstang T V, Manne R. Optimized scaling: a novel approach to linear calibration with closed data sets. Chemom Intell Lab Syst, 1992, 14(1-3): 165-173
6. Geladi P, MacDougall D, Martens H. Linearization and scatter correction for near infrared reflectance spectra of meat. Appl Spectrosc, 1985, 39(3): 491-500
7. Wold S, Antti H, Lindgren F, Ohman J. Orthogonal signal correction of near-infrared spectra. Chemom Intell Lab Syst, 1998, 44(1-2): 175-185
8. Chen D, Hu B, Shao X G, Su Q D. Removal of major interference sources in aqueous near-infrared spectroscopy techniques. Anal Bioanal Chem, 2004, 379(4): 714-719
9. Westerhuis J A, De Jong S, Smilde A K. Direct orthogonal signal correction. Chemom Intell Lab Syst, 2001, 56(1): 13-25
10. Shao X G, Leung A K M, Chau F T. Wavelet: a new trend in chemistry. Acc Chem Res, 2003, 36(4): 276-283
11. Chen D, Hu B, Shao X G, Su Q D. Variable selection by modified IPW (iterative predictor weighting)-PLS (partial least squares) in continuous wavelet regression models. Analyst, 2004, 129(7): 664-669
12. Shao X G, Zhuang Y Z. Determination of chlorogenic acid in plant samples by using near-infrared spectrum with wavelet transform preprocessing. Anal Sci, 2004, 20(3): 451-454
13. Ma C X, Shao X G. Continuous wavelet transform applied to removing the fluctuating background in near-infrared spectra. J Chem Inf Comput Sci, 2004, 44(3): 907-911
14. Norgaard L, Saudaland A, Wagner J, Nielsen J P, Munck L, Engelsen S B. Interval partial least-squares regression (iPLS): a comparative chemometric study with an example from near-infrared spectroscopy. Appl Spectrosc, 2000, 54(3): 413-419
15. Leardi R, Norgaard L. Sequential application of backward interval PLS and genetic algorithms for the selection of relevant spectral regions. J Chemom, 2004, 18(11): 486-497
16. Lindgren F, Geladi P, Rannar S, Wold S. Interactive variable selection (IVS) for PLS1. Theory and algorithms. J Chemom, 1994, 8(5): 349-363
17. Lindgren F, Geladi P, Berglund A, Sjostrom M, Wold S. Interactive variable selection (IVS) for PLS2. chemical applications. J Chemom, 1995, 9(5): 349-363
18. Centner V, Massart D L, De Noord O E, De Jong S, Vandeginste B M, Sterna C. Elimination of uninformative variables for multivariate calibration. Anal Chem, 1996, 68(21): 3851-3858
19. Shao X G, Wang F, Chen D, Su Q. D. A method for near-infrared spectral calibration of complex plant samples with wavelet transform and elimination of uninformative variables. Anal Bioanal Chem, 2004, 378(5): 1382-1387
20. Vapnik V. The nature of statistical learning theory. New York: Springer-Verlag, 1995. 138-167
21. Belousov A I, Verzakov S A, Von Frese J. Applicational aspects of support vector machines. J Chemom, 2002, 16(8-10): 482-489
22. Li Y K, Shao X G, Cai W S. A consensus least squares support vector regression (LS-SVR) for analysis of near-infrared spectra of plant samples. Talanta, 2007, 72(1): 217-222
23. Jin L, Pierna J A F, Xu Q, Wahl F, De Noord O E, Saby C A, Massart D L. Delaunay triangulation method for multivariate calibration. Anal Chim Acta, 2003, 488(1): 1-14
24. Liang Y Z, Kvalheim O M, Robust methods for multivariate analysis-a tutorial review. Chemom Intell Lab Syst, 1996, 32(1): 1-10
25. Belsley D A, Kuh E, Welsch R E. Regression Diagnostics. New York: Wiley, 1980. 6-84
26. Weisberg S, Applied Linear Regression (Third Edition). New York: Wiley, 2005. 167-193
27. Daszykowski M, Kaczmarek K, Heyden V Y, Walczak B. Robust statistics in data analysis: A review Basic concepts. Chemom Intell Lab Syst, 2007, 85(2): 203-219
28. Walczak B, Massart D L. Multiple outlier detection revisited. Chemom Intell Lab Syst, 1998, 41(1): 1-15
29. Devlin J S, Gnanadesikan R, Kettering J R. Robust estimation of dispersion matrix and principal components. J Am Stat Assoc, 1981, 76(374): 354-362
30. Rousseeuw P J, Leroy A M. Robust Regression and Outlier Detection, New York: John Wiley & Sons Inc., 1987, 216-247
31. Pena D, Yohai V. A Fast procedure for outlier diagnostics in large regression systems. J Am Stat Assoc, 1999, 94(446): 434-445
32. Zhang M H, Xu Q S, Massart D L. Robust principal components regression based on principal sensitivity vectors. Chemom Intell Lab Sys, 2003, 67(2): 175-185
33. Hubert M, Rousseeuw P J, Verboven S. A fast method for robust principal components with application to chemometrics. Chemom Intell Lab Sys, 2002, 60(1-2): 101-111
34. Croux C, Ruiz-Gazen A. High breakdown estimators for principal components: The projection-pursuit approach revisited. J Mult Anal, 2005, 95(1): 206-226
35. Branden K V, Hubert M. Robustness properties of a robust partial least squares regression method. Anal Chim Acta, 2004, 515(1): 229-241
36. Cummins D J, Andrews C W. Iteratively reweighted partial least squares: a performance analysis by Monte Carlo simulation. J Chemom, 1995, 9(6): 489-507
37. Gil J A, Romera R. On robust partial least squares PLS methods. J Chemom, 1998, 12(6): 365-378
38. Serneels S, Croux C, Filzmoser P, Van Espen P J. Partial robust M-regression. Chemom Intell Lab Syst, 2005, 79(1-2): 55-64
39. De Vries S, Cajo J F, Braak T. Prediction error in partial least squares regression: A critique on the deviation used in the Unscrambler. Chemom Intell Lab Syst, 1995, 30(2): 239-245
40. Hoy M, Steen K, Martens H. Review of partial least squares regression prediction error in Unscrambler. Chemom Intell Lab Syst, 1998, 44(1-2): 123-133
41. Seidel R. The upper bound theorem for polytopes: An easy proof of its asymptotic version. Comput Geom Theory Appl, 1995, 5(2): 115-116
42. Fernandez P J A, Jin L, Daszykowski M, Wahl F, Massart D L. A methodology to detect outliers/inliers in prediction with PLS. Chemom Intell Lab Syst, 2003, 68(1-2): 17-28
43. Xu Q S, Liang Y Z. Monte Carlo cross validation. Chemom Intell Lab Syst, 2001, 56(1): 1-11
44. Gourvenec S, Fernandez Pierna J A, Massart D L, Rutledge D N. An evaluation of the PoLiSh smoothed regression and the Monte Carlo cross-validation for the determination of the complexity of a PLS model. Chemom Intell Lab Syst, 2003, 68(1): 41-51
45. Chatterjee S, Hadi A S. Influential observations, high leverage points, and outliers in linear regression. Stat Sci, 1986, 1(3): 379-416
46. Haaland D M, Thomas E V. Partial least squares methods for spectral analysis 1: Relation to other multivariate calibration methods and the extraction of qualitative information. Anal Chem, 1988, 60(11): 1193-1202
47. Kalivas J H. Two data sets of near infrared spectra. Chemom Intell Lab Syst, 1997, 37(2): 255-259
48. Koshoubu J, Iwata T, Minami S. Elimination of the uninformative calibration sample subset in the modified UVE-PLS method. Anal Sci, 2001, 17(2): 319-322
49. Wahba G, Wold S. A completely automatic french curve: Fitting spline functions by cross validation. Commun Stat, 1975, 4(1): 1-17