Preprocessing Methods

Comprehensive Chemometrics

Volumn 3, Issue , 2009, Pages 121-231

  Zeaiter, M ^a   Rutledge, D ^b  

^a GLAXOSMITHKLINE   (United Kingdom)

^b Université Paris Saclay   (France)

Author keywords

Alignment methods; Baseline correction; Calibration; Centering; Data set selection; Derivatives; Detrending; Dimensionality reduction; Fourier transform; Normalization; Orthogonal projection; Orthogonal signal correction; Preprocessing; Principal component analysis; Scaling; Smoothing; Transformation methods; Warping

Indexed keywords

EID: 84882562211     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-044452701-1.00074-0     Document Type: Chapter

References (97)

1. Vandeginste B., Massart D., Buydens L., De Jong S., Lewi P., Smeyers-Verbeke J. Handbook of Chemometics and Qualimetrics: Part B 1998, Elsevier, Amsterdam.
2. Dyrby M., Engelsen S.B., Nørgaard L., Bruhn M., Lundsberg Nielsen L. Chemometric Quantitation of the Active Substance in a Pharmaceutical Tablet Using Near Infrared (NIR) Transmittance and NIR FT Raman Spectra. Appl. Spectrosc. 2002, 56(5):579-585.
3. Center of Agricultural and Environmental Engineering Research- Cemagref - Montpellier, France.
4. Wülfert F., Kok W.T., Smilde A.K. Influence of Temperature on Vibrational Spectra and Consequences for the Predictive Ability of Multivariate Models. Anal. Chem. 1998, 70:1761-1767.
5. Research Center of Wine and Vinery, PechRouge, Institut National de Recherche Agronomique INRA - Narbonne, France.
6. Laboratory of Biotechnologies and Environment, LBE, Institut National de Recherche Agronomique INRA - Narbonne, France.
7. Designation E1655-05, in. ASTM Annual Book of Standards 2005, ASTM, Standard practices for infrared multivariate quantitative analysis, West Conshohocken, PA, USA.
8. Honigs D.E., Hiftje G.M., Hirtshfield T. Number of Samples and Wavelengths Required for the Training Set in Near-Infrared Reflectance Spectroscopy. Appl. Spectrosc. 1984, 38(6):844-847.
9. Ferre J., Rius F.X. Constructing D-Optimal Designs from a List of Candidate Samples. Trends Anal. Chem. 1997, 16(2):70-73.
10. Jouan-Rimbaud D., Massart D.L., Saby C.A., Puel C. Characterisation of the Representativity of Selected Sets in Multivariate Calibration and Pattern Recognition. Anal. Chim. Acta 1997, 350:149-161.
11. Kennard R.W., Stone L.A. Computer Aided Design of Experiments. Technometrics 1969, 11:137-148.
12. Pukelsheim F. Optimal Design of Experiments 1993, John Wiley & Sons, New York.
13. de Aguiar P.F., Bourguignon B., Khots M.S., Massart D.L., Phan-Than-Luu R. D-Optimal Designs. Chemom. Intell. Lab. Sys. 1995, 30(2):199-210.
14. Snee R.D. Validation of Regression Models: Methods and Examples. Technometrics 1977, 19:415-428.
15. Isaksson T., Naes T. Selection of Samples for Calibration in Near-Infrared Spectroscopy. II. Selection Based on Spectral Measurements. Appl. Spectrosc. 1990, 44(7):1152-1158.
16. Puchwein G. Selection of Calibration Samples for Near-Infrared Spectrometry by Factor Analysis of Spectra. Anal. Chem. 1988, 60:569-573.
17. Bro R., Smilde A.K. Centering and Scaling in Component Analysis. J. Chemom. 2003, 17(16-33):62-71.
18. Van der Voet H. Corrigendum to 'Comparing the Predictive Accuracy of Models Using a Simple Randomization Test'. Chemom. Intell. Lab. Sys. 1994, 25:313-323.
19. Seasholtz M.B., Kowalski B.R. The Effect of Mean Centering on Prediction in Multivariate Calibration. J. Chemom. 1992, 6:103-111.
20. Zeaiter M., Roger J.-M., Bellon-Maurel V. Robustness of Models Developed by Multivariate Calibration. Part II: The Influence of Preprocessing Methods. Trends Anal. Chem. 2005, 24(5):437-445.
21. Zeaiter, M. On-Line Robust Measurement of Organic Components Using Infrared Spectrosopy and Multivariate Calibration. Ph.D. Thesis, University of Montpellier II Sciences & Technologies of Languedoc, 2004.
22. Barnes R.J., Dhanoa M.S., Lister S.J. Correction of the Description of Standard Normal Variate (SNV) and De-Trend Transformations in Practical Spectroscopy with Applications in Food and Beverage Analysis. J. Near Infrared Spectrosc. 1993, 1:185-186. 2nd ed.
23. Guo Q., Wu W., Massart D.L. The Robust Normal Variate Transform for Pattern Recognition with Near-Infrared Data. Anal. Chim. Acta 1999, 382(1):87-103.
24. Norris K.H., Hart J.R. J. Near Infrared Spectrosc. 1996, 4:23-29.
25. Isaksson T., Næs T. The Effect of Multiplicative Scatter Correction (MSC) and Linearity Improvement in NIR Spectroscopy. Appl. Spectrosc. 1988, 42:1273-1284.
26. Martens H., Næs T. Multivariate Calibration 1991, Wiley, Chichester, pp 7-345.
27. Martens H., Jensen S.A., Geladi P. Nordic Symposium on Applied Statistics. Proceedings of the Nordic Symposium on Applied Statistics 1983, Stokkand Forlag Publishers, Stavanger, Norway, pp 325, ISBN 82-90496-02-8.
28. Martens H., Stark E. Extended Multiplicative Signal Correction and Spectral Interference Subtraction: New Preprocessing Methods for Near Infrared Spectroscopy. J. Pharm. Biomed. Anal. 1991, 9:625-635.
29. Martens H. Extended Multiplicative Scatter Correction (EMSC) http://www.models.kvl.dk/source/EMSCtoolbox/EMSC_Getting_Started.pdf.
30. Tomasi G., van den Bergand F., Andersson C. Correlation Optimized Warping and Dynamic Time Warping as Preprocessing Methods for Chromatographic Data. J. Chemom. 2004, 18:231-241.
31. Forshed J., Schuppe-Koistinen I., Jacobsson S.P. Peak Alignment of NMR Signals by Means of a Genetic Algorithm. Anal. Chim. Acta 2003, 487(2):189-199.
32. Eilers P. Parametric Time Warping. Anal. Chem. 2004, 76:404-411.
33. Johnson K.J., Wright B.W., Jarman K.H., Synovec R.E. High-Speed Peak Matching Algorithm for Retention Time Alignment of Gas Chromatographic Data for Chemometric Analysis. J. Chromatogr. A 2003, 996(1-2):141-155.
34. Kassidas A., MacGregor J.F., Taylor P.A. Synchronization of Batch Trajectories Using Dynamic Time Warping. AIChE J. 1998, 44(4):864-875.
35. Bylund D., Danielsson R., Malmquist G., Markides K.E. Chromatographic Alignment by Warping and Dynamic Programming as a Pre-Processing Tool for PARAFAC Modelling of Liquid Chromatography-Mass Spectrometry Data. J. Chromatogr. A 2002, 961(2):237-244.
36. Nielsen N.-P., Cartensen J.M., Smedsgaard J. Aligning of Single and Multiple Wavelength Chromatographic Profiles for Chemometric Data Analysis Using Correlation Optimised Warping. J. Chromatogr. 1998, 805(1-2):17-35.
37. Pravdova V., Walczak B., Massart D.L. A Comparison of Two Algorithms for Warping of Analytical Signals. Anal. Chim. Acta 2002, 456(1):77-92.
38. Torgrip R.J.O., Åberg M., Karlberg B., Jacobsson S.P. Peak Alignment Using Reduced Set Mapping. J. Chemom. 2003, 17(11):573-582.
39. Walczak B., Wu W. Fuzzy Warping of Chromatograms. Chemom. Intell. Lab. Sys. 2005, 77(1-2):173-180.
40. Nielsen N.-P.V., Carstensen J.M., Smedsgaard J. Aligning of Single and Multiple Wavelength Chromatographic Profiles for Chemometrics Data Analysis Using Correlation Optimised Warping. J. Chromatogr. A 1998, 805:17-35. http://www.ibt.dtu.dk/mycology/cow.htm.
41. Malabat C., Saint-Denis T., Rutledge D.N. Improvement of Chromatographic Analysis by Chemometric Methods Using Warping Algorithms - Application to Milk Analysis 2003, Chimiométrie, Paris, France.
42. Savitzky A., Golay M.J.E. Smoothing and Differentiation of Data by Simplified Least Squares Procedure. Anal. Chem. 1964, 36:1627-1639.
43. Steiner J., Termonia Y., Deltour Y. Comments on Smoothing and Differentiation of Data by Simplified Least Squares Procedure. Anal. Chem. 1972, 11(44):1906.
44. Madden H. Comments on Smoothing and. Differentiation of' Data by Simplified Least Square Procedure. Anal. Chem. 1978, 50(9):1383-1386.
45. Barak P. Smoothing and Differentiation by An Adaptive-Degree Polynomial Filter. Anal. Chem. 1995, 67:2758-2762.
46. Vivó-Truyols G., Schoenmakers P.J. Automatic Selection of Optimal Savitzky-Golay Smoothing. Anal. Chem. 2006, 78:4598-4608.
47. Rännar S., MacGregor J.F., Wold S. Adaptive Batch Monitoring Using Hierarchical PCA. Chemom. Intell. Lab. Sys. 1998, 41(1):73-81.
48. Barros A.S., Rutledge D.N. Segmented Principal Component Transform-Principal Component Analysis. Chemom. Intell. Lab. Sys. 2005, 78(1-2):125-137.
49. Krishnaveni V., Jayaraman S., Manoj Kumar P.M., Shivakumar K., Ramadoss K. Comparison of Independent Component Analysis Algorithms for Removal of Ocular Artifacts from Electroencephalogram. Meas. Sci. Rev. 2005, 5(2):67.
50. Mozaffar, S.; Petr, D. W. Artefact Extraction from EEG Data Using Independent Component Analysis; Technical Report, Information and Telecommunication Technology Center, The University of Kansas, 2002.
51. Jouan-Rimbaud Bouveresse D., Benabid H., Rutledge D.N. Independent Component Analysis as a Pretreatment Method for Parallel Factor Analysis to Eliminate Artefacts from Multiway Data. Anal. Chim. Acta 2007, 589(2):216-224.
52. Hyvärinen A., Oja E. Independent Component Analysis: A Tutorial from. http://www.cis.hut.fi/projects/ica/.
53. Hyvärinen A., Oja E.A. Fast Fixed-Point Algorithm for Independent Component Analysis. Neural Comput. 1997, 9(7):1483-1492.
54. Cardoso J.-F. High-Order Contrasts for Independent Component Analysis. Neural Comput. 1999, 11(1):157-192.
55. Zarzoso V., Murillo-Fuentes J.J., Boloix-Tortosa R., Nandi A.K. Optimal Pairwise Fourth-Order Independent Component Analysis. IEEE Trans. Signal Process 2006, 54:3049-3063.
56. Malinowski E.R., Howery D.G. Factory Analysis in Chemistry 1980, Wiley, New York. 1st ed.
57. Jolliffe I.T. Rotation of Principal Components: Some Comments. Int. J. Climatol. 1987, 7(5):507-510.
58. Wu W., Massart D.L., de Jong S. The Kernel PCA Algorithms for Wide Data. Part I: Theory and Algorithms. Chemom. Intell. Lab. Sys. 1997, 36(2):165-172.
59. Wold H. Estimation of Principal Components and Related Models by Iterative Least Squares. Multivariate Analysis 1966, 391-420. Academic Press, New York. P.R. Krishnaiah (Ed.).
60. Barros A.S., Rutledge D.N. Principal Components Transform-Partial Least Squares: A Novel Method to Accelerate Cross-Validation in PLS Regression. Chemom. Intell. Lab. Sys. 2004, 73(2):245-255.
61. Wavelets in Chemistry 2000, Elsevier, Amsterdam. B. Walczak (Ed.).
62. Jensen A., la Cour-Harbo A. Ripples in Mathematics, The Discrete Wavelet Transform 2001, Springer-Verlag, Berlin.
63. Trygg J., Wold S. PLS Regression on Wavelet Compressed NIR Spectra. Chemom. Intell. Lab. Sys. 1998, 42(1-2):209-220.
64. Brown P.J., Fearn T., Vannucci M. Bayesian Wavelet Regression on Curves with Application to a Spectroscopic Calibration Problem. J. Am. Stat. Assoc. 2001, 96:398-408.
65. Sjoblom J., Svensson O., Josefson M., Kullberg H., Wold S. An Evaluation of Orthogonal Signal Correction Applied to Calibration Transfer of Near Infrared Spectra. Chemom. Intell. Lab. Sys. 1998, 44:229-244.
66. Westerhuis J.A., de Jong S., Smilde A.K. Direct Orthogonal Signal Correction. Chemom. Intell. Lab. Sys. 2001, 56:13-25.
67. Wold S., Antti H., Lindgren F., Ohman J. Orthogonal Signal Correction of Near-Infrared Spectra. Chemom. Intell. Lab. Sys. 1998, 44(1-2):175-185.
68. PLS Toolbox, Eigenvector Research, Inc. Wenatchee, WA. http://software.eigenvector.com/toolbox/pls_toolbox/.
69. Trygg J., Wold S. Orthogonal Projections to Latent Structures (O-PLS). J. Chemom. 2002, 16:119-128.
70. Goicoechea H.C., Olivieri A.C. A Comparison of Orthogonal Signal Correction and Net Analyte Preprocessing Methods. Theoretical and Experimental Study. Chemom. Intell. Lab. Sys. 2001, 56:73-81.
71. Svensson O., Kourti T., MacGregor J.F. An Investigation of Orthogonal Signal Correction Algorithms and Their Characteristics. J. Chemom. 2002, 16:176-188.
72. Azzouz T., Puigdomenech A., Aragay M., Tauler R. Comparison Between Different Data Pre-Treatment Methods in the Analysis of Forage Samples Using Near-Infrared Diffuse Reflectance Spectroscopy and Partial Least-Squares Multivariate Calibration Method. Anal. Chim. Acta 2003, 484(1):121-134.
73. Yee N.G., Coghill G.G. Factor Selection Strategies for Orthogonal Signal Correction Applied to Calibration of Near-Infrared Spectra. Chemom. Intell. Lab. Sys. 2003, 67:145-156.
74. Shen Q., Jiang J.-H., Shen G.-l., Yu R.-Q. Ridge Estimated Orthogonal Signal Correction for Data Preprocessing Prior to PLS Modeling: QSAR Studies of Cyclooxygenase-2 Inhibitors. Chemom. Intell. Lab. Sys. 2006, 82:44-49.
75. Golub G.H., Heath M., Wahba G. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics 1979, 21(2):215-223.
76. Hawkins D.M., Basak S.C., Shi X. QSAR with Few Compounds and Many Features. J. Chem. Inf. Comput. Sci. 2001, 41:663-670.
77. Luypaert J., Heuerding S., Massart D.L., Vander Heyden Y. Direct Orthogonal Signal Correction as Data Pretreatment in the Classification of Clinical Lots of Creams from Near Infrared Spectroscopy Data. Anal. Chim. Acta 2007, 582(1):181-189.
78. Lorber A., Faber N.M., Kowalski B.R. Net Analyte Signal Calculation in Multivariate Calibration. Anal. Chem. 1997, 69:1620-1626.
79. Park K.S., Jun C.H. Hybrid Linear Analysis Based on the Net Analyte Signal in Spectral Responses with Orthogonal Signal Correction. Near Infrared Anal. 2000, 1:1-8.
80. Morgan D.R. Spectral Absorption Pattern Detection and Estimation. II. System Applications and Design Procedures. Appl. Spectrosc. 1977, 31:404-415.
81. Faber N.M., Ferré J., Boqué R., Kalivas J.H. Quantifying Selectivity in Spectrophotometric Multicomponent Analysis. Trends Anal. Chem. 2003, 22:352-361.
82. Lorber A. Error Propagation and Figures of Merit for Quantification by Solving Matrix Equations. Anal. Chem. 1986, 58:1167-1172.
83. Lorber A., Harel A., Goldbart Z., Brenner I.B. Curve Resolution and Figures of Merit Estimation for Determination of Trace Elements in Geological Materials by Inductively Coupled Plasma Atomic-Emission Spectrometry. Anal. Chem. 1987, 59:1260-1266.
84. Andersson C.A. Direct Orthogonalization. Chemom. Intell. Lab. Sys. 1999, 47:51-63.
85. Fearn T. On Orthogonal Signal Correction. Chemom. Intell. Lab. Sys. 2000, 50:47-52.
86. Feudale R.N., Tan H., Brown S.D. Improved Piecewise Orthogonal Signal Correction Algorithm. Appl. Spectrosc. 2003, 57:1201-1206.
87. Rodriguez D., Boschetti C.E., Olivieri A.C. Near-Infrared Spectroscopic Determination of Antioxidants and Organic Acids in Rubbers Assisted by a New Multivariate Calibration Method Based on Direct Orthogonalization. Analyst 2002, 127:304-309.
88. Yoshio T., Michael A.H. Constrained Principal Component Analysis: A Comprehensive Theory. Appl. Algebra Eng. Commun. Comput. 2001, 12:391-419.
89. Yoshio T., Michael A.H. principal Component Analysis with External Information on Both Subjects and Variables. Psychometrika 1991, 56(1):97-120.
90. Hansen P.W. Pre-Processing Method Minimizing the Need for Reference Analyses. J. Chemom. 2001, 2(15):123-131.
91. Roger J.-M., Chauchard F., Bellon-Maurel V. EPO-PLS External Parameter Orthogonalisation of PLS Application to Temperature-Independent Measurement of Sugar Content of Intact Fruits. Chemom. Intell. Lab. Sys. 2003, 66:191-204.
92. Zeaiter M., Roger J.M., Bellon-Maurel V. Dynamic Orthogonal Projection. A New Method to Maintain the On-Line Robustness of Multivariate Calibrations. Application to NIR-Based Monitoring of Wine Fermentations. Chemom. Intell. Lab. Sys. 2006, 80:227-235.
93. Bosq, D.; Lecoutre, J. P. Theorie de l'estimation fonctionnelle. In Collection " Économie et Statistiques Avancées"; Economica, Paris, 1987.
94. Scott D.W. Multivariate Density Estimation: Theory, Practice, and Visualization 1992, John Wiley & Sons, New York, Chichester.
95. Vieu P. Non Parametric Regression: Optimal Local Bandwidth Choice. J. R. Stat. Soc. B 1991, 53:453-464.
96. Roger J., Chauchard F., Williams P. Removing the Block Effects in Calibration by Means of Dynamic Orthogonal Projection. Application to the Year Effect Correction for Wheat Protein Prediction. 2008, 16(3):311-315.
97. Dabros M., Amrhein M., Gujral P., von Stockar U. On-Line Recalibration of Spectral Measurements Using Metabolite Injections and Dynamic Orthogonal Projection. Appl. Spectrosc. 2007, 61:507-513.