Ridge regression optimization using a harmonious approach

Journal of Chemometrics

Volumn 18, Issue 7-8, 2004, Pages 372-384

  Forrester, J B ^a   Kalivas, J H ^a  

^a IDAHO STATE UNIVERSITY   (United States)

Author keywords

Bias variance tradeoff; Degrees of freedom; Effective rank; Harmonious model; L curve; Multivariate calibration; Parsimony; Ridge regression

Indexed keywords

CURVE FITTING; DEGREES OF FREEDOM (MECHANICS);

BIAS-VARIANCE TRADEOFFS; DEGREE OF FREEDOM; EFFECTIVE RANK; HARMONIOUS MODELS; L-CURVE; MULTIVARIATE CALIBRATION; OPTIMISATIONS; PARSIMONY; REGRESSION MODELLING; RIDGE REGRESSION;

REGRESSION ANALYSIS;

ARTICLE; CALCULATION; CALIBRATION; INTERMETHOD COMPARISON; MATHEMATICAL ANALYSIS; MATHEMATICAL MODEL; PARAMETER; PARSIMONY ANALYSIS; RIDGE REGRESSION;

EID: 14944349678     PISSN: 08869383     EISSN: None     Source Type: Journal    
DOI: 10.1002/cem.883     Document Type: Article

References (57)

1. Frank IE, Friedman JH. A statistical view of some chemometrics regression tools. Technometrics 1993; 35: 109-135.
2. Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 1970; 12: 55-67.
3. Kalivas JH. Interrelationships of multivariate regression methods using eigenvector basis sets. J. Chemometrics 1999; 13: 111-132.
4. Kalivas JH. Basis sets for multivariate regression. Anal. Chim. Acta 2001; 428: 31-40.
5. Huang J, Brennan D, Sattler L, Alderman J, Lane B, O'Mathuna C. A comparison of calibration methods based on calibration data size and robustness. Chemometrics Intell. Lab. Syst. 2002; 62: 25-35.
6. Gusanto A, Pawitan Y, Huang J, Lane B. Variable selection in random calibration of near-infrared instruments: ridge regression and partial least squares regression settings. J. Chemometrics 2003; 17: 174-185.
7. Naes T, Isakson T, Fearn T, Davies T. A User-friendly Guide to Multivariate Calibration and Classification. NIR Publications: Chichester, 2002.
8. Golub GH, Heath M, Wahba G. Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 1979; 21: 215-223.
9. Green RL, Kalivas JH. Graphical diagnostics for regression model determinations with consideration of the bias/variance trade-off. Chemometrics Intell. Lab. Syst. 2002; 60: 173-188.
10. Baumann K. Cross-validation as the objective function for variable-selection techniques. Trends Anal. Chem. 2003; 22: 395-406.
11. Shao J. Linear model selection by cross-validation. J. Am. Statist. Assoc. 1993; 88: 486-494.
12. Hansen PC. Analysis of discrete ill-posed problems by means of the L-curve. SIAM Rev. 1992; 34: 561-580.
13. Hansen PC, O'Leary DP. The use of the L-curve in the regularization of discrete ill-posed problems. SIAM J. Sci. Comput. 1993; 14: 1487-1503.
14. Wahba G. Spline Models for Observational Data, vol. 59 of CBMS-NSF Regional Conference Series in Applied Mathematics. SIAM: Philadelphia, PA, 1990.
15. Hoerl AE, Kennard RW. Ridge regression: applications to nonorthogonal problems. Technometrics 1970; 12: 69-82.
16. Hoerl AE, Kennard RW, Baldwin KF. Ridge regression: some simulations. Commun. Statist. 1975; 4: 105-123.
17. Hansen PC. Rank-deficient and Discrete Ill-posed Problems: Numerical Aspects of Linear Inversion. SIAM: Philadelphia, PA, 1998.
18. Kalivas JH, Green RL. Pareto optimal multivariate calibration for spectroscopic data. Appl. Spectrosc. 2001; 55: 1645-1652.
19. Faber K, Kowalski BR. Propagation of measurement errors for the validation of predictions obtained by principal components regression and partial least squares. J. Chemometrics 1997; 11:181-238.
20. Faber K, Kowalski BR. Prediction error in least squares regression: further critique on the derivation used in The Unscrambler. Chemometrics Intell. Lab. Syst. 1996; 34: 283-292.
21. Olivieri AC. A simple approach to uncertainty propagation in preprocessed multivariate calibration. J. Chemometrics 2002; 16: 207-217.
22. Faber NM, Song XH, Hopke PK. Sample-specific standard error of prediction for partial least squares regression. Trends Anal. Chem. 2003; 22: 330-334.
23. Fernández Pierna JA, Jin L, Wahl F, Faber NM, Massart DL. Estimation of partial least squares regression prediction uncertainty when the reference values carry a sizable measurement error. Chemometrics Intell. Lab. Syst. 2003; 65: 281-291.
24. Lorber A, Kowalski BR. Estimation of prediction error for multivariate calibration. J. Chemometrics 1988; 2: 93-109.
25. Faber NM. Uncertainty estimation for multivariate regression coefficients. Chemometrics Intell. Lab. Syst. 2002; 64: 169-179.
26. Kalivas JH, Lang PM. Mathematical Analysis of Spectral Orthogonality. Marcel Dekker: New York, 1994.
27. Steinberg DM, Hunter WG. Experimental design: review and comment. Technometrics 1984; 26: 71-97.
28. Prakash AMC, Stellman CM, Booksh KS. Optical regression: a method for improving quantitative precision of multivariate prediction with single channel spectrometers. Chemometrics Intell. Lab. Syst. 1999; 46: 265-274.
29. Coelho CJ, Galváo RKH, Araujo MCU, Pimentel MF, da Silva EC. A solution to the wavelet transform optimization problem in multicomponent analysis. Chemometrics Intell. Lab. Syst. 2003; 66: 205-217.
30. Höskuldsson A. The H-principle in modeling with applications to Chemometrics. Chemometrics Intell. Lab. Syst. 1992; 14: 139-153.
31. American Society for Testing and Materials. Annual Book of ASTM Standards, Vol. 3.06, E1655, Standard Practices for Infrared Multivariate, Quantitative Analysis. ASTM International: West Conshohocken, PA, 1998.
32. Booksh KS, Wang Z. Extension and application of univariate figures of merit to multivariate calibration. In Chemometrics in Environmental Chemistry-Statistical Methods, Einay J (ed.). Springer: Berlin, 1995; 209-227.
33. Duineveld CAA, Smilde AK, Doornbos DA. Designs for mixture and process variables applied in tablet formulations. Anal. Chim. Acta 1993; 277: 455-465.
34. Kalivas JH. Pareto calibration with built-in wavelength selection. Anal. Chim. Acta 2004; 505: 9-14.
35. Anderson KJ, Kalivas JH. Assessment of Pareto calibration, stability, and wavelength selection. Appl. Spectrosc. 2003; 57: 309-316.
36. Tikhonov AN. Solution of incorrectly formulated problems and the regularization method. Sov. Math. Dokl. 1963; 4: 1035-1038.
37. Hansen PC. The L-curve and its use in the numerical treatment of inverse problems. In Computational Inverse Problems in Electrocardiography, Johnson P (ed.). WIT Press: Southampton, 2001; 119-142.
38. Dox A. On regularized least norm problems. SIAM J. Optim. 1992; 2: 602-618.
39. Censor Y. Pareto optimality in multiobjective problems. Appl. Math. Optim. 1977; 4: 41-59.
40. Cohon JL. Multiobjective Programming and Planning. Academic Press: New York, 1978.
41. Da Cunha NO, Polak E. Constrained minimization under vector-valued criteria in finite dimensional space. J. Math. Anal. Appl. 1967; 19: 103-124.
42. Smilde AK, Knevelman A, Coenegracht PMJ. Introduction of multi-criteria decision making in optimization procedures for high-performance liquid chromatographic separations. J. Chromotogr. 1986; 369: 1-10.
43. Lawson CL, Hanson RJ. Solving Least-squares Problems. Prentice-Hall: Englewood Cliffs, NJ, 1974.
44. Chen LY, Chen JT, Hong HK, Chen CH. Application of Cesàro mean and the L-curve for the deconvolution problem. Soil Dyn. Earthq. Eng. 1995; 14: 361-373.
45. Kaufman L, Neumaier A. PET regularization by envelope guided conjugate gradients. IEEE Trans. Med. Imag. 1996; 15: 385-389.
46. p. Technometrics 1973; 42: 87-94.
47. Myers RH. Classical and Modern Regression with Applications (2nd edn). Duxbury: Pacific Grove, CA, 1990.
48. O'Sullivan F. A statistical perspective on ill-posed inverse problems. Statist. Sci. 1986; 1: 502-527.
49. Ye J. On measuring and correcting the effects of data mining and model selection. J. Am. Statist. Assoc. 1998; 93: 120-131.
50. Golub GH, Heath M, Wahba G. Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 1979; 21: 215-223.
51. Seipel HA, Kalivas JH. Effective rank for multivariate calibration methods. J. Chemometrics 2004; 18: 306-311.
52. Hastie TJ, Tibshirani RJ. Generalized Additive Models. Chapman and Hall: London, 1990.
53. Hastie TJ, Tibshirani RJ, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer: New York, 2001.
54. Wise BM, Gallagher NB, Bro R, Shaver JM. PLS_Toolbox 3.0 for Use with MATLAB. Eigenvector Research: Manson, WA, 2003.
55. Fearn T. A misuse of ridge regression in the calibration of a near infrared reflectance instrument. Appl. Statist. 1983; 32: 73-79.
56. Kalivas JH. Two data sets of near infrared spectra. Chemometrics Intell. Lab. Syst. 1997; 37: 255-259.
57. Geladi P, Swerts J, Lindgren F. Multiwavelength microscopic image analysis of a piece of painted chinaware: classification and regression. Chemometrics Intell. Lab. Syst. 1994; 24: 145-167.