Metrics for evaluating performance of prognostic techniques

2008 International Conference on Prognostics and Health Management, PHM 2008

Volumn , Issue , 2008, Pages

  Saxena, Abhinav ^a   Celaya, Jose ^a   Balaban, Edward ^a   Goebel, Kai ^a   Saha, Bhaskar ^a   Saha, Sankalita ^a   Schwabacher, Mark ^a  

^a NASA AMES RESEARCH CENTER   (United States)

Author keywords

Performance metrics; Prognostics and health management

Indexed keywords

APPLICATIONS; AUTOMOBILE ELECTRONIC EQUIPMENT; MAINTENANCE;

COMMON GROUND; DOMAIN DYNAMICS; HEALTH MAINTENANCE; METRICS (CO); PERFORMANCE EVALUATION (PE); PERFORMANCE METRICS; PROGNOSTICS AND HEALTH MANAGEMENT; REAL APPLICATIONS; REMAINING USEFUL LIFE (RUL); RESEARCH COMMUNITIES; TIME SCALING; USER REQUIREMENTS;

HEALTH;

EID: 58449128705     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/PHM.2008.4711436     Document Type: Conference Paper

References (54)

1. G. Vachtsevanos, F. L. Lewis, M. Roemer, A. Hess, and B. Wu, Intelligent Fault Diagnosis and Prognosis for Engineering Systems, 1st ed. Hoboken, New Jersey: John Wiley & Sons, Inc., 2006.
2. I. Jolliffe, T. and D. Stephenson, B., Forecast Verification: A Practitioner's Guide in Atmospheric Science, 1st ed. West Sussex, UK: John Wiley & Sons Ltd., 2003.
3. G. W. Brier and R. A. Allen, "Verification of Weather Forecasts," in Compendium of Meteorology, T. F. Malone, Ed. Boston: American Meteorological Society, pp. 841-848.
4. M. Schwabacher, "A Survey of Data Driven Prognostics," in AIAA Infotech@Aerospace Conference, 2005.
5. M. Schwabacher and K. Goebel, "A Survey of Artificial Intelligence for Prognostics," in AAAI Fall Symposium, Arlington, VA, 2007.
6. ISO, "Condition Monitoring and Diagnostics of Machines - Prognostics part 1: General Guidelines," in ISOI3381-1:2004(E). vol. ISO/IEC Directives Part 2, I. O. f. S. (ISO), Ed.: ISO, 2004, p. 14.
7. B. L. Ferrell, "JSF Prognostics and Health Management," in IEEE Aerospace Conference, 1999, p. 471.
8. D. L. Goodman, S. Wood, and A. Turner, "Return-on-Investment (ROI) for Electronic Prognostics in Mil/Aero Systems," in IEEE Autotestcon Orlando, FL, 2005.
9. S. Vohnout, "Electronic Prognostics System Implementation on Power Actuator," in IEEE Aerospace Conference Big Sky, Montana, 2008.
10. S. Wood and D. Goodman, "Return-on-Investment (ROI) for Electronic Prognostics in High Reliability Telecom Applications," in Annual International Telecommunications Energy Conference, 2006.
11. J. E. Dzakowic and G. S. Valentine, "Advanced Techniques For The Verification And Validation Of Prognostics & Health Management Capabilities," in Machinery Failure Prevention Technologies (MFPT 60), Virginia Beach, VA, 2007.
12. D. W. Brown, P. W. Kalgren, C. S. Byington, and R. F. Orsagh, "Electronic Prognostics - A Case Study Using Global Positioning System (GPS)," in IEEE Autotestcon Orlando, FL, 2005.
13. J. Gu, D. Barker, and M. Pecht, "Prognostics Implementation of Electronics Under Vibration Loading," Microelectronics Reliability, vol. 47, 2007.
14. J. P. Hofmeister, P. L. T. Walter, D. Goodman, E. G. Ortiz, M. G. P. Adams, and T. A. Tracy, "Ball Grid Array (BGA) Solder Joint Intermittency: Detection: SJ BIST," in IEEE Aerospace Conference Big Sky, Montana, 2008.
15. C. Wilkinson, D. Humphrey, B. Vermeire, and J. Houston, "Prognostic and Health Management for Avionics," in IEEE Aerospace Conference, 2004.
16. C. Teal and B. Larsen, "Technology Update II: Wire Systems Diagnostics & Prognostics," in Digital Avionics Systems Conference, 2003.
17. N. Vichare and M. Pecht, "Enabling Electronic Prognostics Using Thermal Data," in 12th International Workshop on Thermal investigations of ICs, 2006.
18. N. Vichare, P. Rodgers, V. Eveloy, and M. Pecht, "Environment and Usage Monitoring of Electronic Products for Health Assessment and Product Design," Quality Technology & Quantitative Management, vol. 4, pp. 235-250, 2007.
19. J. W. Simons and D. A. Shoekey, "Prognostics Modeling of Solder Joints in Electronic Components," in IEEE Aerospace Conference, 2006.
20. M. Pepe, The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford, UK: Oxford University Press, 2003.
21. F. E. Harrell Jr., K. L. Lee, and D. B. Mark, "Tutorial in Biostatistics: Multivariate Prognostics Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors," Statistics in Medicine, vol. 15, pp. 361-387, 1996.
22. I. Zelic, N. Lavrac, P. Najdenov, and Z. Rener-Prime, "Working Notes, Workshop on Prognostic Models in Medicine: Artificial Intelligence and Decision Analytic Approaches," in Joint European Conference on Artificial Intelligence in Medicine and Medical Decision Making, AIMDM'99, Aalborg, Denmark, 1999.
23. A. Abu-Hanna and P. J. F. Lucas, "Prognostics Models in Medicine: AI and Statistical Approaches," Methods of Information in Medicine, vol. 40, pp. 1-5, 2001.
24. E. L. Kaplan and P. Meier, "Nonparametric Estimation from Incomplete Observations," Journal of the American Statistical Association, vol. 53, pp. 457-481, 1958.
25. L. J. Bond, S. R. Doctor, D. B. Jarrell, and J. W. D. Bond, "Improved Economics of Nuclear Plant Life Management," in Second International Symposium on Nuclear Power Plant Life Management Shanghai, China: IAEA, 2007, p. 26.
26. D. B. Jarrell, "Completing the Last Step in O&M Cost reduction," Pacific Northwest National Laboratory, p. 8.
27. R. Carbone and J. Armstrong, "Note. Evaluation of Extrapolative Forecasting Methods: Results of a Survey of Academicians and Practitioners," Journal of Forecasting, vol. 1, pp. 215-217, 1982.
28. R. J. Hyndman and A. B. Koehler, "Another Look at Measures of Forecast Accuracy," International Journal of Forecasting, vol. 22, pp. 679-688, 2006.
29. S. Makridakis, A. Andersen, R. Carbone, R. Fildes, M. Hibon, R. Lewandowski, J. Newton, E. Parzen, and R. Winkler, "The Accuracy of Extrapolation (Time Series) Methods: Results of a Forecasting Competition," Journal of Forecasting, vol. 1, pp. 111-153, 1982.
30. S. Makridakis and M. Hibon, "The M3-Competition: Results, Conclusions, and Implications," International Journal of Forecasting, vol. 16, pp. 451-476, 2000.
31. F. X. Diebold and R. Mariano, "Comparing Predictive Accuracy," Journal of Business and Economic Statistics, vol. 13, pp. 253 - 265, 1995.
32. H. Theil, Applied Economic Forecasting. Chicago, IL: Rand McNally and Company, 1966.
33. W. Marquering and M. Verbeek, "A Multivariate Nonparametric Test for Return and Volatility Timing," Finance Research Letters, vol. 4, pp. 250 - 260, 2004.
34. B. Ebert, et.al., "Forecast Verification - Issues, Methods and FAQ." Last accessed. 2007, url: http://www.bom.gov.au/bmrc/wefor/staff/eee/ verif/verifwebpage.html.
35. D. W. Wichern, Flores, B. E., "Evaluating forecasts: a look at aggregate bias and accuracy measures," Journal of Forecasting, vol. 24, pp. 433-451, 2005.
36. T. N. Palmer, A. Alessandri, U. Andersen, P. Cantelaube, M. Davey, P. De le cluse, M. De qu, E. Diez, F. J. Doblas-Reyes, H. Feddersen, R. Graham, S. Gualdi, J. F. Gueremy, R. Hagedorn, M. Hoshen, N. Keenlyside, M. Latif, A. Lazar, E. Maisonnave, V. Marletto, A. P. Morse, B. Orfila, P. Rogel, J. M. Terres, and M. C. Thomson, "Development of A European Multimodel System for Seasonal-to-Interannual Prediction (Demeter)," Bulletin of the American Meteorological Society, vol. 85, pp. 853-872, June 01, 2004 2004.
37. G. Giebel, R. Brownsword, and G. Kariniotakis, "The State-of-The-Art in Short-Term Prediction of Wind Power - A Literature Review," Riso National Laboratory, Rosklide, Denmark 2003.
38. H. Madsen, G. Kariniotakis, H. A. Nielsen, T. S. Nielsen, and P. Pinson, "A Protocol for Standardizing the Performance Evaluation of Short-Term Wind Power Prediction Models," Technical University of Denmark, IMM, Lyngby, Denmark, Deliverable ENK5-CT-2002-00665, 2004.
39. O. Gusikhin, N. Rychtyckyj, and D. Filev, "Intelligent Systems in the Automotive Industry: Applications and Trends.," Knowledge and Information Systems, vol. 12, pp. 147-168, 2007.
40. F. L. Greitzer and R. A. Pawlowski, "Embedded Prognostics Health Monitoring," in Proceedings of the International Instrumentation Symposium Embedded Health Monitoring Workshop, United States, 2002, p. Size: vp.
41. L. Jianhui, M. Namburu, K. Pattipati, A. L. Q. Liu Qiao, M. A. K. M. Kawamoto, and S. A. C. S. Chigusa, "Model-Based Prognostic Techniques [Maintenance Applications]," in AUTOTESTCON 2003. IEEE Systems Readiness Technology Conference. Proceedings, 2003, pp. 330-340.
42. D. Djurdjanovic, J. Lee, and J. Ni, "Watchdog Agent-an Infotronics-Based Prognostics Approach for Product Performance Degradation Assessment and Prediction," Advanced Engineering Informatics, vol. 17, pp. 109-125, 2003.
43. G. Vachtsevanos and P. Wang, "Fault Prognosis Using Dynamic Wavelet Neural Networks," in AUTOTESTCON Proceedings, 2001. IEEE Systems Readiness Technology Conference, 2001, pp. 857-870.
44. K. Goebel and P. Bonissone, "Prognostic Information Fusion for Constant Load Systems," in Proceedings of the 7th Annual Conference on Information Fusion. vol. 2, 2005, pp. 1247 - 1255.
45. D. C. Hoaglin, F. Mosteller, and J. W. Tukey, "Understanding Robust and Exploratory Data Analysis," in Probability and Mathematical Statistics: John Wiley & Sons, 1983.
46. C. S. Byington, Roemer, M.J., Kalgren, P.W., "Verification and Validation of Diagnostic/Prognostic Algorithms," in Machinery Failure Prevention Technology Conference (MFPT 59) Virginia Beach, VA, 2005.
47. G. Vachtsevanos, "Performance Metrics for Fault Prognosis of Complex Systems," in AUTOTESTCON 2003. IEEE Systems Readiness Technology Conference., 2003, pp. 341 - 345.
48. A. V. Aho, J. D. Ullman, and J. E. Hopcroft, Data Structures and Algorithms: Addison Wesley, 1983.
49. "Electronics Prognostics R&D Needs Definition," in Electronics Prognostics Workshop II Miami, FL: Electronics Prognostics Technology Task Group, Defence Technical Information Center (DTIC) 2006.
50. P. A. Thompson, "A Statistician in Search of a Population," International Journal of Forecasting, vol. 8, pp. 103-104, 1992.
51. J. S. Armstrong and F. Collopy, "Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons," International Journal of Forecasting, vol. 8, pp. 69-80, 1992.
52. C. Chatfield, "A Commentary on Error Measures," International Journal of Forecasting, vol. 8, pp. 100-102, 1992.
53. D. A. Ahlburg, "Error Measures and the Choice of a Forecast Method," International Journal of Forecasting, vol. 8, pp. 99-100, 1992.
54. R. L. Winkler and A. H. Murphy, "On Seeking a Best Performance Measure or a Best Forecasting Method," International Journal of Forecasting, vol. 8, pp. 104-107, 1992.