Automated Bayesian quality control of streaming rain gauge data

Environmental Modelling and Software

Volumn 40, Issue , 2013, Pages 289-301

  Hill, David J ^a  

^a THOMPSON RIVERS UNIVERSITY   (Canada)

Author keywords

Automated quality control; Bayesian modeling; Data fusion; Environmental sensing; Radar rainfall; Real time modeling

Indexed keywords

BAYESIAN MODELING; DATA ERRORS; DYNAMIC BAYESIAN NETWORKS; ENVIRONMENTAL SENSING; ERROR DETECTORS; EXPECTATION MAXIMIZATION; FALSE ALARM RATE; FALSE NEGATIVE RATE; MOVING WINDOW; RADAR-RAINFALL; RAIN GAUGE DATA; RAIN GAUGES; RAINFALL MEASUREMENTS; REAL TIME MODELING; REAL-TIME APPLICATION; RULE-BASED CLASSIFIER; STATISTICAL LEARNING METHODS; TEMPORAL COVERAGE; TIPPING BUCKET RAIN GAUGE; WEATHER RADAR DATA;

DATA COMMUNICATION SYSTEMS; DATA FUSION; GAGES; MEASUREMENT ERRORS; METEOROLOGICAL RADAR; QUALITY CONTROL; RAIN; RAIN GAGES; SENSOR NETWORKS; UNCERTAINTY ANALYSIS;

RADAR MEASUREMENT;

ACCURACY ASSESSMENT; BAYESIAN ANALYSIS; DATA ASSIMILATION; ENVIRONMENTAL MODELING; ERROR ANALYSIS; MEASUREMENT METHOD; QUALITY CONTROL; RADAR; RAINFALL; RAINGAUGE; REAL TIME; STREAM; TELEMETRY; UNCERTAINTY ANALYSIS;

EID: 84871753633     PISSN: 13648152     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.envsoft.2012.10.006     Document Type: Article

References (71)

1. Alameddine I., Cha Y.-K., Reckhow K.H. An evaluation of automated structure learning with Bayesian networks: an application to estuarine chlorophyll dynamics. Environmental Modelling and Software 2011, 26:163-172.
2. Amenu G., Markus M., Kumar P., Demissie M. Hydrologic applications of Minimal Resource Allocations Network (MRAN). Journal of Hydrologic Engineering 2007, 12(1):124-129.
3. Arulampalam M.S., Maskell S., Gordon N., Clapp T. A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Transactions on Signal Processing 2002, 50(2):174-188.
4. Austin P.M. Relation between measured radar reflectivity and surface rainfall. Monthly Weather Review 1987, 115:1053-1070.
5. Baeck M.L., Smith J.A. Rainfall estimation by the WSR-88D for heavy rainfall events. Weather and Forecasting 1998, 13:416-436.
6. Battan L.J. Vertical air motions and the Z-R relation. Journal of Meteorology 1976, 15:1120-1121.
7. Berne A., Delrieu G., Creutin J.D., Obled C. Temporal and spatial resolution of rainfall measurements required for urban hydrology. Journal of Hydrology 2004, 299(3-4):166-179.
8. Casella G., Robert C.P. Rao-Blackwellisation of sampling schemes. Biometrika 1996, 83(1):81-94.
9. Ciach G.J. Local random errors in tippin-bucket rain gauge measurements. Journal of Atmospheric and Oceanic Technology 2003, 20:752-759.
10. Ciach G.J., Krajewski W.F. On the estimation of radar rainfall error variance. Advances in Water Resources 1999, 22:585-595.
11. Crum T.D., Alberty R.L. The WSR-88D and the WSR-88D operational support facility. Bulletin of the American Meteorological Society 1993, 74:1669-1687.
12. Crum T.D., Saffle R.E., Wilson J.W. An update on the NEXRAD program and future WSR-88D support to operations. Weather and Forecasting 1998, 13:253-262.
13. Dereszynski E.W., Dietterich T.G. Probabilistic models for anomaly detection in remote sensor data streams. Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence (UAI-2007) 2007, 75-82.
14. Doucet A., de Freitas N., Murphy K., Russell S. Rao-Blackwellised particle filtering for dynamic Bayesian networks. Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (UAI2000) 2000, 176-183.
15. Doucet A., Godsill S., Andrieu C. On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing 2000, 10(3):197-208.
16. Fankhauser R. Influence o fsystematic errors from tipping bucket gauges on urban runoff simulation. Use of historical rainfall series for hydrological modeling. Proceedings of the 3rd International Workshop on Rainfall in Urban Areas 1997, 45-52. R. Fankhauser, T. Einfalt, K. Arnbjerg-Nielsen (Eds.).
17. Frühwirth R. Track fitting with long-tailed noise: a Bayesian approach. Computer Physics Communications 1995, 85:189-199.
18. Fulton R., Breidenback J.P., Seo D.J., Miller D.A., O'Bannon T. The WSR-88D rainfall algorithm. Weather and Forecasting 1998, 13(2):377-398.
19. Glaudemans M., Tilles P., Lawrence B. Interactive quality control and operational product generation of hourly multi-sensor precipitation estimates in the NWS. Proceedings of the 23rd Conference on Hydrology 2008, American Meteorological Society, New Orleans, 5B.3.
20. Grimes D.I.F., Pardo-Igúzquiza E., Bonifacio R. Optimal areal rainfall estimation using raingauges and satellite data. Journal of Hydrology 1999, 222:93-108.
21. Habib E., Krajewski W.F., Kruger A. Sampling errors of tipping-bucket rain gauge measurements. Journal of Hydrologic Engineering 2001, 6(2):159-166.
22. Hill D.J., Liu Y., Marini L., Kooper R., Rodriguez A., Minsker B.S., Myers J., McLaren T. A virtual sensor system for creating real-time customized environmental data products. Environmental Modelling and Software 2011, 10.1016/j.envsoft.2011.09.001.
23. Hill D.J., Minsker B.S. Anomaly detection in streaming environmental data: a data-driven modeling approach. Environmental Modelling and Software 2010, 25(9):1014-1022. 10.1016/j.envsoft.2009.08.010.
24. Hill D.J., Minsker B.S., Amir E. Real-time Bayesian anomaly detection in streaming environmental data. Water Resources Research 2009, 45:W00D28. 10.1029/2008WR006956.
25. Humphrey M.D., Istok J.D., Lee J.Y., Hevesi J.A., Flint A.L. A new method for automated dynamic calibration of tipping bucket raingauges. Journal of Atmospheric and Oceanic Technology 1997, 14:1513-1519.
26. Kalman R.E. A new approach to linear filtering and prediction problems. Transactions of the ASME - Journal of Basic Engineering 1960, 82:35-45.
27. Klazura G.E., Imy D.A. A description of the initial set of analysis products available from the NEXRAD WSR-88D system. Bulletin of the American Meteorological Society 1993, 74:1293-1311.
28. Koller D., Lerner U. Sampling in factored dynamic systems. Sequential Monte Carlo Methods in Practice 2000, Springer-Verlag, New York. A. Doucet, J.F.G. de Freitas, N. Gordon (Eds.).
29. Krajewski W.F., Kruger A., Nešpor V. Experimental and numerical studies of small-scale rainfall measurements and variability. Water Science and Technology 1998, 14:1513-1519.
30. Krajewski W.F., Smith J.A. Radar hydrology: rainfall estimation. Advances in Water Resources 2002, 25(8 - 12):1387-1394.
31. Krajewski W.F., Villarini G., Smith J.A. Radar rainfall uncertainties: where are we after thirty years of effort?. Bulletin of the American Meteorological Society 2010, 91:87-94.
32. Legates D.R., Willmott C.J. Mean seasonal and spatial variability in gauge-corrected, global precipitation. International Journal of Climatology 1990, 10:111-127.
33. Lerner U., Parr R. Inference in hybrid networks: theoretical limits and practical algorithms. Proceedings of the 17th Annual Conference on Uncertainty in Artificial Intelligence (UAI-01) Seattle, Washington, August 2001 2001, 310-318.
34. Marshall J.S., Palmer W.M. The distribution of raindrops with size. Journal of Meteorology 1948, 5:15-166.
35. Meinhold R.J., Singpurwalla N.D. Robustification of Kalman filter models. Journal of the American Statistical Association 1989, 84(406):479-486.
36. Molini A., Lanza L.G., La Barbera P. The impact of tipping-bucket raingauge measurement errors on design rainfall for urban-scale applications. Hydrological Processes 2005, 19:1073-1088.
37. Morin E., Goodrich D.C., Maddox R.A., Gao X., Gupta H.V., Sorooshian S. Spatial patterns in thunderstorm rainfall events and their coupling with watershed hydrological response. Advances in Water Resources 2006, 29:843-860.
38. Morin E., Krajewski W.F., Goodrich D.C., Gao X., Sorooshian S. Estimating rainfall intensities from weather radar data: the scale-dependency problem. Journal of Hydrology 2003, 4:1105-1118.
39. Morin E., Maddox R.A., Goodrich D., Sorooshian S. Radar Z-R relationship for summer monsoon storms in Arizona. Weather and Forecasting 2005, 20:627-679.
40. Neal R.M., Hinton G.E. A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models 1998, 355-368. Kluwer Academic Publishers, Dordrecht. M.I. Jordan (Ed.).
41. Nešpor V., Sevruk B. Estimation of wind-induced error of rainfall gauge measurements using a numerical simulation. Journal of Atmospheric and Oceanic Technology 1999, 16:450-464.
42. Niemczynowicz J. The dynamic calibration of tipping bucket raingauges. Nordic Hydrology 1986, 17:203-214.
43. NRC (National Research Council) Confronting the Nation's Water Problems: the Role of Research 2004, National Academies Press, Washington, D.C.
44. NRC (National Research Council) Informing Decisions in a Changing Climate 2009, National Academies Press, Washington, D.C.
45. NSF (National Science Foundation), 2005. Sensors for Environmental Observatories Report of the NSF Sponsored Workshop December 2004. NSF, Arlington, VA.
46. Puig V., Quevedo J., Figueras J., Ferrier S., Cembrano G., Salamero M., Wilhelmi G. Fault diagnosis of rain gauges of Barcelona's urban Sewer system using interval models. Proceedings of the Advances in Artificial Intelligence - IBERAMIA 2002 2002, 1-10. Springer.
47. Russell S., Norvig P. Artificial Intelligence: a Modern Approach 2003, Prentice-Hall Inc, New Jersey.
48. Schweppe F.C. Uncertain Dynamic Systems 1973, Prentice-Hall Inc, New Jersey.
49. Seo D.-J., Smith J.A. Rainfall estimation using raingages and radar - a Bayesian approach: 1. Derivation of estimators. Stochastic Hydrology and Hydraulics 1991, 5(1):17-21.
50. Seo D.-J., Breidenback J. Real-time correction of spatially nonuniform bias in radar rainfall data using rain gauge measurements. Journal of Hydrometeorology 2002, 3:93-111.
51. Seo D.-J., Breidenbach J.P., Johnson E.R. Real-time estimation of mean field bias in radar rainfall data. Journal of Hydrology 1999, 223:131-147.
52. Sevruk B. Methods of Correcting for Systematic Error in Point Precipitation Measurements for Operational Use. Operational Hydrology Report, No. 21 1982, World Meteorological Organization.
53. Schilling W. Rainfall data for urban hydrology: what do we need?. Atmospheric Research 1991, 27(1-3):5-21.
54. Shumway R.H., Stoffer D.S. An approach to time series smoothing and forecasting using the EM algorithm. Journal of Time Series Analysis 1982, 3(4):253-264.
55. Smith J.A., Krajewski W.F. Estimation of the mean field bias of radar rainfall estimates. Journal of Applied Meteorology 1991, 30:397-412.
56. Smith J.A., Krajewski W.F. A modeling study of rainfall rate-reflectivity relationships. Water Resources Research 1993, 29:2505-2514.
57. Spall J.C. An overview of key developments in dynamic modeling and estimation. Bayesian Analysis of Time Series and Dynamic Models 1988, Marcel Dekker, Inc, New York. J.C. Spall (Ed.).
58. Steiner M., Smith J.A. Convective versus stratiform rainfall: an ice-mircophysical and kinematic conceptual model. Atmospheric Research 1998, 47-48:317-326.
59. Steiner M., Smith J.A., Burges S.J., Alonso C.V., Darden R.W. Effect of bias adjustment and rain gauge data quality control on radar rainfall estimation. Water Resources Research 1999, 36(8):2487-2503.
60. Sun X., Mein R.G., Keenan T.D., Elliott J.F. Flood estimation using radar and raingauge data. Journal of Hydrology 2000, 239:4-18.
61. Upton G.J.G., Rahimi A.R. On-line detection of errors in tipping-bucket raingauges. Journal of Hydrology 2003, 278:197-212.
62. van der Merwe, R., de Freitas, J.F.G., Doucet, A., Wan, E.A., 2000. The Unscented Particle Filter (Tech. Rep.). University of Cambridge, Department of Engineering.
63. Vasiloff S.V., Howard K.W., Zhang J. Difficulties with correcting radar rainfall estimates based on rain gauge data: a case study of severe weather in Montana on 16-17 June 2007. Weather and Forecasting 2009, 24:1334-1344.
64. Villarini G., Smith J.A., Baeck M.L., Sturdevant-Rees P., Krajewski W.F. Radar analysis of extreme rainfall and flooding in urban drainage basins. Journal of Hydrology 2010, 381:266-286.
65. Villarini G., Mandapaka P.V., Krajewski W.F. Rainfall and sampling errors: a rain gauge perspective. Journal of Geophysical Research 2008, 113:D11102.
66. Villarini G., Krajewski W.F. Empirically-based modeling of spatial sampling uncertainties associated with rainfall measurements by rain gauges. Advances in Water Resources 2008, 31:1015-1023.
67. Villarini G., Krajewski W.F. Sensitivity studies of the model s of radar-rainfall uncertainties. Journal of Applied Meteorology and Climatology 2010, 49(2):288-309.
68. Wilson J.W. Integration of radar and raingage data for improved rainfall measurement. Journal of Applied Meteorology 1970, 9:489-497.
69. Xiaoyang L., Jietai M., Yuanjing Z., Jiren L. Runoff simulation using radar and raingauge data. Advances in Atmospheric Sciences 2003, 20(2):213-218.
70. Yilmaz K., Hogue T.S., Jsu K.-L., Sorooshian S., Gupta H., Wagener T. Intercomparison of raingauge, radar, and satellite-based precipitation estimates with emphasis on hydrologic forecasting. Journal of Hydrometeorology 2005, 6(4):497-517.
71. Zhang J., Howard K., Langston C., Vasiloff S., Kaney B., Arthur A., Van Cooten S., Kelleher K.I., Kitzmiller D., Ding F., Seo D.-J., Wells E., Dempsey C. National mosaic and multi-sensor QPE (NMQ) system. Bulletin of the American Meteorological Society October 2011, 1321-1338.