The comparability of bioassessments: A review of conceptual and methodological issues

Journal of the North American Benthological Society

Volumn 30, Issue 3, 2011, Pages 680-701

  Cao, Yong ^a,b   Hawkins, Charles P ^a  

^a UTAH STATE UNIVERSITY   (United States)

^b UNIVERSITY OF ILLINOIS   (United States)

Author keywords

bioassessment; biological indices; calibration; data comparability; data quality; ecological assessments; macroinvertebrates; sampling methods

Indexed keywords

BIOASSAY; BIOINDICATOR; COMPARATIVE STUDY; CONCEPTUAL FRAMEWORK; DATA QUALITY; ECOLOGICAL MODELING; ECOLOGICAL THEORY; LITERATURE REVIEW; MACROINVERTEBRATE; SAMPLING;

EID: 79960249745     PISSN: 08873593     EISSN: None     Source Type: Journal    
DOI: 10.1899/10-067.1     Document Type: Review

References (150)

1. ARMITAGE, P. D., D. MOSS, J. F. WRIGHT, AND M. T. FURSE. 1983. The performance of a new biological water quality score system based on macroinvertebrates over a wide range of unpolluted running-water sites. Water Research 17: 333-347.
2. AROVIITA, J., H. MYKRÄ, T. MUOTKA, AND H. HAMALAINEN. 2009. Influence of geographical extent on typology- and model-based assessments of taxonomic completeness of river macroinvertebrates. Freshwater Biology 54: 1774-1787.
3. ASTIN, L. 2006. Data synthesis and bioindicator development for nontidal streams in the interstate Potomac River Basin, USA. Ecological Indicators 6:664-685. (Pubitemid 44038653)
4. ASTIN, L. 2007. Developing biological indicators from diverse data: the Potomac Basin-wide Index of Benthic Integrity (B-IBI). Ecological Indicators 7:895-908. (Pubitemid 47238657)
5. ASTM (AMERICAN SOCIETY OF TESTING AND MATERIALS). 1995. Biological effects and environmental fate. Volume 11.05. Annual book of standards. American Society of Testing and Materials, Philadelphia, Pennsylvania.
6. BAILEY, R., R. H. NORRIS, AND T. B. REYNOLDSON. 2004. Bioassessment of freshwater ecosystems: using the reference condition approach. Springer, New York.
7. BALTANÁ S, A. 1992. On the use of some methods for the estimation of species richness. Oikos 65:484-492.
8. BARBOUR, M. T., J. GERRITSEN, B. D. SNYDER, AND J. B. STRIBLING. 1999. Rapid bioassessment protocols for use in wadeable streams and rivers: periphyton, benthic macroinvertebrates, and fish. EPA 841-B-99-002. US Environmental Protection Agency, Washington, DC.
9. BIRK, S., AND D. HERING. 2006. Direct comparison of assessment methods using benthic macroinvertebrates: a contribution to the EU Water Framework Directive intercalibration exercise. Hydrobiologia 566:401-415. (Pubitemid 44078680)
10. BLOCKSOM, K. A. 2003. A performance comparison of metric scoring methods for a multimetric index for Mid- Atlantic Highlands stream. Environmental Management 31:670-682. (Pubitemid 36959018)
11. BLOCKSOM, K. A., B. C. AUTREY, M. PASSMORE, AND L. REYNOLDS. 2008. A comparison of single and multiple habitat protocols for collecting macroinvertebrates in wadeable streams. Journal of the American Water Resources Association 44:1-17.
12. BLOCKSOM, K. A., AND J. E. FLOTEMERSCH. 2005. Comparison of macroinvertebrate sampling methods for nonwadeable streams. Environmental Monitoring and Assessment 102:243-262. (Pubitemid 40403483)
13. BRADLEY, D. C., AND S. J. ORMEROD. 2001. Community persistence among stream invertebrates tracks the North Atlantic Oscillation. Journal of Animal Ecology 70:987-996. (Pubitemid 33146348)
14. BREIMAN, L. 2001. Random forests. Machine Learning 45: 5-32.
15. BUFFAGNI, A., S. ERBA, M. CAZZOLA, J. MURRAY-BLIGH, H. SOSZKA, AND P. GENONI. 2006. The STAR common metrics approach to the WFD intercalibration process: full application for small, lowland rivers in three European countries. Hydrobiologia 566:379-399. (Pubitemid 44078679)
16. BUFFAGNI, A., S. ERBA, AND M. T. FURSE. 2007. A simple procedure to harmonize class boundaries of assessment systems at the pan-European scale. Environmental Science and Policy 10:709-924.
17. BUFFAGNI, A., AND M. FURSE. 2006. Intercalibration and comparison-major results and conclusions from the STAR project. Hydrobiologia 566:357-364. (Pubitemid 44078677)
18. CAO, Y., AND C. P. HAWKINS. 2005. Simulating biological impairment for evaluating ecological indicators. Journal of Applied Ecology 42:954-965. (Pubitemid 41727036)
19. CAO, Y., C. P. HAWKINS, D. P. LARSEN, AND J. VAN SICKLE. 2007a. Effects of sample standardization on mean species detectabilities and estimates of relative differences in species richness among assemblages. American Naturalist 170:381-395. (Pubitemid 47312844)
20. CAO, Y., C. P. HAWKINS, J. OLSEN, AND M. NELSON. 2007b. Modelling natural environmental gradients improves the accuracy and precision of diatom-based indicators for Idaho streams. Journal of the North American Benthological Society 26:566-585. (Pubitemid 47216912)
21. CAO, Y., C. P. HAWKINS, AND A. W. STOREY. 2005. A method for measuring the comparability of different sampling methods used in biological surveys: implications for data integration and synthesis. Freshwater Biology 50: 1105-1115. (Pubitemid 40743233)
22. CAO, Y., C. P. HAWKINS, AND M. R. VINSON. 2003. Measuring and controlling data quality in biological assemblage surveys with special reference to stream benthic macroinvertebrates. Freshwater Biology 48:1898-1911. (Pubitemid 37245118)
23. CAO, Y., W. P. WILLIAMS, AND A. W. BARK. 1997. Effects of sample size (number of replicates) on similarity measures in river Aufwuchs community analysis. Water Environment Research 69:107-114. (Pubitemid 27044969)
24. CAO, Y., D. D. WILLIAMS, AND D. P. LARSEN. 2002. Comparisons of ecological communities: the problem of sample representativeness. Ecological Monographs 72:41-56. (Pubitemid 34124492)
25. CARLISLE, D. M., AND C. P. HAWKINS. 2008. Land use and the structure of western US stream invertebrate assemblages: predictive models and ecological traits. Journal of the North American Benthological Society 27:986-999.
26. CARTER, J. L., AND V. H. RESH. 2001. After site selection and before data analysis: sampling, sorting, and laboratory procedures used in stream benthic macroinvertebrate monitoring programs by USA state agencies. Journal of the North American Benthological Society 20:658-682. (Pubitemid 34009646)
27. CLARKE, R. T., M. T. FURSE, R. J. M. GUNN, J. M. WINDER, AND J. F. WRIGHT. 2003. Sampling variation in macroinvertebrate data and implications for river quality indices. Freshwater Biology 47:1735-1751. (Pubitemid 35021985)
28. CLARKE, R. T., AND D. HERING. 2006. Errors and uncertainty in bioassessment methods-major results and conclusions from the STAR project and their applications using STARBUGS. Hydrobiologia 566:433-439. (Pubitemid 44078681)
29. CLEWS, E., AND S. J. ORMEROD. 2009. Appraising riparian management effects on benthic macroinvertebrates in the Wye River system. Aquatic Conservation: Marine and Freshwater Ecosystems 20:73-81.
30. CUFFNEY, T. E., M. D. BILGER, AND A. M. HAIGLER. 2007. Ambiguous taxa: efforts on the characterization and interpretation of macroinvertebrate assemblages. Journal of the North American Benthological Society 26: 286-307. (Pubitemid 46849402)
31. CULL, C. A., S. E. MANLEY, I. M. STRATTON, H. A. W.NEIL, I. S. ROSS, R. R. HOLMAN, R. C. TURNER, AND D. R. MATTHEWS. 1997. Approach to maintaining comparability of biochemical data during long-term clinical trials. Clinical Chemistry 43:1913-1918. (Pubitemid 27452281)
32. DAVIES, S., AND S. JACKSON. 2006. The biological condition gradient: a descriptive model for interpreting changes in aquatic ecosystems. Ecological Applications 16:1251-1266. (Pubitemid 44223205)
33. DIAMOND, J., M. T. BARBOUR, AND J. STRIBLING. 1996. Characterizing and comparing bioassessment methods and their results: a perspective. Journal of the North American Benthological Society 15:713-727. (Pubitemid 27052762)
34. DIAMOND, J., J. STRIBLING, AND C. O. YODER. 1998. Determining comparability of bioassessment methods and their results. Proceedings of the NWQMC National Monitoring Conference, July 7-9, Reno, Nevada. (Available from: http://acwi.gov/monitoring/conference/98proceedings/Papers/27-diam.htm)
35. DUFRÊNE, M., AND P. LEGENDRE. 1997. Species assemblages and indicator species: the need for a flexible asymmetrical approach. Ecological Monographs 67:345-366.
36. DURANCE, I., AND S. J. ORMEROD. 2007. Climate change effects on upland stream macroinvertebrates over a 25-year period. Global Change Biology 13:942-957. (Pubitemid 46672531)
37. EATON, L. E., AND D. R. LENAT. 1991. Comparison of a rapid bioassessment method with North Carolina's qualitative macroinvertebrate collection method. Journal of the North American Benthological Society 10:335-338.
38. ELLIOTT, J. M., AND C. M. DRAKE. 1981. A comparative study of 7 grabs used for sampling benthic macroinvertebrates in rivers. Freshwater Biology 11:99-120. (Pubitemid 11004129)
39. ENGEL, S. R., AND J. R. VOSHELL. 2002. Volunteer biological monitoring: can it accurately assess the ecological condition of streams? American Entomologists 48:164-177.
40. ERBA, S., M. T. FURSE, R. BALESTRINI, A. CHRISTODOULIDES, T. OFENBOCK, W. VAN DE BUND, J. G. WASSON, AND A. BUFFAGNI. 2009. The validation of common European class boundaries for river benthic macroinvertebrates to facilitate the intercalibration process of the Water Framework Directive. Hydrobiologia 633:17-31.
41. EUROPEAN COMMISSION. 2000. Directive 2000/60/EC of the European Parliament and of the Council of 23 October 2000 establishing a framework for Community action in the field of water policy. Official Journal of the European Communities L 327, 22.12.2000, 1-72.
42. FAITH, D. O., P. R. MINCHIN, AND L. BELBIN. 1987. Compositional dissimilarity as a robust measure of ecological distance. Vegetatio 69:57-68.
43. FEIO, M. J., T. B. REYNOLDSON, AND M. A. S. GRAÇA. 2006. The influence of taxonomic level on the performance of a predictive model for water quality assessment. Canadian Journal of Fisheries and Aquatic Sciences 63: 367-376.
44. FRIBERG, N., L. SANDIN, M. T. FURSE, S. E. LARSEN, R. T. CLARKE, AND P. HAASE. 2006. Comparison of macroinvertebrate sampling methods in Europe. Hydrobiologia 566:365-378. (Pubitemid 44078678)
45. FURSE, M. T., J. F. WRIGHT, P. D. ARMITAGE, AND D. MOSS. 1981. An appraisal of pond-net samples for biological monitoring of lotic macroinvertebrates. Water Research 15:679-689. (Pubitemid 11067953)
46. GAGE, J. D., D. J. HUGHES, AND J. L. GONZALEZ VECINO. 2000. Sieve size influence in estimating biomass, abundance and diversity in samples of deep-sea macrobenthos. Marine Ecology Progress Series 225:97-107. (Pubitemid 34161489)
47. GAO (GENERAL ACCOUNTING OFFICE). 2002. Water quality: inconsistent state approaches complicate Nation's efforts to identify its most polluted waters. GAO-02-186. US General Accounting Office, Washington, DC.
48. GERRITSEN, J., J. BURTON, AND M. BARBOUR. 2000. A stream condition index for West Virginia wadeable streams. Technical Report. Region 3, Environmental Services Division, Office of Science and Technology, Office of Water, US Environmental Protection Agency, Washington, DC. (Available from: )
49. GERRITSEN, J., AND E. W. LEPPO. 2005. Biological condition gradient for tiered aquatic life use in New Jersey. Region 2, Office of Science and Technology, US Environmental Protection Agency, and New Jersey Department of Environmental Protection, Trenton, New Jersey. (Available from: )
50. GERTH, W. J., AND A. T. HERLIHY. 2006. Effect of sampling different habitat types in regional macroinvertebrate bioassessment surveys. Journal of the North American Benthological Society 25:501-512. (Pubitemid 43837014)
51. GHETTI, P. F., AND G. BONAZZI. 1977. A comparison between various criteria for the interpretation of biological data in the analysis of the quality of running waters. Water Research 11:819-831. (Pubitemid 9039484)
52. GILBERT, B., P. J. DILLON, K. M. SOMERS, R. REID, AND L. SCOTT. 2008. Response of benthic macroinvertebrate communities to El NiÑo related drought events in six upland streams in south-central Ontario. Canadian Journal of Fisheries and Aquatic Sciences 65:890-905. (Pubitemid 351716767)
53. GILLIOM, R. J., W. M. ALLEY, AND M. E. GURTZ. 2005. Design of the National Water-Quality Assessment Program: occurrence and distribution of water-quality conditions. US Geological Survey Circular 1112. US Geological Survey, National Water-Quality Assessment (NAWQA) Program, Reston, Virginia. (Available from: )
54. GOTELLI, N. J., AND G. R. GRAVES. 1996. Null models in ecology. Smithsonian Institution Press, Washington, DC.
55. GREEN, R. H. 1980. Multivariate approaches in ecology: the assessment of ecological similarity. Annual Review of Ecology and Systematics 11:1-14.
56. GRIFFITH, J. F., L. A. AUMAND, I. M. LEE, C. D. MCGEE, L. L. OTHMAN, K. L. RITTER, K. O. WALKER, AND S. B. WEISBERG. 2006. Comparison and verification of bacterial water quality indicator measurement methods using ambient coastal water samples. Environmental Monitoring and Assessment 116:335-344. (Pubitemid 43899074)
57. HAASE, P., S. LOHSE, S. PAULS, K. SCHINDEHUTTE, A. SUNDERMANN, P. ROLAUFFS, AND D. HERING. 2004. Assessing streams in Germany with benthic invertebrates: development of a practical standardized protocol for macroinvertebrate sampling and sorting. Limnologica 34:349-365. (Pubitemid 40102831)
58. HAASE, P., J. MURRAY-BLIGH, S. LOHSE, S. PAULS, A. SUNDERMAN, R. GUNN, AND R. CLARKE. 2006. Assessing the impact of errors in sorting and identifying macroinvertebrate samples. Hydrobiologia 566:505-521. (Pubitemid 44078685)
59. HAWKES, H. A. 1979. Invertebrates as indicators of river water quality. Pages 2:1-2:45 in A. James and L. Evison (editors). Biological indicators of water quality. John Wiley and Sons, Chichester, UK.
60. HAWKINS, C. P. 2006. Quantifying biological integrity by taxonomic completeness: evaluation of a potential indicator for use in regional- and global-scale assessments. Ecological Applications 16:1277-1294.
61. HAWKINS, C. P., Y. CAO, AND R. ROPER. 2010a. Method of predicting reference conditions affects the performance and interpretation of ecological indices. Freshwater Biology 55:1066-1085.
62. HAWKINS, C. P., R. H. NORRIS, J. N. HOGUE, AND J. W. FEMINELLA. 2000. Development and evaluation of predictive models for measuring the biological integrity of streams. Ecological Application 10:1456-1477. (Pubitemid 32140010)
63. HAWKINS, C. P., J. R. OLSEN, AND R. A. HILL. 2010b. The reference condition: predicting benchmarks for ecological and water-quality assessment. Journal of the North American Benthological Society 29:312-343.
64. HEINZ CENTER. 2002. The state of the Nation's ecosystems: measuring the lands, waters, and living resources of the United States. Cambridge University Press, Washington, DC.
65. HERBST, D. B., AND E. L. SILLDORFF. 2006. Comparison of the performance of different bioassessment methods: similar evaluations of biotic integrity from separate programs and procedures. Journal of the North American Benthological Society 25:513-530. (Pubitemid 43837015)
66. HERLIHY, A. T., S. G. PAULSEN, J.VAN SICKLE, J. L. STODDARD, C. P. HAWKINS, AND L. YUAN. 2008. Striving for consistency in a national assessment: the challenges of applying a reference- condition approach at a continental scale. Journal of the North American Benthological Society 27:860-877.
67. HILSENHOFF, W. L. 1987. An improved biotic index of organic stream pollution. Great Lakes Entomologist 20:31-39.
68. HOSE, G., E. TURAK, AND N.WADDELL. 2004. Reproducibility of AUSRIVAS rapid bioassessments using macroinvertebrates. Journal of the North American Benthological Society 23:126-139. (Pubitemid 38241663)
69. HOUSTON, L., M. T. BARBOUR, D. LENAT, AND D. PENROSE. 2002. Multi-agency comparison of aquatic macroinvertebratebased stream bioassessment methodologies. Ecological Indicators 1:279-292. (Pubitemid 33806099)
70. HUGHES, R. M., S. G. PAULSEN, AND J. L. STODDARD. 2000. EMAP-surface water: a multi-assemblage, probability survey of ecological integrity in the U.S.A. Hydrobiologia 422/423:429-443. (Pubitemid 30471511)
71. HUGHES, R. M., AND D. V. PECK. 2008. Acquiring data for large aquatic resource surveys: the art of compromise among science, logistics, and reality. Journal of the North American Benthological Society 27:837-859.
72. HUMPHREY, C. L., A. W. STOREY, AND L. THURTELL. 2000. AUSRIVAS: operator sample processing errors and temporal variability-implications for model sensitivity. Pages 143-165 in J. F. Wright, D. W. Sutcliffe, and M. T. Furse (editors). Assessing the biological quality of fresh waters. Freshwater BiologicalAssociation, Ambleside, UK.
73. HURLBERT, S. H. 1971. The nonconcept of species diversity: a critique and alternative parameters. Ecology 52:577-586.
74. HURST, C. P., C. P. CATTERALL, AND J. CHASELING. 2008. A comparison of two methods for generating artificial multi-assemblage ecological datasets. Ecological Informatics 3:286-294.
75. ITFM (INTERGOVERNMENTAL TASK FORCE ON MONITORING WATER QUALITY). 1994. Report of the Interagency Biological Methods Workshop. U.S. Geological Survey Open-file Report 94-490. US Geological Survey, Reston, Virginia.
76. ITFM (INTERGOVERNMENTAL TASK FORCE ON MONITORING WATER QUALITY). 1995. Strategy for improving water-quality monitoring in the United States. Final Report of the Intergovernmental Task Force on Monitoring Water Quality. U.S. Geological Survey Open-file Report 95-742. US Geological Survey, Reston, Virginia.
77. KARR, J. R., AND E. W. CHU. 1999. Restoring life in running waters: better biological monitoring. Island Press, Washington, DC.
78. KELLY, M. G. 1999. Progress towards quality assurance of benthic diatom and phytoplankton analyses in the UK. Pages 208-215 in J. Prygiel, B. A. Whitton, and J. Bukowska (editors). Use of algae for monitoring rivers III. Agence de l'Eau Artois-Picardie, Douai, France, (Available from: Agence de l'Eau Artois-Picardie, 200, Rue Marceline, Centre Tertiaire de l'Arsenal BP818, F- 59508 DOUAI CEDEX, France.)
79. KELLY, M. G. 2000. Use of similarity measures for quality control of benthic diatom samples. Water Research 35: 2784-2788.
80. KENKEL, N. C. 2006. On selecting an appropriate multivariate analysis. Canadian Journal of Plant Science 86:663-676. (Pubitemid 46712259)
81. KLEMM, D. L., K. A. BLOCKSOM, F. A. FULK, A. T. HERLIHY, R. M. HUGHES, P. R. KAUFMANN, D. V. PECK, J. L. STODDARD, W. T. THOENY, M. B. GRIFFITH, AND W. S. DAVIS. 2003. Development and evaluation of a Macroinvertebrate Biotic Integrity Index (MBII) for regionally assessing Mid-Atlantic Highlands streams. Environmental Management 31:656-669. (Pubitemid 36959017)
82. LAMB, E. G., E. BAYNE, G. HOLLOWAY, J. SCHIECK, S. BOUTIN, J. HERBERS, AND D. L. HAUGHLAND. 2009. Indices for monitoring biodiversity change: are some more effective than others? Ecological Indicators 9:432-444. (Pubitemid 50232385)
83. LEIBOLD, M. A., J. M. CHASE, J. B. SHURIN, AND A. L. DOWNING. 1997. Species turnover and regulation of trophic structure. Annual Review of Ecology and Systematics 28:467-494. (Pubitemid 28012300)
84. LENAT, D. R. 1993. A biotic index for the southeastern United States: derivation and list of tolerance values, with criteria for assigning water-quality rating. Journal of the North American Benthological Society 12:279-290.
85. LENAT, D. R., AND V. H. RESH. 2001. Taxonomy and stream ecology: the benefits of genus- and species-level identifications. Journal of the North American Benthological Society 20:287-298. (Pubitemid 32449516)
86. LENZ, B. N., AND M. MILLER. 1996. Comparison of aquatic macroinvertebrate samples collected using different field methods. Fact Sheet FS-216-96. National Water- Quality Assessment Program, US Geological Survey, Madison, Wisconsin. (Available from: . usgs.gov/pubs/FS-216-96/fs-216-96.pdf)
87. LLANSÓ, R. J., J. H. VøLSTAD, D. M. DAUER, AND J. R. DEW. 2009. Assessing benthic community condition in Chesapeake Bay: does the use of different benthic indices matter? Environmental Monitoring and Assessment 150: 119-127.
88. MACKEY, A. P., D. A. COOLING, AND A. D. BERRIE. 1984. An evaluation of sampling strategies for qualitative surveys of macro-invertebrates in rivers, using pond nets. Journal of Applied Ecology 21:515-534. (Pubitemid 16273356)
89. MAXTED, J. R., M. T. BARBOUR, J. GERRITSEN, V. PORETTI, N. PRIMROSE, A. SILVIA, D. PENROSE, AND R. RENFROW. 2000. Assessment framework for mid-Atlantic coastal plain streams using benthic macroinvertebrates. Journal of the North American Benthological Society 19:128-144. (Pubitemid 30244573)
90. MAZOR, R. D., A. H. PURCELL, AND V. H. RESH. 2009. Longterm variability in bioassessments: a twenty-year study from two northern California streams. Environmental Management 43:1269-1286.
91. MAZOR, R. D., T. B. REYNOLDSON, D. M. ROSENBERG, AND V. H. RESH. 2006. Effects of biotic assemblage, classification, and assessment method on bioassessment performance. Canadian Journal of Fisheries and Aquatic Sciences 63: 394-411.
92. MDCB (METHODS AND DATA COMPARABILITY BOARD). 2001. Towards a definition of performance-based laboratory methods: a position paper developed by the Methods and Data Comparability Board (MDCB). Technical Report 01-02. National Water Quality Monitoring Council. (Available from: )
93. MDCB (METHODS AND DATA COMPARABILITY BOARD). 2003. The value of data comparability. Fact Sheet. National Water Quality Monitoring Council. (Available from: . water.usgs.gov/methods/about/publications/valcomp-fs. pdf)
94. MEDEIROS, E. S. F., AND L. MALTCHIK. 2001. Fish assemblage stability in an intermittently flowing stream from the Brazilian semiarid region. Austral Ecology 26:156-164. (Pubitemid 33719681)
95. MINCHIN, P. R. 1987. An evaluation of the relative robustness of techniques for ecological ordination. Vegetatio 69: 89-107.
96. MOSS, D., M. T. FURSE, J. F. WRIGHT, AND P. D. ARMITAGE. 1987. The prediction of the macroinvertebrate fauna of unpolluted running-water sites in Great Britain using environmental data. Freshwater Biology 17:41-52.
97. NICHOLS, S. J., AND R. H. NORRIS. 2006. River condition assessment may depend on the sub-sampling method: field live-sort versus laboratory sub-sampling of invertebrates for bioassessment. Hydrobiologia 572:195-213. (Pubitemid 44496482)
98. NORRIS, R. H., AND C. P. HAWKINS. 2000. Monitoring river health. Hydrobiologia 435:5-17.
99. NYDICK, K. R., B. M. LAFRANCOIS, J. S. BARON, AND B. M. JOHNSON. 2003. Lake specific responses to elevated atmospheric nitrogen deposition in the Colorado Rocky Mountains, U.S.A. Hydrobiologia 510:103-114. (Pubitemid 38086414)
100. ODE, P. R., C. P. HAWKINS, AND R. D. MAZOR. 2008. Comparability of biological assessments derived from predictive models and multimetric indices of increasing geographic scope. Journal of the North American Benthological Society 27:967-985.
101. OLSEN, A. R., AND D. V. PECK. 2008. Survey design and extent estimates for the Wadeable Streams Assessment. Journal of the North American Benthological Society 27:822-836.
102. OSTERMILLER, J. D., AND C. P. HAWKINS. 2004. Effect of sampling error on bioassessments of stream ecosystems: application to RIVPACS-type models. Journal of the North American Benthological Society 23:363-382. (Pubitemid 38729978)
103. OVERTON, J. M., T. C. YOUNG, AND W. S. OVERTON. 1993. Using 'found' data to augment a probability sample: procedure and case study. Environmental Monitoring and Assessment 26:65-83.
104. PARSONS, M., AND R. H. NORRIS. 1996. The effect of habitatspecific sampling on biological assessment of water quality using a predictive model. Freshwater Biology 36: 419-434.
105. PLOTKIN, J. B., AND H. C. MULLER-LANDAU. 2002. Sampling the species composition of a landscape. Ecology 83: 3344-3356. (Pubitemid 36524650)
106. PONT, D., B. HUGUENY, U. BEIER, D. GOFFAUX, A. MELCHER, R. NOBLE, C. ROGERS, N. ROSET, AND S. SCHMUTZ. 2006. Assessing river biotic condition at a continental scale: a European approach using functional metrics and fish assemblages. Journal of Applied Ecology 43:70-80.
107. QUATAERT, P., J. BREINE, AND I. SIMOENS. 2004. Comparison of the European Fish Index with the standardised European model, the spatially based models (eco-regional and European), and existing methods. Institute for Forestry and Game Management, Groenendaal-Hoeilaart, Belgium.
108. REECE, P. F., T. B. REYNOLDSON, J. S. RICHARDSON, AND D. M. ROSENBERG. 2001. Implications of seasonal variation for biomonitoring with predictive models in the Fraser River catchment, British Columbia. Canadian Journal of Fisheries and Aquatic Sciences 58:1411-1418. (Pubitemid 32770134)
109. REHN, A. C., P. R. ODE, AND C. P. HAWKINS. 2007. Comparisons of targeted-riffle and reach-wide benthic macroinvertebrate samples: implications for data sharing in stream-condition assessments. Journal of the North American Benthological Society 26:332-348. (Pubitemid 46849405)
110. REYNOLDSON, T. B., R. C. BAILEY, K. E. DAY, AND R. H. NORRIS. 1995. Biological guidelines for freshwater sediment based on BEnthic Assessment SedimenT (the BEAST) using a multivariate approach for predicting biological state. Australian Journal of Ecology 20:198-219.
111. RILEY, L. A., M. F. DYBDAHL, AND R. O. HALL. 2008. Invasive species impact: asymmetric interactions between invasive and endemic freshwater snails. Journal of the North American Benthological Society 27:509-520.
112. ROSE, P., L. METZELING, AND S. CATZIKRIS. 2008. Can macroinvertebrate rapid bioassessment methods be used to assess river health during drought in south eastern Australian streams? Freshwater Biology 53:2626-2638.
113. ROTH, N., J. VøLSTAD, G. MERCURIO, AND M. SOUTHERLAND. 2002. Biological indicator variability and stream monitoring integration: a Maryland case study. EPA/903-02/ 008. Office of Environmental Information and Mid- Atlantic Integrated Assessment Team, US Environmental Protection Agency, Fort Meade, Maryland.
114. ROY, T. V., C. T. ROBINSON, AND G. W. MINSHALL. 2003. Development of macroinvertebrate-based index for bioassessment of Idaho streams. Environmental Management 27:627-636. (Pubitemid 32185146)
115. SANDERS, H. L. 1968. Marine benthic diversity: a comparative study. American Naturalist 102:243-282.
116. SHELTON, A. D., AND K. A. BLOCKSOM. 2004. A review of biological assessment tools and biocriteria for rivers and streams in New England states. EPA/600/R-04/168. US Environmental Protection Agency, Cincinnati, Ohio.
117. SLACK, K. V., L. J. TILLEY, AND S. S. KENNELLY. 1991. Mesh-size effects on drift sample composition as determined with a triple net sampler. Hydrobiologia 209:215-226.
118. SOKAL, R. R., AND F. J. ROHLF. 1987. Biometry: the principles and practice of statistics in biological research. W. H. Freeman and Company, San Francisco, California.
119. SOMERFIELD, P. G., AND K. R. CLARKE. 1996. A comparison of some methods commonly used for collection of sublittoral sediments and their associated fauna. Marine Environmental Research 43:145-156.
120. SOUTHERLAND, M., G. ROGERS, M. KLINE, R. MORGAN, D. BOWARD, P. KAZYAK, R. KLAUDA, AND S. STRANKO. 2005. New biological indicators to better assess the condition of Maryland streams. Monitoring and Non-Tidal Assessment Division, Maryland Department of Natural Resources, Annapolis, Maryland. (Available from: )
121. STATE OF MAINE. 2003. Code of Maine Rules 06-096. Chapter 579: Classification attainment evaluation using biological criteria for rivers and streams. Department of Environmental Protection, Augusta, Maine. (Available from: biomonitoring/material.htm)
122. STATE OF OHIO. 2003. Ohio Administrative Code at OAC Chapter 3745-1. State of Ohio Water Quality Standards. State of Ohio, Columbus, Ohio. (Available from: http:// www.epa.state.oh.us/portals/35/rules/01-00-eff031510.pdf )
123. STEPENUCK, K. F., R. L. CRUNKILTON, M. A. BOZEK, AND L. WANG. 2008. Comparison of macroinvertebrate-derived stream quality metrics between snag and riffle habitats. Journal of the American Water Resources Association 44:670-678. (Pubitemid 351712957)
124. STEVENS, D. 1994. Implementation of a national monitoring program. Journal of Environmental Management 42: 1-29.
125. STODDARD, J. 2006. Use of ecological regions in aquatic assessments of ecological condition. Environmental Management 34:61-70.
126. STODDARD, J. L., A. T. HERLIHY, D. V. PECK, R. M. HUGHES, T. R. WHITTIER, AND E. TARQUINIO. 2008. A process for creating multimetric indices for large-scale aquatic surveys. Journal of the North American Benthological Society 27:878-891.
127. STODDARD, J. L., D. P. LARSEN, C.P.HAWKINS, R. K. JOHNSON, AND R. H. NORRIS. 2006. Setting expectations for the ecological condition of running waters: the concept of reference conditions. Ecological Applications 16:1267-1276. (Pubitemid 44223206)
128. STOREY, A. W., D. H. D. EDWARD, AND P. GAZEY. 1991. Surber and kick sampling: a comparison for the assessment of macroinvertebrate community structure in streams of south-western Australia. Hydrobiologia 211:111-121.
129. STRIBLING, J. B., S. R. MOULTON, AND G. T. LESTER. 2003. Determining the quality of taxonomic data. Journal of the North American Benthological Society 22:621-631. (Pubitemid 37481650)
130. SUTER, G. W. 2007. Ecological risk assessment. 2nd edition. CRC Press, Boca Raton, Florida.
131. TAYLOR, B. W., A. R. MCINTOSH, AND B. L. PECKARSKY. 2001. Sampling stream invertebrates using electroshocking techniques: implications for basic and applied research. Canadian Journal of Fisheries and Aquatic Sciences 58: 437-445. (Pubitemid 32333547)
132. TETRA TECH. 2005. Comparability analysis of benthic macroinvertebrate methods in Montana. A report to Montana Department of Environmental Quality, Helena, Montana, prepared by Tetra Tech, Inc., Owings Mills, Maryland. (Available from: )
133. TREBITZ, A. S., B. H. HILL, AND F. H. MCCORMICK. 2003. Sensitivity of indices of biotic integrity to simulated fish assemblage changes. Environmental Management 32: 499-515. (Pubitemid 38112307)
134. TURNER, A. M., AND J. C. TREXLER. 1997. Sampling aquatic invertebrates from marshes: evaluating the options. Journal of the North American Benthological Society 16:694-709. (Pubitemid 27398799)
135. USEPA (US ENVIRONMENTAL PROTECTION AGENCY). 1990. Methods for measuring the acute toxicity of effluents and receiving waters to aquatic organisms. 4th edition. EPA/ 600-4-90-0027. Office of Research and Development, US Environmental Protection Agency, Cincinnati, Ohio.
136. USEPA (US ENVIRONMENTAL PROTECTION AGENCY). 1992. Framework for ecological risk assessment. EPA/630/ R-92/001. Risk Assessment Forum, US Environmental Protection Agency, Washington, DC.
137. USEPA (US ENVIRONMENTAL PROTECTION AGENCY). 2003. Draftreport on the environment 2003. EPA 260-R-02-006. US Environmental Protection Agency, Washington, DC.
138. USEPA (US ENVIRONMENTAL PROTECTION AGENCY). 2005a. Proof of concept for integrating bioassessment results from three states probabilistic monitoring programs. EPA-903-R-05-003. Office of Environmental Information, US Environmental Protection Agency, Washington, DC.
139. USEPA (US ENVIRONMENTAL PROTECTION AGENCY). 2005b. Use of biological information to better define designated aquatic life uses in state and tribal water quality standards: tiered aquatic life uses. Draft. EPA-822-R- 05-001. Health and Ecological Criteria Division (4304T), Office of Science and Technology, Office of Water, US Environmental Protection Agency, Washington, DC.
140. USEPA (US ENVIRONMENTAL PROTECTION AGENCY). 2006. Wadeable streams assessment: a collaborative survey of the Nation's streams. EPA 841-B-06-002. Office of Research and Development, Office of Water, US Environmental Protection Agency, Washington, DC.
141. VAN SICKLE, J. 1997. Using mean similarity dendrograms to evaluate classification. Journal of Agricultural, Biological, and Environmental Statistics 2:370-388. (Pubitemid 127478511)
142. VøLSTAD, J., N. ROTH, M. SOUTHERLAND, AND G. MERCURIO. 2003. Pilot study for Montgomery County and Maryland DNR data integration: comparison of benthic macroinvertebrate sampling protocols for freshwater streams. EPA/903/R-03/005. Office of Environmental Information and Mid-Atlantic Integrated Assessment Team, US Environmental Protection Agency, Fort Meade, Maryland.
143. WAITE, I. R., A. T. HERLIHY, D. P. LARSEN, N. S. URQUHART, AND D. J. KLEMM. 2004. The effects of macroinvertebrate taxonomic resolution in large landscape bioassessments: an example from the Mid-Atlantic Highlands, U.S.A. Freshwater Biology 49:474-489. (Pubitemid 38448311)
144. WALTERS, D. M., M. J. BLUM, B. RASHLEIGH, B. J. FREEMAN, B. A. PORTER,
145. WEILHOEFER, C. L., AND Y. PAN. 2007. A comparison of diatom assemblages generated by two sampling protocols. Journal of the North American Benthological Society 26:308-318. (Pubitemid 46849403)
146. WOLDA, H. 1981. Similarity indices, sample size and diversity. Oecologia (Berlin) 50:296-302.
147. WRIGHT, J. F., D. W. SUTCLIFFE, AND M. T. FURSE. 2000. Assessing the biological quality of fresh waters: RIVPACS and other techniques. Freshwater Biological Association, Ambleside, UK.
148. YODER, C. P., AND G. D. DAVIS. 1996. The Ohio EPA bioassessment comparability project: a preliminary analysis. Ohio EPA Technical Bulletin MAS/1996-12-4. Ohio Environmental Protection Agency, Columbus, Ohio. (Available from: portals/35/documents/biocomp.pdf)
149. YUAN, L. L., C. P. HAWKINS, AND J. VAN SICKLE. 2008. Effects of regionalization decisions on an O/E index for the US national assessment. Journal of the North American Benthological Society 27:892-905.
150. ZAR, J. H. 1999. Biostatistical analysis. 4th edition. Prentice Hall, Upper Saddle River, New Jersey.