Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation

PLoS ONE

Volumn 10, Issue 7, 2015, Pages

  Beijbom, Oscar ^a   Edmunds, Peter J ^b   Roelfsema, Chris ^c   Smith, Jennifer ^d   Kline, David I ^d   Neal, Benjamin P ^c,d   Dunlap, Matthew J ^e   Moriarty, Vincent ^b   Fan, Tung Yung ^f   Tan, Chih Jui ^f   Chan, Stephen ^a   Treibitz, Tali ^g   Gamst, Anthony ^h   Mitchell, B Greg ^d   Kriegman, David ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b CALIFORNIA STATE UNIVERSITY   (United States)

^c UNIVERSITY OF QUEENSLAND   (Australia)

^d INTEGRATIVE OCEANOGRAPHY DIVISION   (United States)

^e UNIVERSITY OF HAWAII   (United States)

^f NATIONAL MUSEUM OF MARINE BIOLOGY AND AQUARIUM   (Taiwan)

^g UNIVERSITY OF HAIFA   (Israel)

^h UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY; ARTICLE; AUTOMATION; BENTHOS; CONTROLLED STUDY; CORAL REEF; FEASIBILITY STUDY; HUMAN; IMAGE ANALYSIS; IMAGE DISPLAY; IMAGE PROCESSING; INTERMETHOD COMPARISON; QUANTITATIVE ANALYSIS; ALGORITHM; ANIMAL; ANTHOZOA; AUTOMATED PATTERN RECOGNITION; CLIMATE CHANGE; COMPARATIVE STUDY; ECOSYSTEM; ENVIRONMENTAL MONITORING; OBSERVER VARIATION; PHYSIOLOGY; PROCEDURES; REPRODUCIBILITY; SEAWEED; STATISTICAL MODEL;

ALGORITHMS; ANIMALS; ANTHOZOA; CLIMATE CHANGE; CORAL REEFS; ECOSYSTEM; ENVIRONMENTAL MONITORING; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MODELS, STATISTICAL; OBSERVER VARIATION; PATTERN RECOGNITION, AUTOMATED; REPRODUCIBILITY OF RESULTS; SEAWEED;

EID: 84941313125     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0130312     Document Type: Article

References (51)

1. Costanza R, D'Arge R, Groot R De, Farber S, Grasso M, Hannon B, et al. The value of the world's ecosystem services and natural capital. Ecol Econ. 1998;25:3-15.
2. Cesar H, Burke L, Pet-Soede L. The economics of worldwide coral reef degradation. Cesar Environ Econ Consult. 2003;
3. Hoegh-Guldberg O, Mumby PJ, Hooten AJ, Steneck RS, Greenfield P, Gomez E, et al. Coral reefs under rapid climate change and ocean acidification. Science. 2007;318:1737-1742. PMID: 18079392
4. Gardner TA, Côté IM, Gill JA, Grant A, Watkinson AR. Long-term region-wide declines in Caribbean corals. Science. 2003;301:958-960. PMID: 12869698
5. Bruno JF, Selig ER. Regional Decline of Coral Cover in the Indo-Pacific: Timing, Extent, and Subregional Comparisons. PLoS One. 2007;2:e711. PMID: 17684557
6. Van Hooidonk R, Maynard JA, Planes S. Temporary refugia for coral reefs in a warming world. NatClim Change. 2013;3:508-511.
7. Hughes TP, Baird AH, Bellwood DR, Card M, Connolly SR, Folke C, et al. Climate change, human impacts, and the resilience of coral reefs. Science. 2003;301:929-933. PMID: 12920289
8. Bellwood DR, Hughes TP, Folke C, Nyström M. Confronting the coral reef crisis. Nature. 2004;429:827-833. PMID: 15215854
9. Jaffe JS, Moore KD, McLean J, Strand MP. Underwater optical imaging: status and prospects. Oceanogr. 2001;14:66-76.
10. Pante E, Dustan P. Getting to the Point: Accuracy of Point Count in Monitoring Ecosystem Change. J Mar. Biol. 2012;2012.
11. Gleason ACR, Gracias N, Lirman D, Gintert BE, Smith TB, Dick MC, et al. Landscape video mosaic from a mesophotic coral reef. Coral Reefs. 2010;29:253-253.
12. Carleton JH, Done TJ. Quantitative video sampling of coral reef benthos: large-scale application. Coral Reefs. 1995;14:35-46.
13. Ninio R, Delean JSC, Osborne K, Sweatman H. Estimating cover of benthic organisms from underwater videoimages: variability associated with multiple observers. Mar. Ecol Prog Ser. 2003;265:107-116.
14. Singh H, Armstrong R, Gilbes F, Eustice R, Roman C, Pizarro O, et al. Imaging coral I: imaging coral habitats with the SeaBED AUV. Subsurf Sens Technol Appl. 2004;5:25-42.
15. Nicholson JW, Healey AJ. The present state of autonomous underwater vehicle (AUV) applications and technologies. J Mar. Technol Society. 2008;42:44-51.
16. Eleftheriou A. Methods for the study of marine benthos. John Wiley & Sons; 2013.
17. González-Rivero M, Bongaerts P, Beijbom O, Pizarro O, Friedman A, Rodriguez-Ramirez A, et al. The Catlin Seaview Survey-kilometre-scale seascape assessment, and monitoring of coral reef ecosystems. AquatConserv. 2014;24:184-198.
18. Molloy PP, Evanson M, Nellas AC, Rist JL, Marcus JE, Koldewey HJ, et al. How much sampling does it take to detect trends in coral-reef habitat using photoquadrat surveys? AquatConserv. 2013;23:820-837.
19. Kohler KE, Gill SM. Coral Point Count with Excel extensions (CPCe): A Visual Basic program for the determination of coral and substrate coverage using random point count methodology. Comput Geosci. 2006;32:1259-1269.
20. Trygonis V, Sini M. photoQuad: A dedicated seabed image processing software, and a comparative error analysis of four photoquadrat methods. J Exp Mar. Bio Ecol. 2012;424:99-108.
21. Porter JW, Kosmynin V, Patterson KL, Porter KG, Jaap WC, Wheaton JL, et al. Detection of coral reef change by the Florida Keys coral reef monitoring project. The Everglades, Florida Bay, and coral reefs of the Florida Keys: An ecosystem sourcebook. 2002;749-769.
22. Marcos MSA, Soriano M, Saloma C. Classification of coral reef images from underwater video using neural networks. Optics Express. 2005;13:8766-8771. PMID: 19498910
23. Stokes MD, Deane GB. Automated processing of coral reef benthic images. Limnol Oceanogr Meth. 2009;7:157-168.
24. Beijbom O, Edmunds PJ, Kline DI, Mitchell BG, Kriegman D. Automated annotation of coral reef survey images. Proc IEEEComputSocConf Comput Vis Pattern Recognit 2012;1:1170-1177.
25. Bewley M, Douillard B, Nourani-Vatani N, Friedman A, Pizarro O, Williams S. Automated species detection: An experimental approach to kelp detection from sea-floor AUV images. Proc Australas Conf Rob Autom. 2012;
26. Shihavuddin ASM, Gracias N, Garcia R, Gleason AC, Gintert B. Image-Based Coral Reef Classification and Thematic Mapping. Remote Sens. 2013;5:1809-1841.
27. Phinn SR, Roelfsema CM, Mumby PJ. Multi-scale, object-based image analysis for mapping geomorphic and ecological zones on coral reefs. Int J Remote Sens. 2012;33:3768-3797.
28. Fan R-E, Chang K-W, Hsieh C-J, Wang X-R, Lin C-J. LIBLINEAR: A library for large linear classification. J Mach Learn Res. 2008;9:1871-1874.
29. Perronnin F, Dance C. Fisher kernels on visual vocabularies for image categorization. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2007;1:1-8.
30. Chatfield K, Lempitsky V, Vedaldi A, Zisserman A. The devil is in the details: an evaluation of recent feature encoding methods. Proc Br Mach Vis Conf. 2011;
31. Branson S, Beijbom O, Belongie S. Efficient large-scale structured learning. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2013. pp. 1806-1813.
32. Carletta J. Assessing agreement on classification tasks: the kappa statistic. Comput Linguist. 1996;22:249-254.
33. Rifkin R, Klautau A. In defense of one-vs-all classification. J Mach Learn Res. 2004;5:101-141.
34. Lewis DD, Catlett J. Heterogenous Uncertainty Sampling for Supervised Learning. Proc Int Conf Mach Learn. 1994;94:148-156.
35. Solow A, Davis C, Hu Q. Estimating the taxonomic composition of a sample when individuals are classified with error. Mar. Ecol Prog Ser. 2001;216:309-311.
36. Hopkins DJ, King G. A method of automated nonparametric content analysis for social science. Am J Pol Sci. 2010;54:229-247.
37. Good P. Permutation, parametric and bootstrap tests of hypotheses. Springer; 2005.
38. Allouche O, Tsoar A, Kadmon R. Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS). J Appl Ecol. 2006;43:1223-1232.
39. Manel S, Williams HC, Ormerod SJ. Evaluating presence-absence models in ecology: the need to account for prevalence. J Appl Ecol. 2001;38:921-931.
40. Monserud RA, Leemans R. Comparing global vegetation maps with the Kappa statistic. Ecol Model. 1992;62:275-293.
41. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;159-174.
42. Fleiss JL, Levin B, Paik MC The measurement of interrater agreement. Stat Methods Rates Prop. 1981;2:212-236.
43. Miller I, Müller R. Validity and reproducibility of benthic cover estimates made during broadscale surveys of coral reefs by manta tow. Coral Reefs. 1999;18:353-356.
44. Zawada DG, Mazel CH. Fluorescence-Based Classification of Caribbean Coral Reef Organisms and Substrates. PLoSOne. 2014;9:e84570. doi: 10.1371/journal.pone.0084570 PMID: 24482676
45. Treibitz T, Neal BP, Kline DI, Beijbom O, Roberts PL, Mitchell BG, et al. Wide Field-of-View Fluorescence Imaging of Coral Reefs. Sci Rep. 2015;5.
46. Buskey EJ, Hyatt CJ. Use of the FlowCAM for semi-automated recognition and enumeration of red tide cells (Karenia brevis) in natural plankton samples. Harmful Algae. 2006;5:685-692.
47. Sosik HM, Olson RJ. Automated taxonomic classification of phytoplankton sampled with imaging-inflow cytometry. Limnol Oceanogr Methods. 2007;5:204-216.
48. Olson RJ, Sosik HM. A submersible imaging-in-flow instrument to analyze nano- and microplankton: Imaging FlowCytobot. Limnol Oceanogr Methods. 2007;5:195-203.
49. Beijbom O. Cost-Effective Sampling for Pairs of Annotators. arXiv preprint arXiv:14107074. 2014;
50. Pan SJ, Yang Q. A survey on transfer learning. IEEE Trans Knowl Data Eng. 2010;22:1345-1359.
51. Beijbom O. Domain Adaptations for Computer Vision Applications. arXiv preprint arXiv:12114860. 2012;