Open and scalable analytics of large Earth observation datasets: From scenes to multidimensional arrays using SciDB and GDAL

ISPRS Journal of Photogrammetry and Remote Sensing

Volumn 138, Issue , 2018, Pages 47-56

  Appel, Marius ^a   Lahn, Florian ^a   Buytaert, Wouter ^b   Pebesma, Edzer ^a  

^a UNIVERSITY OF MÜNSTER   (Germany)

^b IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Big data; Database systems; Earth observation; Open science; Scalable analytics; Spatial information science

Indexed keywords

CLIMATE CHANGE; INFORMATION MANAGEMENT; LAND USE; LARGE DATASET; OBSERVATORIES; OPEN SOURCE SOFTWARE; OPEN SYSTEMS; ORTHOGONAL FUNCTIONS; SATELLITE IMAGERY; TIME SERIES;

DATA ABSTRACTION; DATA VOLUME; EARTH OBSERVATIONS; EARTH'S SURFACE; GEO-SPATIAL DATA; MULTIDIMENSIONAL ARRAYS; OPEN SCIENCE; SCALABLE ANALYTIC; SPATIAL INFORMATION SCIENCE; SPATIO-TEMPORAL MODELS;

CLIMATE MODELS;

ARRAY; CLIMATE MODELING; DATA MANAGEMENT; DATA SET; EMPIRICAL ORTHOGONAL FUNCTION ANALYSIS; EOS; LAND USE CHANGE; LANDSAT; SOFTWARE;

EID: 85042181249     PISSN: 09242716     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.isprsjprs.2018.01.014     Document Type: Article

References (56)

1. Appel, M., Lahn, F., Pebesma, E., Buytaert, W., Moulds, S., 2016. Scalable Earth-observation analytics for geoscientists: spacetime extensions to the array database SciDB. In: EGU General Assembly Conference Abstracts. Vol. 18.
2. Bakcsa, Z., Bauer-Marschallinger, B., Baum, D., Boresch, A., Ertl, M., Harustak, M., Paulik, C., Reimer, C., 2016. SMDC Benchmark Study Report. Tech. Rep., AWST GmbH and TU Wien.
3. Baumann, P., A database array algebra for spatio-temporal data and beyond. International Workshop on Next Generation Information Technologies and Systems, 1999, Springer, 76–93.
4. Baumann, P., Dehmel, A., Furtado, P., Ritsch, R., Widmann, N., The multidimensional database system RasDaMan. Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data. SIGMOD ’98, 1998, ACM, New York, NY, USA, 575–577 http://doi.acm.org/10.1145/276304.276386.
5. Baumann, P., Holsten, S., A comparative analysis of array models for databases. Database Theory and Application, Bio-Science and Bio-Technology, 2011, Springer, 80–89.
6. Baumann, P., Mazzetti, P., Ungar, J., Barbera, R., Barboni, D., Beccati, A., Bigagli, L., Boldrini, E., Bruno, R., Calanducci, A., et al. Big data analytics for earth sciences: the EarthServer approach. Int. J. Digital Earth 9:1 (2016), 3–29, 10.1080/17538947.2014.1003106.
7. Cheng, G., Han, J., Lu, X., Remote sensing image scene classification: benchmark and state of the art. Proc. IEEE 105:10 (2017), 1865–1883 Oct.
8. Choi, J., Dongarra, J.J., Pozo, R., Walker, D.W., ScaLAPACK: a scalable linear algebra library for distributed memory concurrent computers. Fourth Symposium on the Frontiers of Massively Parallel Computation, 1992, 1992, IEEE, 120–127.
9. Cressie, N., Wikle, C.K., Statistics for spatio-temporal data. 2015, John Wiley & Sons.
10. Dean, J., Ghemawat, S., MapReduce: simplified data processing on large clusters. Commun. ACM 51:1 (2008), 107–113.
11. DeVries, B., Verbesselt, J., Kooistra, L., Herold, M., Robust monitoring of small-scale forest disturbances in a tropical montane forest using Landsat time series. Remote Sens. Environ. 161 (2015), 107–121.
12. European Space Agency, 2016. Sentinel-2 MSI user guide. https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi/data-formats (accessed: 2016–11–29).
13. Eyring, V., Bony, S., Meehl, G.A., Senior, C.A., Stevens, B., Stouffer, R.J., Taylor, K.E., Overview of the coupled model intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geoscientific Model Dev. 9:5 (2016), 1937–1958.
14. Fisher, P.F., The pixel: a snare and a delusion. Int. J. Remote Sens. 18:3 (1997), 679–685.
15. Goor, E., Dries, J., 2017. Proba-V Mission Exploitation Platform. In: EGU General Assembly Conference Abstracts. Vol. 19.
16. Gorelick, N., Hancher, M., Dixon, M., Ilyushchenko, S., Thau, D., Moore, R., Google Earth engine: planetary-scale geospatial analysis for everyone. Remote Sens. Environ. 202 (2017), 18–27 Big Remotely Sensed Data: tools, applications and experiences http://www.sciencedirect.com/science/article/pii/S0034425717302900.
17. Hannachi, A., Jolliffe, I., Stephenson, D., Empirical orthogonal functions and related techniques in atmospheric science: a review. Int. J. Climatol. 27:9 (2007), 1119–1152.
18. Hansen, M.C., Potapov, P.V., Moore, R., Hancher, M., Turubanova, S., Tyukavina, A., Thau, D., Stehman, S., Goetz, S., Loveland, T., et al. High-resolution global maps of 21st-century forest cover change. Science 342:6160 (2013), 850–853.
19. Haynes, D., Manson, S., Shook, E., Terra Populus architecture for integrated big geospatial services. Trans. GIS 21:3 (2017), 546–559, 10.1111/tgis.12286.
20. Hoyer, S., Hamman, J., xarray: N-D labeled Arrays and Datasets in Python. J. Open Res. Software, 5(1), 2017.
21. Huffman, G.J., Bolvin, D.T., Nelkin, E.J., Wolff, D.B., Adler, R.F., Gu, G., Hong, Y., Bowman, K.P., Stocker, E.F., The TRMM multisatellite precipitation analysis (TMPA): quasi-global, multiyear, combined-sensor precipitation estimates at fine scales. J. Hydrometeorol. 8:1 (2007), 38–55.
22. Inglada, J., 2016. Is Google Earth Engine Evil? http://jordiinglada.net/wp/2016/05/12/is-google-earth-engine-evil-2 (accessed: 2016–11–29).
23. Kini, A., Emanuele, R., 2014. Geotrellis: Adding Geospatial Capabilities to Spark. Spark Summit.
24. Lewis, A., Oliver, S., Lymburner, L., Evans, B., Wyborn, L., Mueller, N., Raevksi, G., Hooke, J., Woodcock, R., Sixsmith, J., Wu, W., Tan, P., Li, F., Killough, B., Minchin, S., Roberts, D., Ayers, D., Bala, B., Dwyer, J., Dekker, A., Dhu, T., Hicks, A., Ip, A., Purss, M., Richards, C., Sagar, S., Trenham, C., Wang, P., Wang, L.-W., The Australian geoscience data cube foundations and lessons learned. Remote Sens. Environ. 202 (2017), 276–292 big Remotely Sensed Data: tools, applications and experience http://www.sciencedirect.com/science/article/pii/S0034425717301086.
25. Liu, H., 2014. Comparing NetCDF and a multidimensional array database on managing and querying large hydrologic datasets: a case study of SciDB. Master's thesis. TU Delft, Delft University of Technology.
26. Lu, M., Pebesma, E., Sanchez, A., Verbesselt, J., Spatio-temporal change detection from multidimensional arrays: detecting deforestation from MODIS time series. ISPRS J. Photogramm. Remote Sens. 117 (2016), 227–236.
27. Mahmud, M.R., Matsuyama, H., Hosaka, T., Numata, S., Hashim, M., Temporal downscaling of TRMM rain-rate images using principal component analysis during heavy tropical thunderstorm seasons. J. Hydrometeorol. 16:5 (2015), 2264–2275.
28. Mehta, P., Dorkenwald, S., Zhao, D., Kaftan, T., Cheung, A., Balazinska, M., Rokem, A., Connolly, A., Vanderplas, J., AlSayyad, Y., Comparative evaluation of Big-data systems on scientific image analytics workloads. Proc. VLDB Endow. 10:11 (2017), 1226–1237, 10.14778/3137628.3137634 Aug.
29. NASA, 2014. NEX Global Daily Downscaled Climate Projections. https://nex.nasa.gov/nex/projects/1356/ (accessed: 2016–11–29).
30. NASA, 2015. Product Description: TRMM_3B42_daily. http://mirador.gsfc.nasa.gov/collections/TRMM_3B42_daily__007.shtml (accessed: 2016–11–29).
31. Nativi, S., Mazzetti, P., Craglia, M., A view-based model of data-cube to support big earth data systems interoperability. Big Earth Data 1:1-2 (2017), 75–99, 10.1080/20964471.2017.1404232.
32. Paradigm4, Inc., 2016a. Run R programs within SciDB queries. https://github.com/Paradigm4/r_exec (accessed: 2016–11–29).
33. Paradigm4, Inc., 2016b. Very simple HTTP service for SciDB. https://github.com/Paradigm4/shim (accessed: 2016–11–29).
34. Pebesma, E., Wagner, W., Schramm, M., Von Beringe, A., Paulik, C., Neteler, M., Reiche, J., Verbesselt, J., Dries, J., Goor, E., Mistelbauer, T., Briese, C., Notarnicola, C., Monsorno, R., Marin, C., Jacob, A., Kempeneers, P., Soille, P., Nov. 2017. OpenEO - a Common, Open Source Interface Between Earth Observation Data Infrastructures and Front- End Applications.
35. Planthaber, G., Stonebraker, M., Frew, J., EarthDB: scalable analysis of MODIS data using SciDB. Proceedings of the 1st ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data, 2012, ACM, 11–19.
36. Rew, R., Davis, G., NetCDF: an interface for scientific data access. IEEE computer graphics and applications 10:4 (1990), 76–82.
37. Rocklin, M., 2015. Dask: Parallel Computation with Blocked algorithms and Task Scheduling. In: Huff, K., Bergstra, J. (Eds.), Proceedings of the 14th Python in Science Conference. pp. 130–136.
38. Roy, D.P., Wulder, M., Loveland, T., Woodcock, C., Allen, R., Anderson, M., Helder, D., Irons, J., Johnson, D., Kennedy, R., et al. Landsat-8: Science and product vision for terrestrial global change research. Remote Sens. Environ. 145 (2014), 154–172.
39. Scheider, S., Gräler, B., Pebesma, E., Stasch, C., Modeling spatiotemporal information generation. Int. J. Geogr. Inform. Sci. 30:10 (2016), 1980–2008, 10.1080/13658816.2016.1151520.
40. Schmidt, M., Lucas, R., Bunting, P., Verbesselt, J., Armston, J., Multi-resolution time series imagery for forest disturbance and regrowth monitoring in Queensland, Australia. Remote Sens. Environ. 158 (2015), 156–168.
41. Shvachko, K., Kuang, H., Radia, S., Chansler, R., The Hadoop Distributed File System. 26th Symposium on Mass Storage Systems and Technologies (MSST), 2010, IEEE, 1–10.
42. Stöckli, R., Vermote, E., Saleous, N., Simmon, R., Herring, D., 2005. The Blue Marble Next Generation-A true color earth dataset including seasonal dynamics from MODIS. Published by the NASA Earth Observatory.
43. Stonebraker, M., Brown, P., Zhang, D., Becla, J., SciDB: a database management system for applications with complex analytics. Comput. Sci. Eng. 15:3 (2013), 54–62 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6461866.
44. Taft, R., Vartak, M., Satish, N.R., Sundaram, N., Madden, S., Stonebraker, M., GenBase: a complex analytics genomics benchmark. Proceedings of the 2014 ACM SIGMOD international conference on Management of data, 2014, ACM, 177–188.
45. Tan, Z., Yue, P., Gong, J., An array database approach for earth observation data management and processing. ISPRS Int. J. Geo-Information, 6(7), 2017 http://www.mdpi.com/2220-9964/6/7/220.
46. Taylor, K.E., Stouffer, R.J., Meehl, G.A., An overview of CMIP5 and the experiment design. Bull. Am. Meteorol. Soc., 93(4), 2012, 485.
47. Verbesselt, J., Hyndman, R., Newnham, G., Culvenor, D., Detecting trend and seasonal changes in satellite image time series. Remote Sens. Environ. 114:1 (2010), 106–115.
48. Verbesselt, J., Zeileis, A., Herold, M., Near real-time disturbance detection using satellite image time series. Remote Sens. Environ. 123 (2012), 98–108.
49. Wagner, W., 2015. Big Data Infrastructures for Processing Sentinel Data. Photogrammetric Week 2015, pp. 93–104. http://www.ifp.uni-stuttgart.de/publications/phowo15/110Wagner.pdf.
50. Warmerdam, F., The Geospatial Data Abstraction Library. Open Source Approaches in Spatial Data Handling, 2008, Springer, 87–104.
51. Warmerdam, F., 2016. GDAL Utilities. http://gdal.org/gdal_utilities.html (accessed: 2016–11–29).
52. Watts, L.M., Laffan, S.W., Effectiveness of the BFAST algorithm for detecting vegetation response patterns in a semi-arid region. Remote Sens. Environ. 154 (2014), 234–245 http://www.sciencedirect.com/science/article/pii/S0034425714003204.
53. Weigel, T., Kindermann, S., Lautenschlager, M., 2015. Versioning for CMIP6 in the Earth System Grid Federation. In: EGU General Assembly Conference Abstracts. Vol. 17.
54. Xia, G.S., Hu, J., Hu, F., Shi, B., Bai, X., Zhong, Y., Zhang, L., Lu, X., AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification. IEEE Trans. Geosci. Remote Sens. 55:7 (2017), 3965–3981 July.
55. Zaharia, M., Xin, R.S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M.J., Ghodsi, A., Gonzalez, J., Shenker, S., Stoica, I., Apache spark: a unified engine for big data processing. Commun. ACM 59:11 (2016), 56–65 Oct http://doi.acm.org/10.1145/2934664.
56. Zhu, X.X., Tuia, D., Mou, L., Xia, G.S., Zhang, L., Xu, F., Fraundorfer, F., Deep learning in remote sensing: a comprehensive review and list of resources. IEEE Geosci. Remote Sens. Mag. 5:4 (2017), 8–36 December.