A parallel file system with application-aware data layout policies for massive remote sensing image processing in digital earth

IEEE Transactions on Parallel and Distributed Systems

Volumn 26, Issue 6, 2015, Pages 1497-1508

  Wang, Lizhe ^a   Ma, Yan ^a   Zomaya, Albert Y ^b   Ranjan, Rajiv ^c   Chen, Dan ^d  

^a INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

^b UNIVERSITY OF SYDNEY   (Australia)

^c CSIRO   (Australia)

^d WUHAN UNIVERSITY   (China)

Author keywords

big data; data intensive computing; digital earth; Parallel file system; parallel I O; remote sensing image processing

Indexed keywords

FILE ORGANIZATION; IMAGE PROCESSING; IMAGE RECONSTRUCTION; REMOTE SENSING; SPACE OPTICS;

DATA-INTENSIVE COMPUTING; DIGITAL EARTH; PARALLEL FILE SYSTEM; PARALLEL I/O; REMOTE SENSING IMAGE PROCESSING;

BIG DATA;

EID: 84929334224     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2014.2322362     Document Type: Article

References (41)

1. H. Guo, X. Fan, and C. Wang, "A digital earth prototype system: DEPS/CAS," Int. J. Digital Earth, vol. 2, no. 1, pp. 3-15, 2009.
2. A. J. Plaza and C.-I. Chang (eds). High Performance Computing in Remote Sensing. Chapman & Hall/CRC, 2007. ISBN: 1584886625
3. Y. Liu, B. Chen, H. Yu, Y. Zhao, Z. Huang, and Y. Fang, "Applying GPU and POSIX thread technologies in massive remote sensing image data processing," in Proc. IEEE Int. Conf. Geoinformat., 2011, pp. 1-6.
4. A. Rosenqvist, M. Shimada, B. Chapman, A. Freeman, G. De Grandi, S. Saatchi, and Y. Rauste, "The global rain forest mapping project - A review," Int. J. Remote Sensing, vol. 21, nos. 6/7, pp. 1375-1387, 2000.
5. R. Xu, M. Araya-Polo, and B. Chapman, "Filesystem aware scalable I/O framework for data-intensive parallel applications," in Proc. IEEE 27th Int. Symp. Parallel Distrib. Process. Workshops PhD Forum, 2013, pp. 2007-2014.
6. C. Yong, X.-H. Sun, R. Thakur, H. Song, and H. Jin, "Improving parallel I/O performance with data layout awareness," in Proc. IEEE Int. Conf. Cluster Comput., Sep. 2010, pp. 302-311.
7. J. M. May, Parallel I/O for High Performance Computing. San Francisco, CA, USA: Morgan Kaufmann, 2001.
8. W. Joachim, J. Larsson Traff, and H. Ritzdorf, "Fast parallel noncontiguous file access," in Proc. ACM/IEEE Conf. Supercomput., 2003, p. 60.
9. J. L. Träff, R. Hempel, H. Ritzdorf, and F. Zimmermann, "Flattening on the fly: Efficient handling of MPI derived datatypes," Berlin, Germany, Springer-Verlag, 1999, pp. 109-116.
10. H. Song et al., "A segment-level adaptive data layout scheme for improved load balance in parallel file systems," in Proc. 11th IEEE/ACMInt. Symp. Cluster, Cloud Grid Comput., 2011, pp. 414-423.
11. M. Moore, D. Bonnie, B. Ligon, M. Marshall, W. Ligon, N. Mills, E. Quarles, and S. Sampson, "OrangeFS: Advancing PVFS," FAST poster session, 2011.
12. OrangeFS: Orange file system project. [Online]. Available: http://www.orangefs.org, 2014.
13. R. Thakur, W. Gropp, and E. Lusk, "On implementing MPI-IO portably and with high performance," in Proc. 6th Workshop I/O Parallel Distrib. Syst., 1999, pp. 23-32.
14. J. Wu, P. Wyckoff, and D. Panda, "Supporting efficient noncontiguous access in PVFS over InfiniBand," in Proc. IEEE Int. Conf. Cluster Comput., 2003, pp. 344-351.
15. B. Furht and A. Escalante, Handbook of Data Intensive Computing. Berlin, Germany: Springer, 2011.
16. I. F. Haddad, "PVFS: A parallel virtual file system for linux clusters," Linux J., no. 80, p. 5, 2000.
17. "High-performance storage architecture and scalable cluster file system,"White Paper, Dec. 2007.
18. D. Nagle, D. Serenyi, and A. Matthews, "The panasas activescale storage cluster: Delivering scalable high bandwidth storage," in Proc. ACM/IEEE Conf. Supercomput., 2004, p. 53.
19. F. Schmuck and R. Haskin, "GPFS: A shared-disk file system for large computing clusters," in Proc. 1st USENIX Conf. File Storage Technol., vol. 2, 2002, p. 19.
20. R. Jain, P. Sarkar, and D. Subhraveti, "Gpfs-SNC: An enterprise cluster file system for big data[J]," IBM J. Res. Dev., vol. 57, no. 3/4, pp. 5:1-5:10, 2013.
21. N. Nieuwejaar, D. Kotz, A. Purakayastha, C. S. Ellis, and M. Best, "File-access characteristics of parallel scientific workloads," IEEE Tran. Parallel Distrib. Syst., vol. 7, no. 10, pp. 1075-1089, Oct. 1996.
22. S. Narayan and J. A. Chandy, "I/O characterization on a parallel file system," in Proc. Int. Symp. Perform. Eval. Comput. Telecommun. Syst., Jul. 2010, pp. 133-140.
23. R. Thakur, W. Gropp, and E. Lusk, "Data sieving and collective I/O in ROMIO," in Proc. 7th Symp. Frontiers Massively Parallel Comput., Feb. 1999, pp. 182-189.
24. Y. Lu, Y. Chen, P. Amritkar, R. Thakur, and Y. Zhuang, "A new data sieving approach for high performance I/O," in Future Information Technology, Application, and Service, Berlin, Germany: Springer, 2012, pp. 111-121.
25. R. Thakur and A. Choudhary, "An extended two-phase method for accessing sections of out-of-core arrays," Sci. Program., vol. 5, no. 4, pp. 301-317, 1996.
26. P. M. Dickens and R. Thakur, "Evaluation of collective I/O implementations on parallel architectures," J. Parallel Distrib. Comput., vol. 61, no. 8, pp. 1052-1076, 2001.
27. R. Koller and R. Rangaswami, "I/O deduplication: Utilizing content similarity to improve I/O performance," ACM Tran. Storage, vol. 6, no. 3, p. 13, Sep. 2010.
28. J. Zhang and P. Honeyman, "Replication control in distributed file systems," Center for Inf. Technol. Integration, University of Michigan, Ann Arbor, MI, USA, CITI Technical Report 04-01, 2004.
29. Y. Ma, L. Zhao, and D. Liu, "An asynchronous parallelized and scalable image resampling algorithm with parallel I/O," in Proc. Int. Conf. Comput. Sci., Berlin Heidelberg, 2009, pp. 357-366.
30. K. Huang, G. Li, D. Liu, and W. Zhang, "A parallel file system based on spatial information object," in Network and Parallel Computing, Berlin, Germany: Springer, 2005, pp. 153-162.
31. Y. Zhanwu, Z. Li, and S. Zheng, "An object-based storage model for distributed remote sensing images," in Proc. GNSS Integr. Geospatial Appl. Int. Soc. Opt. Photon. 2006, p. 64180A.
32. M. Rafique, A. Mustafa, R. Butt, and D. S. Nikolopoulos, "Dmabased prefetching for I/O-intensive workloads on the cell architecture," in Proc. 5th Conf. Comput. Frontiers, 2008, pp. 23-32.
33. Y. Wang, Y. Ma, P. Liu, D. Liu, and J. Xie, "An optimized image mosaic algorithm with parallel io and dynamic grouped parallel strategy based on minimal spanning tree," in Proc. Int. Conf. Grid Cooperative Comput., 2010, pp. 501-506.
34. J. K. Lawder, "Calculation of mappings between one and ndimensional values using the Hilbert space-filling curve," School of Comput. Sci. Inf. Syst., Birkbeck College, University of London, London Research Report BBKCS-00-01, Aug. 2000.
35. X. Sun H, Y. Chen, and Y. Yin, "Data layout optimization for petascale file systems," in Proc. 4th Annu. Workshop Petascale Data Storage, 2009, pp. 11-15.
36. H. Song, H. Jin, J. He, X.-H. Sun, and R. Thakur, "A server-level adaptive data layout strategy for parallel file systems," in Proc. IEEE 26th Int. Conf. Parallel Distrib. Process. Symp. Workshops, PhD Forum, 2012, pp. 2095-2103.
37. B. Moon, H.V. Jagadish, C. Faloutsos, and J. H. Saltz, "Analysis of the clustering properties of the Hilbert space-filling curve," IEEE Trans. Knowledge Data Eng., vol. 13, no. 1, pp. 124-141, Feb. 2001.
38. J. Lawder, "The application of space-filling curves to the storage and retrieval of multi-dimensional data," Birkbeck College, University of London, London, U.K., Tech. Rep. JL/1/99, 1999.
39. H. Shan and J. Shalf, "Using IOR to Analyze the I/O performance for HPC Platforms," presented at the Cray User Group Conf., Seattle, WA, USA, 2007.
40. Y. Xue, D. Palmer-Brown, and H. Guo, "The use of highperformance and high-throughput computing for the fertilization of digital earth and global change studies," pp. 185-210.
41. H. Guo, "Digital Earth: a new challenge and new vision," Int. J. Digital Earth, vol. 5, no. 1, pp. 1-3, 2012.