Hyperion: High Volume Stream Archival for Retrospective Querying

USENIX 2007 - 2007 USENIX Annual Technical Conference

Volumn , Issue , 2007, Pages 45-58

  Desnoyers, Peter J ^a   Shenoy, Prashant ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA STREAMS; DIGITAL STORAGE; INDEXING (OF INFORMATION); ONLINE SYSTEMS; QUERY PROCESSING;

AFTER-THE-FACT; ANOMALY DETECTION; DATA ARCHIVING; DATA ARRIVAL RATE; DATA STREAM; FILESYSTEM; HIGH VOLUMES; HYPERION; NETWORK MONITORING SYSTEMS; PACKETS PER SECONDS;

ANOMALY DETECTION;

EID: 79960804909     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. ABADI, D. J., AHMAD, Y., BALAZINSKA, M., ÇETINTEMEL, U., CHERNIACK, M., HWANG, J.-H., LINDNER, W., MASKEY, A. S., RASIN, A., RYVKINA, E., TATBUL, N., XING, Y., AND ZDONI, S. The Design of the Borealis Stream Processing Engine. In Proc. Conf. on Innovative Data Systems Research (Jan. 2005).
2. BERNSTEIN, D. Syn cookies. Published at http://cr.yp.to/syncookies.html.
3. BLOOM, B. Space/time tradeoffs in hash coding with allowable errors. Communications of the ACM 13, 7 (July 1970), 422-426.
4. CHAN, C.-Y., AND IOANNIDIS, Y. E. Bitmap index design and evaluation. In Proc. ACM SIGMOD Intl. Conf. on Management of Data (June 1998), pp. 355-366.
5. CHERITON, D. The V Distributed System. Communications of the ACM 31, 3 (Mar. 1988), 314-333.
6. CRANOR, C., JOHNSON, T., SPATASCHEK, O., AND SHKAPENYUK, V. Gigascope: a stream database for network applications. In Proc. 2003 ACM SIGMOD Intl. Conf. on Management of data (2003), pp. 647-651.
7. DESNOYERS, P., GANESAN, D., AND SHENOY, P. TSAR: a two tier sensor storage architecture using interval skip graphs. In Proc. 3rd Intl. Conf on Embedded networked sensor systems (SenSys05) (2005), pp. 39-50.
8. DESNOYERS, P., AND SHENOY, P. Hyperion: High Volume Stream Archival for Retrospective Querying. Tech. Rep. TR46-06, University of Massachusetts, Sept. 2006.
9. DREGER, H., FELDMANN, A., PAXSON, V., AND SOMMER, R. Operational experiences with high-volume network intrusion detection. In Proc. 11th ACM Conf. on Computer and communications security (CCS-04) (2004), pp. 2-11.
10. ENDACE INC. Endace DAG4.3GE Network Monitoring Card. available at http://www.endace.com, 2006.
11. FALOUTSOS, C. Signature-Based Text Retrieval Methods: A Survey. IEEE Data Engineering Bulletin 13, 1 (1990), 25-32.
12. FALOUTSOS, C., AND CHAN, R. Fast Text Access Methods for Optical and Large Magnetic Disks: Designs and Performance Comparison. In VLDB-88: Proc. 14th Intl. Conf. on Very Large Data Bases (1988), pp. 280-293.
13. FALOUTSOS, C., AND CHRISTODOULAKIS, S. Signature files: an access method for documents and its analytical performance evaluation. ACM Trans. Inf. Syst. 2, 4 (1984), 267-288.
14. GANESAN, D., ESTRIN, D., AND HEIDEMANN, J. Dimensions: Distributed multi-resolution storage and mining of networked sensors. ACM Computer Communication Review 33, 1 (January 2003), 143-148.
15. HUEBSCH, R., CHUN, B., HELLERSTEIN, J. M., LOO, B. T., MANIATIS, P., ROSCOE, T., SHENKER, S., STOICA, I., AND YUMEREFENDI, A. R. The Architecture of PIER: an Internet-Scale Query Processor. In Proc. Conf. on Innovative Data Systems Research (CIDR) (Jan. 2005).
16. IANNACCONE, G., DIOT, C., MCAULEY, D., MOORE, A., PRATT, I., AND RIZZO, L. The CoMo White Paper. Tech. Rep. IRC-TR-04-17, Intel Research, Sept. 2004.
17. KARP, R. M., SHENKER, S., AND PAPADIMITRIOU, C. H. A simple algorithm for finding frequent elements in streams and bags. ACM Trans. Database Syst. 28, 1 (2003), 51-55.
18. LI, X., BIAN, F., ZHANG, H., DIOT, C., GOVINDAN, R., HONG, W., AND IANNACCONE, G. Advanced Indexing Techniques for Wide-Area Network Monitoring. In Proc. 1st IEEE Intl. Workshop on Networking Meets Databases (NetDB) (2005).
19. MOORE, A., HALL, J., KREIBICH, C., HARRIS, E., AND PRATT, I. Architecture of a Network Monitor. Passive & Active Measurement Workshop 2003 (PAM2003) (2003).
20. MOTWANI, R., WIDOM, J., ARASU, A., BABCOCK, B., BABU, S., DATAR, M., MAKU, G., OLSTON, C., ROSENSTEIN, J., AND VARMA, R. Query processing, approximation, and resource management in a data stream management system. In Proc. Conf. on Innovative Data Systems Research (CIDR) (2003).
21. NDIAYE, B., NIE, X., PATHAK, U., AND SUSAIRAJ, M. A Quantitative Comparison between Raw Devices and File Systems for implementing Oracle Databases. http://www.oracle.com/technology/deploy/performance/WhitePapers.html, Apr. 2004.
22. ROSENBLUM, M., AND OUSTERHOUT, J. K. The design and implementation of a log-structured file system. ACM Transactions on Computer Systems 10, 1 (1992), 26-52.
23. SACKS-DAVIS, R., AND RAMAMOHANARAO, K. A two level superimposed coding scheme for partial match retrieval. Information Systems 8, 4 (1983), 273-289.
24. SELTZER, M. I., SMITH, K. A., BALAKRISHNAN, H., CHANG, J., MCMAINS, S., AND PADMANABHAN, V. N. File System Logging versus Clustering: A Performance Comparison. In USENIX Winter Technical Conference (1995), pp. 249-264.
25. SHRIVER, E., GABBER, E., HUANG, L., AND STEIN, C. A. Storage management for web proxies. In Proc. USENIX Annual Technical Conference (2001), pp. 203-216.
26. STOCKINGER, K. Design and Implementation of Bitmap Indices for Scientific Data. In Intl. Database Engineering & Applications Symposium (July 2001).
27. STONEBRAKER, M., CETINTEMEL, U., AND ZDONIK, S. The 8 requirements of real-time stream processing. SIGMOD Record 34, 4 (2005), 42-47.
28. STREAMBASE, I. StreamBase: Real-Time, Low Latency Data Processing with a Stream Processing Engine. from http://www.streambase.com, 2006.
29. SWEENEY, A., DOUCETTE, D., HU, W., ANDERSON, C., NISHIMOTO, M., AND PECK, G. Scalability in the XFS File System. In USENIX Annual Technical Conference (Jan. 1996).
30. TOBAGI, F. A., PANG, J., BAIRD, R., AND GANG, M. Streaming RAID: a disk array management system for video files. In Proc. 1st ACM Intl. Conf. on Multimedia (1993), pp. 393-400.
31. Umass trace repository. Available at http://traces.cs.umass.edu.
32. WANG, W., ZHAO, Y., AND BUNT, R. HyLog: A High Performance Approach to Managing Disk Layout. In Proc. 3rd USENIX Conf. on File and Storage Technologies (FAST) (2004), pp. 145-158.