A user-friendly tool to transform large scale administrative data into wide table format using a mapreduce program with a pig latin based script

BMC Medical Informatics and Decision Making

Volumn 12, Issue 1, 2012, Pages

  Horiguchi, Hiromasa ^a   Yasunaga, Hideo ^a   Hashimoto, Hideki ^b   Ohe, Kazuhiko ^a  

^a UNIVERSITY OF TOKYO   (Japan)

^b UNIVERSITY OF TOKYO   (Japan)

Author keywords

Large scale administrative data; MapReduce; Pig Latin; User defined functions

Indexed keywords

ARTICLE; COMPUTER INTERFACE; COMPUTER PROGRAM; INFORMATION PROCESSING; INFORMATION SYSTEM; METHODOLOGY; QUALITY CONTROL;

AUTOMATIC DATA PROCESSING; BENCHMARKING; DATA DISPLAY; INFORMATION MANAGEMENT; SOFTWARE DESIGN; USER-COMPUTER INTERFACE;

MLCS; MLOWN;

EID: 84871397691     PISSN: None     EISSN: 14726947     Source Type: Journal    
DOI: 10.1186/1472-6947-12-151     Document Type: Article

References (20)

1. With great data comes great responsibility: publishing comparative effectiveness research in epidemiology. Hernán MA, Epidemiology 2011 22 290 291 10.1097/EDE.0b013e3182114039 21464646
2. The new world of data linkages in clinical epidemiology: are we being brave or foolhardy? Weiss NS, Epidemiology 2011 22 292 294 10.1097/EDE. 0b013e318210aca5 21464647
3. Making observational studies count: shaping the future of comparative effectiveness research. Dreyer NA, Epidemiology 2011 22 295 297 10.1097/EDE.0b013e3182126569 21464648
4. Nonexperimental comparative effectiveness research using linked healthcare databases. Stürmer T, Jonsson FM, Poole C, Brookhart MA, Epidemiology 2011 22 298 301 10.1097/EDE.0b013e318212640c 21464649
5. Improving automated database studies. Ray WA, Epidemiology 2011 22 302 304 10.1097/EDE.0b013e31820f31e1 21464650
6. Development and use of the Japanese case-mix system. Matsuda S, Ishikawa KB, Kuwabara K, Fujimori K, Fushimi K, Hashimoto H, Eurohealth 2008 14 25 30
7. Variation in cancer surgical outcomes associated with physician and nurse staffing: a retrospective observational study using the Japanese Diagnosis Procedure Combination Database. Yasunaga H, Hashimoto H, Horiguchi H, Miyata S, Matsuda S, BMC Health Serv Res 2012 12 129 10.1186/1472-6963-12-129 22640411
8. Prevalence of malignant hyperthermia and relationship with anesthetics in Japan: data from the Diagnosis Procedure Combination Database. Sumitani M, Uchida K, Yasunaga H, Horiguchi H, Kusakabe Y, Matsuda S, Yamada Y, Anesthesiology 2011 114 84 90 10.1097/ALN.0b013e318200197d 21169796
9. Risk factors of pulmonary embolism and the effects of fondaparinux after total hip and knee arthroplasty: a retrospective observational study using a national database in Japan. Nagase Y, Yasunaga H, Horiguchi H, Hashimoto H, Shoda N, Kadono Y, Matsuda S, Nakamura K, Tanaka S, J Bone Joint Surg Am 2011 92 146 (1 7
10. Incidence of severe adverse events requiring hospital care after trastuzumab infusion for metastatic breast cancer: a nationwide survey using an administrative claim database. Horiguchi H, Yasunaga H, Hashimoto H, Matsuda S, Breast J 2011 17 683 685 10.1111/j.1524-4741.2011.01170.x 21973020
11. Impact of hospital volume and laser usage on postoperative complications and in-hospital mortality after transurethral surgery of benign prostate hyperplasia: Japanese Diagnosis Procedure Combination Database. Sugihara H, Yasunaga H, Horiguchi H, Nishimatsu H, Kume H, Matsuda S, Homma Y, J Urol 2011 185 2248 2253 10.1016/j.juro.2011.01.080 21497849
12. Quantitative assessment of the advantages of laparoscopic gastrectomy and the impact of volume-related hospital characteristics on resource use and outcomes of gastrectomy patients in Japan. Kuwabara K, Matsuda S, Fushimi K, Ishikawa KB, Horiguchi H, Fujimori K, Yasunaga H, Miyata H, Ann Surg 2011 253 64 70 10.1097/SLA.0b013e318204e524 21116173
13. ISO/IEC 9075-*, Database languages SQL Geneva, Switzerland: ISO 2003
14. The Apache™ Hadoop™ project http:// hadoop.apache.org/ 23307825
15. MapReduce: a flexible data processing tool. Dean J, Ghemawat S, Commun ACM 2010 53 72 77
16. An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics. Taylor RC, BMC Bioinforma 2010 11 Suppl 12 1 10.1186/1471-2105-11-S12-S1
17. Pig-Apache Software Foundation project home page http://pig.apache.org/
18. MapReduce: simplified data processing on large clusters. Dean J, Ghemawat S, Commun ACM 2008 57 107 113
19. PVM: a framework for parallel distributed computing. Sunderam VS, Concurrency: Practice and Experience 1990 2 315 339 10.1002/cpe.4330020404
20. ACM SIGMOD International Conference on Management of Data: 9-12 June 2008; Vancouver. Olston C, Reed B, Srivastava U, Kumar R, Tomkins A: Pig Latin: a not-so-foreign language for data processing. In Proceedings of the, ACM; 2008 2008 1099 1110 23305810