Forecasting with Big Data: A Review

Annals of Data Science

Volumn 2, Issue 1, 2015, Pages 5-19

  Hassani, Hossein ^a,b   Silva, Emmanuel Sirimal ^a  

^a BOURNEMOUTH UNIVERSITY   (United Kingdom)

^b International Institute for Energy Studies (IIES)   (Iran)

Author keywords

Big data; Forecasting; Statistics; Technique

Indexed keywords

BAYESIAN NETWORKS; BIG DATA; POPULATION STATISTICS; SIGNAL TO NOISE RATIO;

ALGORITHM ARCHITECTURES; BAYESIAN MODELLING; ECONOMIC ENERGY; FACTOR MODEL; HARDWARE ALGORITHM; HARDWARE AND SOFTWARE; MODERN AGES; SOFTWARE ALGORITHMS; STATISTICAL SIGNIFICANCE; TECHNIQUE;

FORECASTING;

EID: 84991460509     PISSN: 21985804     EISSN: 21985812     Source Type: Journal    
DOI: 10.1007/s40745-015-0029-9     Document Type: Review

References (96)

1. Alessi L, Barigozzi M, Capasso M (2009) Forecasting large datasets with conditionally heteroskedastic dynamic common factors. Working Paper No. 1115, European Central Bank
2. Altissimo F, Cristadoro R, Forni M, Lippi M, Veronese G (2010) New Eurocoin: tracking economic growth in real time. Rev Econ Stat 92(4):1024–1034 DOI: 10.1162/REST_a_00045
3. Arribas-Bel D (2014) Accidental, open and everywhere: emerging data sources for the understanding of cities. Appl Geogr 49:45–53 DOI: 10.1016/j.apgeog.2013.09.012
4. Bacon T (2013) Big bang? When ‘Big Data’ gets too Big. http://www.eyefortravel.com/mobile-and-technology/big-bang-when-%E2%80%98big-data%E2%80%99-gets-too-big. Accessed 16 May 2013
5. Bańbura M, Giannone D, Lenza M (2014) Conditional forecasts and scenario analysis with vector autoregressions for large cross-section. Working Papers ECARES ECARES 2014–2015, ULB - Universite Libre de Bruxelles
6. Bańbura M, Giannone D, Reichlin L (2010) Large bayesian vector autoregressions. J Appl Econom 25(1):71–92 DOI: 10.1002/jae.1137
7. Bańbura M, Modugno M (2014) Maximum likelihood sstimation of factor models on datasets with arbitrary pattern of missing data. J Appl Econom 29(1):133–160 DOI: 10.1002/jae.2306
8. Banerjee A, Marcellino M, Masten I (2013) Forecasting with factor-augmented error correction models. Int J Forecast (in Press)
9. Bernanke B, Boivin J, Eliasz PS (2005) Measuring the effects of monetary policy: a factor-augmented vector autoregressive approach. Quart J Econ 120(1):387–422
10. Bernstein D (2013) Big data’s greatest power: predictive analysis. http://www.equest.com/cartoons/cartoons-2013/big-datas-greatest-power-predictive-analytics/. Accessed 2 Dec 2013
11. Berry M (2000) Data mining techniques and algorithms. Wiley, New York
12. Biau O, D’Elia A (2009) Euro area GDP forecasting using large survey datasets. A random forest approach. http://unstats.un.org/unsd/nationalaccount/workshops/2010/moscow/AC223-S73Bk4.PDF. Accessed 27 June 2013
13. Bordoloi S, Biswas D, Singh S, Manna UK, Saggar S (2010) Macroeconomic forecasting using dynamic factor models. Reserv Bank India Occas Pap 31(2):69–83
14. Bounie D (2012) International production and dissemination information: results, methodological issues, and statistical perspectives. Int J Commun 6:1001–1021
15. Boyd D, Crawford K (2012) Critical questions for big data. Inf Commun Soc 15(5):662–679 DOI: 10.1080/1369118X.2012.678878
16. Brown B, Chui M, Manyika J (2011) Are you ready for the era of ‘big data’? In: Guzzo RA (ed) McKinsey quarterly. McKinsey & Company, New York
17. Camacho M, Sancho I (2003) Spanish diffusion indexes. Span Econ Rev 5(3):173–203 DOI: 10.1007/s10108-003-0062-2
18. Carriero A, Clark TE, Marcellino M (2012a) Real-time nowcasting with a Bayesian mixed frequency model with stochastic volatility. Working Paper, No. 1227, Federal Reserve Bank of Cleveland
19. Carriero A, Kapetanios G, Marcellino M (2012b) Forecasting government bond yields with large Bayesian vector autoregressions. J Bank Financ 36(7):2026–2047 DOI: 10.1016/j.jbankfin.2012.03.008
20. Carriero A, Kapetanios G, Marcellino M (2011) Forecasting large datasets with Bayesian reduced rank multivariate models. J Appl Econom 26(5):735–761 DOI: 10.1002/jae.1150
21. Choi H, Varian H (2012) Predicting the resent with google trends. Econ Rec 88(s1):2–9 DOI: 10.1111/j.1475-4932.2012.00809.x
22. Cukier K (2010) Data, data everywhere. The economist. http://www.economist.com/node/15557443. Accessed 17 Jul 2012
23. De Mol C, Giannone D, Reichlin L (2008) Forecasting using a large number of predictors: is Bayesian shrinkage a valid alternative to principal components? J Econom 146(2):318–328 DOI: 10.1016/j.jeconom.2008.08.011
24. Diebold FX (2003) ’Big Data’ Dynamic factor models for macroeconomic measurement and forecasting. In: Dewatripont M, Hansen LP, Turnovsky S (eds) Advances in economics and econometrics, eighth world congress of the econometric society. Cambridge University Press, Cambridge, pp 115–122
25. Doz C, Giannone D, Reichlin L (2012) A quasi-maximum likelihood approach for large, approximate dynamic factor models. Rev Econ Stat 99(4):1014–1024 DOI: 10.1162/REST_a_00225
26. Dumbill E (2012) What is big data? An introduction to the big data landscape. http://strata.oreilly.com/2012/01/what-is-big-data.html. Accessed 11 Jan 2012
27. Duhigg C (2012) How companies learn your secrets. http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html. Accessed 16 Feb 2012
28. Efron B (2010) Large-scale inference: empirical Bayes methods for sstimation, testing and prediction. Cambridge University Press, Cambridge DOI: 10.1017/CBO9780511761362
29. Einav L, Levin JD (2013) The data revolution and economic analysis. Working Paper No. 19035, National Bureau of Economic Research
30. Engle RF (2002) Dynamic conditional correlation: a simple class of multivariate generalized autoregressive conditional heteroskedasticity models. J Bus Econ Stat 20(3):339–350 DOI: 10.1198/073500102288618487
31. Engle RF, Kroner K (1995) Multivariate simultaneous GARCH. Econom Theory 11(1):122–150 DOI: 10.1017/S0266466600009063
32. Figueiredo FMR (2010) Forecasting Brazilian inflation using a large dataset. Working Paper No. 228, Central Bank of Brazil. http://www.bcb.gov.br/pec/wps/ingl/wps228.pdf
33. Fischer U, Schildt C, Hartmann C, Lehner W (2013) Forecasting the data cube: A model configuration advisor for multi-dimensional data sets. In: IEEE 29th International conference on data engineering (ICDE), Brisbane, 8–12 April 2013
34. Forni M, Hallin M, Lippi M, Reichlin L (2000) The generalized factor model: identification and estimation. Rev Econ Stat 82(4):540–554 DOI: 10.1162/003465300559037
35. Forni M, Hallin M, Lippi M, Reichlin L (2005) The generalized dynamic factor model: one-sided estimation and forecasting. J Am Stat Assoc 100(471):830–840 DOI: 10.1198/016214504000002050
36. Frutos S, Menasalva E, Montes C, Segovia J (2003) Calculating economic indexes per household and censal section from official Spanish databases. Intellt Data Anal 7(6):603–613
37. Gantz J, Reinsel D (2012) The digital universe in 2020: big data, bigger digital shadows, and biggest growth in far east. http://www.emc.com/collateral/analyst-reports/idc-the-digital-universe-in-2020.pdf. Accessed 5 Jan 2014
38. Garber SC (2009) Census mail list trimming using SAS data mining. In: RRD Research Report, Washington, DC, 9 May 2009
39. Ghodsi M (2014) A brief review of data mining applications in the energy industry. Int J Energy Stat 2(1):49–57
40. Gilary A (2011) Recursive partitioning for racial classification cells. In: Proceedings of the survey research methods section, American Statistical Association-Session 628: survey analysis and issues with ata quality 2011, Miami Beach, p 2706
41. Giovanelli A (2012) Nonlinear forecasting using large datasets: evidence on US and Euro area economies. CEIS Tor Vergata, Research paper series, 10(13). doi:10.2139/ssrn.2172399 DOI: 10.2139/ssrn.2172399
42. Gupta R, Kabundi A, Miller S, Uwilingiye J (2013) Using large datasets to forecast sectoral unemployment. Stat Methods Appl 23(2):229–264 DOI: 10.1007/s10260-013-0243-6
43. Gursun G, Crovella M, Matta I (2011) Describing and forecasting video access patterns. In: INFOCOM ’11, Proceedings of the 30th IEEE international conference on computer communications, IEEE, 2011. http://www.cs.bu.edu/techreports/pdf/2010-037-video-access-patterns.pdf. Accessed 10 Nov 2010
44. Hamm S (2013) How big data can boost weather forecasting. http://readwrite.com/2013/02/28/how-big-data-can-boost-weather-forecasting#awesm=ou64ZEaKe2HtUu. Accessed 20 Nov 2014
45. Han J, Kamber M, Pie J (2012) Data mining: concepts and techniques. Elsevier, Inc., San Francisco DOI: 10.1007/978-1-4419-1428-6_3752
46. Hand DJ (2009) Mining the past to determine the future: problems and possibilities. Int J Forecast 25(3):441–451 DOI: 10.1016/j.ijforecast.2008.09.004
47. Hassani H, Heravi S, Zhigljavsky A (2009) Forecasting European industrial production with singular spectrum analysis. Int J Forecast 25(1):103–118 DOI: 10.1016/j.ijforecast.2008.09.007
48. Hassani H, Heravi S, Zhigljavsky A (2013) Forecasting UK industrial production with multivariate singular spectrum analysis. J Forecast 32(5):395–408 DOI: 10.1002/for.2244
49. Hassani H, Saporta G, Silva ES (2014) Data mining and official statistics: the past, the present & the future. Big Data 2(1):BD1–BD10 DOI: 10.1089/big.2013.0038
50. Hassani H, Webster A, Silva ES, Heravi H (2015) Forecasting US tourist arrivals using optimal singular spectrum analysis. Tour Manag 46:322–335
51. Hyndman RJ, Athanasopoulos G (2013) Forecasting: principles and practice. Otexts, Melbourne
52. Jadhav DK (2013) Big data: the new challenges in data mining. Int J Innov Res ComputSci & Technol 1(2):39–42
53. Kansara VA (2011) The long view | how ealtime data is reshaping the fashion business. http://www.businessoffashion.com/2011/08/the-long-view-how-realtime-data-is-reshaping-the-fashion-business.html. Accessed 22 May 2014
54. Kapetanios G, Marcellino M (2009) A parametric estimation method for dynamic factor models of large dimensions. J Time Ser Anal 30(2):208–238 DOI: 10.1111/j.1467-9892.2009.00607.x
55. Knapp A (2013) Forecasting the weather with big data and the fourth dimension. http://www.forbes.com/sites/alexknapp/2013/06/13/forecasting-the-weather-with-big-data-and-the-fourth-dimension/2/. Accessed 20 Nov 2014
56. Koop GM (2013) Forecasting with medium and large Bayesian VARs. J Appl Econom 28(2):177–203 DOI: 10.1002/jae.1270
57. Kopoin A, Moran K, Paré JP (2013) Forecasting regional GDP with factor models: how useful are national and international data? Econ Lett 121(2):267–270 DOI: 10.1016/j.econlet.2013.08.007
58. Kurgan L, Musilek P (2006) A survey of knowledge discover and data mining process models. Knowl Eng Rev 21(1):1–24 DOI: 10.1017/S0269888906000737
59. Laney D (2001) 3D Data management: controlling data volume, velocity and variety. http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf. Accessed 13 Dec 2013
60. Lohr S (2013) The age of big data. http://www.nytimes.com/2012/02/12/sunday-review/big-datas-impact-in-the-world.html?pagewanted=all&_r=1&. Accessed 17 Sep 2012
61. Lutz RW, Buhlmann P (2006) Boosting for high multivariate responses in high-dimensional linear regression. Stat Sin 16:471–494
62. Madden S (2012) From databases to big data. IEEE Internet Comput 16(3):4–6 DOI: 10.1109/MIC.2012.50
63. Manyika J, Chui, M, Brown B, Bughin J, Dobbs R, Roxburgh C, Byers AH (2011) Big data: the next frontier for innovation, competition, and productivity. McKinsey Global Institute. http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation. Accessed 20 Nov 2014
64. Marz N, Warren J (2013) Big Data: principles and best practices of scalable reatime data systems. Manning Publications, St. Louis
65. McCarthy JS, Jacob T, Atkinson D (2009) Innovative uses of data mining techniques in the production of official statistics. In: UN statistical commission session on innovations in official statistics, New York, 20 Feb 2009
66. McCarthy J, Jacob T, McCracken A (2010) Modeling NASS survey non-response using classification trees. In: RDD Research Report, Washington DC, 1 Nov 2010
67. Needham J (2013) Disruptive possibilities: how big data changes everything. O’Reilly media. Available via: http://chimera.labs.oreilly.com/books/1234000000914/index.html. Accessed 25 Jul 2012
68. Nordbotten S (1996) Neural network imputation applied to the Norwegian 1990 population census data. J Off Stat 12(4):385–401
69. Nguyen HT, Nabney IT (2010) Short-term electricity demand and gas price forecasts using wavelet transforms and adaptive models. Energy 35(9):3674–3685 DOI: 10.1016/j.energy.2010.05.013
70. Ouysse R (2013) Forecasting using a large number of predictors: Bayesian model averaging versus principal components regression. Australian School of Business Research Paper, No. 2013ECON04, pp 1–34. http://research.economics.unsw.edu.au/RePEc/papers/2013-04.pdf. Accessed 29 May 2013
71. Paaß G, Kindermann J (2003) Bayesian regression mixtures of experts for geo-referenced data. Intell Data Anal 7(6):567–582
72. Poynter R (2013) Big data successes and limitations: what researchers and marketers need to know. http://www.visioncritical.com/blog/big-data-successes-and-limitations. Accessed 20 Nov 2014
73. Press G (2013) A very short history of big data. http://www.forbes.com/sites/gilpress/2013/05/09/a-very-short-history-of-big-data/2/. Accessed 28 Jul 2013
74. Pyle D (2003) Business modeling and data mining. Elsevier Science, Philadelphia
75. Rey T, Wells C (2013) Integrating data mining and forecasting. OR/MS Today, New York
76. Richards NM, King JH (2013) Three paradoxes of big data. Stanf Law Rev Online 66(41):41–46
77. Schumacher C (2007) Forecasting German GDP using alternative factor odels based on large datasets. J Forecast 26(4):271–302 DOI: 10.1002/for.1026
78. Schumacher C, Breitung J (2008) Real-time forecasting of German GDP based on a large factor model with monthly and quarterly data. Int J Forecast 24(3):386–398 DOI: 10.1016/j.ijforecast.2008.03.008
79. Shi Y (2014) Big Data: history, current status, and challenges going forward. Bridge 44(4):6–11
80. Sigrist F, Kunsch HR, Stahel WA (2012) SPDE based modeling of large space-time data sets. http://arxiv.org/pdf/1204.6118v4.pdf. Accessed 20 Nov 2014
81. Silva ES (2013) A combination forecast for energy-related CO2 emissions in the United States. Int J Energy Stat 1(4):269–279 DOI: 10.1142/S2335680413500191
82. Silva ES, Hassani H (2015) On the use of singular spectrum analysis for forecasting US trade before, during and after the 2008 recession. Int Econ. doi:10.1016/j.inteco.2014.11.003 DOI: 10.1016/j.inteco.2014.11.003
83. Silva ES, Wu Y, Ojiako U (2013) Developing risk management as a competitive capability. Strateg Change 22(5–6):281–294 DOI: 10.1002/jsc.1940
84. Silver N (2012) The signal and the noise: the art and science of prediction. Penguin Books, Westmins
85. Skupin A, Agarwal P (2007) Introduction: what is a self-organizing map? In: Agarwal P, Skupin A (eds) Self-organizing maps: applications in geographic information science. John Wiley, Chichester
86. Smolan R, Erwitt J (2012) The human face of big data. Sterling Publishing, New York
87. Stock JH, Watson MW (2002) Forecasting using principal components from a large number of predictors. J Am Stat Assoc 97(460):1167–1179 DOI: 10.1198/016214502388618960
88. Stock JH, Watson MW (2006) Forecasting with many predictors. In: Elliott G, Granger CWJ, Timmermann A (eds) Handbook of economic forecasting. Elsevier, Amsterdam, pp 517–554
89. Stock JH, Watson MW (2008) Forecasting in dynamic factor models subject to structural instability. In: Hendry David F, Castle J, Shephard N (eds) The methodology and practice of econometrics: a festschrift in honour of professor. Oxford University Press, Oxford, pp 173–205
90. Thornton D (2013) The Problem with Big Data. http://moneyweek.com/arria-nlg-the-problem-with-big-data/. Accessed 25 Nov 2013
91. Tucker P (2013) The future is not a destination. http://www.slate.com/articles/technology/future_tense/2013/10/futurist_magazine_s_predictions_on_quantum_computing_big_data_and_more.html. Accessed 2 Oct 2013
92. Varian HR (2014) Big data: new tricks for econometrics. J Econ Perspect 28(2):3–28 DOI: 10.1257/jep.28.2.3
93. Wang X (2013) Electricity consumption forecasting in the age of big data. Telkomnika 11(9):5262–5266
94. Walker A (2014) Trends in big data: a forecast for 2014. http://www.csc.com/big_data/publications/91710/105057-trends_in_big_data_a_forecast_for_2014
95. West G (2013) Big data needs a big theory to go with it. http://www.scientificamerican.com/article/big-data-needs-big-theory/. Accessed 31 May 2013
96. Wu S, Kang N, Yang L (2007) Fraudulent behaviour forecast in telecom industry based on data mining technology. Commun IIMA 7(4):1–6