Exploration versus exploitation in space, mind, and society

Trends in Cognitive Sciences

Volumn 19, Issue 1, 2015, Pages 46-54

  Hills, Thomas T ^a   Todd, Peter M ^b   Lazer, David ^c,d,e   Redish, A David ^f   Couzin, Iain D ^g,h   Bateson, Melissa ⁱ   Cools, Roshan ^j,k   Dukas, Reuven ^l   Giraldeau, Luc Alain ^m   Macy, Michael ⁿ   Page, Scott E ^o   Shiffrin, Richard M ^b   Stephens, David W ^f   Wolfe, Jeremy W ^p  

^a UNIVERSITY OF WARWICK   (United Kingdom)

^b INDIANA UNIVERSITY   (United States)

^c NORTHEASTERN UNIVERSITY   (United States)

^d Northeastern University   (United States)

^e HARVARD UNIVERSITY   (United States)

^f UNIVERSITY OF MINNESOTA   (United States)

^g PRINCETON UNIVERSITY   (United States)

^h MAX PLANCK INSTITUTE FOR ORNITHOLOGY   (Germany)

ⁱ NEWCASTLE UNIVERSITY   (United Kingdom)

^j RADBOUD UNIVERSITY MEDICAL CENTER   (Netherlands)

^k RADBOUD UNIVERSITY   (Netherlands)

^l MCMASTER UNIVERSITY   (Canada)

^m UNIVERSITÉ DU QUÉBEC À MONTRÉAL   (Canada)

ⁿ Department of Obstetrics Gynecology   (United States)

^o UNIVERSITY OF MICHIGAN   (United States)

^p HARVARD MEDICAL SCHOOL   (United States)

more..

Author keywords

[No Author keywords available]

Indexed keywords

COGNITIVE SYSTEMS;

DIVERSE DOMAINS; EVOLUTIONARY ORIGIN; FORMAL PROPERTIES; INFORMATION SEARCH; MEMORY SEARCHES; MULTIPLE DOMAINS; NEURAL MECHANISMS; SIMILAR SOLUTION;

BIOLOGY;

COGNITION; ENVIRONMENTAL EXPLOITATION; EXPLORATORY BEHAVIOR; FORAGING; HUMAN; NEUROBIOLOGY; NONHUMAN; PROBLEM SOLVING; REVIEW; SOCIETY; SPACE; SPECIES; ANIMAL; APPETITE; BRAIN; PHYSIOLOGY; SOCIAL BEHAVIOR; VISION;

ANIMALS; APPETITIVE BEHAVIOR; BRAIN; COGNITION; EXPLORATORY BEHAVIOR; HUMANS; PROBLEM SOLVING; SOCIAL BEHAVIOR; VISUAL PERCEPTION;

EID: 84919497631     PISSN: 13646613     EISSN: 1879307X     Source Type: Journal    
DOI: 10.1016/j.tics.2014.10.004     Document Type: Review

References (120)

1. Cohen J., et al. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos. Trans. R. Soc. Lond. B: Biol. Sci. 2007, 362:933.
2. March J.G. Exploration and exploitation in organizational learning. Organ. Sci. 1991, 2:71-87.
3. Treisman A.M., Gelade G. A feature-integration theory of attention. Cogn. Psychol. 1980, 12:97-136.
4. Wolfe J.M., et al. Guided search: an alternative to the feature integration model for visual search. J. Exp. Psychol. Hum. Percept. Perform. 1989, 15:419-433.
5. Benhamou S. How many animals really do the Lévy walk?. Ecology 2007, 88:1962-1969.
6. Hills T.T., et al. Optimal foraging in semantic memory. Psychol. Rev. 2012, 119:431-440.
7. Korf R.E. Depth-first iterative-deepening: an optimal admissible tree search. Artif. Intell. 1985, 27:97-109.
8. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press.
9. Hills T.T. Animal foraging and the evolution of goal-directed cognition. Cogn. Sci. 2006, 30:3-41.
10. Stephens D.W., et al. Foraging: Behavior and Ecology 2007, University of Chicago Press.
11. Bonabeau E., et al. Swarm Intelligence: From Natural to Artificial Systems 1999, Oxford University Press.
12. Deneubourg J.L., et al. Random behaviour, amplification processes and number of participants: how they contribute to the foraging properties of ants. Physica D 1986, 22:176-186.
13. King A.J., et al. Is the true "wisdom of the crowd" to copy successful individuals?. Trends Cogn. Sci. 2009, 13:197-200.
14. Hills T.T., Dukas R. The evolution of cognitive search. Cognitive Search: Evolution, Algorithms, and the Brain 2012, 11-24. MIT Press. P.M. Todd (Ed.).
15. Charnov E.L. Optimal foraging, the marginal value theorem. Theor. Popul. Biol. 1976, 9:129-136.
16. Stephens D.W., Krebs J.R. Foraging Theory 1987, Princeton University Press.
17. Chun M.M., Wolfe J.M. Just say no: how are visual searches terminated when there is no target present?. Cogn. Psychol. 1996, 30:39-78.
18. Wolfe J.M., et al. Rare items often missed in visual searches. Nature 2005, 435:439-440.
19. Kristjansson A., et al. Fortune and reversals of fortune in visual search: reward contingencies for pop-out targets affect search efficiency and target repetition effects. Atten. Percept. Psychophys. 2010, 72:1229-1236.
20. Cousineau D., Shiffrin R.M. Termination of a visual search with large display size effects. Spat. Vis. 2004, 17:327-352.
21. Hills T., Todd P.M. Population heterogeneity and individual differences in an assortative agent-based marriage and divorce model (MADAM) using search with relaxing expectations. J. Artif. Soc. Social Simul. 2008, 11:5.
22. Pirolli P. Rational analyses of information foraging on the web. Cogn. Sci. 2005, 29:343-373.
23. Gigerenzer G., et al. Efficient cognition through limited search. Ecological Rationality: Intelligence in the World 2012, 608. Oxford University Press. P.M. Todd, G. Gigerenzer (Eds.).
24. Wilke A., et al. Fishing for the right words: decision rules for human foraging behavior in internal search tasks. Cogn. Sci. 2009, 33:497-529.
25. Hutchinson J.M.C., et al. Patch leaving in humans: can a generalist adapt its rules to dispersal of items across patches?. Anim. Behav. 2008, 75:1331-1349.
26. Scheibehenne B., et al. Expectations of clumpy resources influence predictions of sequential events. Evol. Hum. Behav. 2011, 32:326-333.
27. Dayan P., Berridge K.C. Model-based and model-free Pavlovian reward learning: revaluation, revision, and revelation. Cogn. Affect. Behav. Neurosci. 2014, 14:473-492.
28. Blanco N.J., et al. The influence of depression symptoms on exploratory decision-making. Cognition 2013, 129:563-568.
29. Otto A.R., et al. Physiological and behavioral signatures of reflective exploratory choice. Cogn. Affect. Behav. Neurosci. 2014, 14:1167-1183.
30. Markant D.B., Gureckis T.M. Is it better to select or to receive? Learning via active and passive hypothesis testing. J. Exp. Psychol. Gen. 2014, 143:94-122.
31. Sims C.R., et al. Melioration as rational choice: sequential decision making in uncertain environments. Psychol. Rev. 2013, 120:139.
32. Metcalfe J., Jacobs J. People's study time allocation and its relation to animal foraging. Behav. Processes 2010, 83:213-221.
33. Hills T.T., et al. The central executive as a search process: priming exploration and exploitation across domains. J. Exp. Psychol. Gen. 2010, 139:590-609.
34. Hills T.T., Pachur T. Dynamic search and working memory in social recall. J. Exp. Psychol. Learn. Mem. Cogn. 2012, 38:218-228.
35. Rakow T., et al. Biased samples not mode of presentation: re-examining the apparent underweighting of rare events in experience-based choice. Organ. Behav. Hum. Decis. Process. 2008, 106:168-179.
36. Unsworth N. Individual differences in working memory capacity and episodic retrieval: examining the dynamics of delayed and continuous distractor free recall. J. Exp. Psychol. Learn. Mem. Cogn. 2007, 33:1020-1034.
37. Hills T., et al. Mechanisms of age-related decline in memory search across the adult life span. Dev. Psychol. 2013, 49:2396-2404.
38. Sobel K., et al. Individual differences in working memory capacity and visual search: the roles of top-down and bottom-up processing. Psychon. Bull. Rev. 2007, 14:840-845.
39. Anderson D.E., et al. A common discrete resource for visual working memory and visual search. Psychol. Sci. 2013, 24:929-938.
40. Wolfe J.M., Danielson J. Visual foraging behavior: when are the berries riper on the other side of the screen?. J. Vis. 2012, 12:265.
41. Fuster J. The Prefrontal Cortex 2008, Elsevier.
42. Goldman-Rakic P.S. Cellular basis of working memory. Neuron 1995, 14:477-485.
43. Jones M.W., Wilson M.A. Theta rhythms coordinate hippocampal-prefrontal interactions in a spatial memory task. PLoS Biol. 2005, 3:e402.
44. Hyman J.M., et al. Working memory performance correlates with prefrontal-hippocampal theta interactions but not with prefrontal neuron firing rates. Front. Integr. Neurosci. 2010, 4:1-13.
45. Benchenane K., et al. Coherent theta oscillations and reorganization of spike timing in the hippocampal-prefrontal network upon learning. Neuron 2010, 66:921-936.
46. Voss J.L., et al. Hippocampal brain-network coordination during volitional exploratory behavior enhances learning. Nat. Neurosci. 2010, 14:115-120.
47. Schacter D.L., Addis D.R. On the nature of medial temporal lobe contributions to the constructive simulation of future events. Philos. Trans. R. Soc. Lond. B: Biol. Sci. 2009, 364:1245-1253.
48. Barnes T.D., et al. Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature 2005, 437:1158-1161.
49. Smith K.S., Graybiel A.M. A dual operator view of habitual behavior reflecting cortical and striatal dynamics. Neuron 2013, 79:361-374.
50. Dezfouli A., Balleine B.W. Habits, action sequences and reinforcement learning. Eur. J. Neurosci. 2012, 35:1036-1051.
51. Robbins T., Roberts A. Differential regulation of fronto-executive function by the monoamines and acetylcholine. Cereb. Cortex 2007, 17:i151-i160.
52. Cools R., D'Esposito M. Inverted-U-shaped dopamine actions on human working memory and cognitive control. Biol. Psychiatry 2011, 69:e113-e125.
53. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 1998, 80:1-27.
54. Barron A.B., et al. The roles of dopamine and related compounds in reward-seeking behavior across animal phyla. Front. Behav. Neurosci. 2010, 4:1-9.
55. Redish A.D., et al. Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol. Rev. 2007, 114:784-805.
56. Beeler J.A., et al. Tonic dopamine modulates exploitation of reward learning. Front. Behav. Neurosci. 2010, 4:1-14.
57. Servan-Schreiber D., et al. A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 1990, 249:892-895.
58. Durstewitz D., Seamans J.K. The dual-state theory of prefrontal cortex dopamine function with relevance to catechol-O-methyltransferase genotypes and schizophrenia. Biol. Psychiatry 2008, 64:739-749.
59. Yu A.J., Dayan P. Uncertainty, neuromodulation, and attention. Neuron 2005, 46:681-692.
60. Aston-Jones G., Cohen J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 2005, 28:403-450.
61. Kehagia A.A., et al. Learning and cognitive flexibility: frontostriatal function and monoaminergic modulation. Curr. Opin. Neurobiol. 2010, 20:199-204.
62. Huys Q.J.M., et al. Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol. 2012, 8:e1002410.
63. Winstanley C.A., et al. Search, goals, and the brain. Cognitive Search: Evolution, Algorithms, and the Brain 2012, 125-156. MIT Press. P.M. Todd (Ed.).
64. Frisch von K. The Dance Language and Orientation of Bees 1967, Harvard University Press.
65. Liang Z.S., et al. Molecular determinants of scouting behavior in honey bees. Science 2012, 335:1225-1228.
66. Beekman M., et al. What makes a honeybee scout?. Behav. Ecol. Sociobiol. 2007, 61:985-995.
67. Mason W.A., et al. Propagation of innovations in networked groups. J. Exp. Psychol. Gen. 2008, 137:422-433.
68. Lazer D., Friedman A. The network structure of exploration and exploitation. Admin. Sci. Quart. 2007, 52:667-694.
69. Kollman K. Decentralization and the search for policy solutions. J. Law Econ. Organ. 2000, 16:102-128.
70. Sorenson O., Sorensen J.B. Finding the right mix: franchising, organizational learning, and chain performance. Strat. Mgmt J. 2001, 22:713-724.
71. He Z-L., Wong P-K. Exploration vs. exploitation: an empirical test of the ambidexterity hypothesis. Organ. Sci. 2004, 15:481-494.
72. Rendell L., et al. Why copy others? Insights from the social learning strategies tournament. Science 2010, 328:208-213.
73. Wisdom T.N., et al. Social learning strategies in networked groups. Cogn. Sci. 2013, 37:1383-1425.
74. Berdahl A., et al. Emergent sensing of complex environments by mobile animal groups. Science 2013, 339:574-576.
75. Roberts M.E., Goldstone R.L. Adaptive group coordination and role differentiation. PLoS ONE 2011, 6:e22377.
76. Page S.E. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies 2007, Princeton University Press.
77. Krause S., et al. Swarm intelligence in humans: diversity can trump ability. Anim. Behav. 2011, 81:941-948.
78. Nemeth C.J., Wachtler J. Creative problem solving as a result of majority vs minority influence. Eur. J. Soc. Psychol. 1983, 13:45-55.
79. Couzin I.D., et al. Uninformed individuals promote democratic consensus in animal groups. Science 2011, 334:1578-1580.
80. Habermas J., McCarthy T. A Theory of Communicative Action 1985, Beacon Press.
81. Cognitive Search: Evolution, Algorithms, and the Brain (Strüngmann Forum Reports) 2012, MIT Press. P.M. Todd (Ed.).
82. Goldstone R.L., Gureckis T.M. Collective behavior. Top. Cogn. Sci. 2009, 1:412-438.
83. Gavrilets S. Evolution and speciation on holey adaptive landscapes. Trends Ecol. Evol. 1997, 12:307-312.
84. Gureckis T.M., Love B.C. Learning in noise: dynamic decision-making in a variable environment. J. Math. Psychol. 2009, 53:180-193.
85. Otto A., et al. Navigating through abstract decision spaces: evaluating the role of state generalization in a dynamic decision-making task. Psychon. Bull. Rev. 2009, 16:957-963.
86. Borge-Holthoefer J., et al. Modeling abnormal priming in Alzheimer's patients with a free association network. PLoS ONE 2011, 6:e22651.
87. Ramscar M., et al. The myth of cognitive decline: non-linear dynamics of lifelong learning. Top. Cogn. Sci. 2014, 6:5-42.
88. Nonacs P. State dependent behavior and the marginal value theorem. Behav. Ecol. 2001, 12:71-83.
89. Beachly W.M., et al. On the economics of sit-and-wait foraging: site selection and assessment. Behav. Ecol. 1995, 6:258-268.
90. Kareiva P., Odell G. Swarms of predators exhibit 'preytaxis' if individual predators use area-restricted search. Am. Nat. 1987, 130:233-270.
91. Viswanathan G., et al. Optimizing the success of random searches. Nature 1999, 401:911-914.
92. Hills T.T., et al. Adaptive Lévy processes and area-restricted search in human foraging. PLoS ONE 2013, 8:e60488.
93. Ferreira A.S., et al. The influence of the environment on Lévy random search efficiency: fractality and memory effects. Physica A 2012, 391:3234-3246.
94. Klein R. Inhibitory tagging system facilitates visual search. Nature 1988, 334:430-431.
95. Peterson M.S., et al. Visual search has memory. Psychol. Sci. 2001, 12:287-292.
96. Wolfe J.M., et al. Attention is fast but volition is slow. Nature 2000, 406:691.
97. Buckner R.L., Carroll D.C. Self-projection and the brain. Trends Cogn. Sci. 2007, 11:49-57.
98. Redish A.D. The Mind within the Brain: How We Make Decisions and How those Decisions Go Wrong 2013, Oxford University Press.
99. Kurth-Nelson Z., et al. A theoretical account of cognitive effects in delay discounting. Eur. J. Neurosci. 2012, 35:1052-1064.
100. Otto A.R., et al. Working-memory capacity protects model-based learning from stress. Proc. Natl. Acad. Sci. U.S.A. 2013, 110:20941-20946.
101. Genovesio A., et al. Prefrontal-parietal function: from foraging to foresight. Trends Cogn. Sci. 2014, 18:72-81.
102. Tolman E.C. The determiners of behavior at a choice point. Psychol. Rev. 1938, 45:1-41.
103. Johnson A., Redish A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 2007, 27:12176-12189.
104. Schacter D.L., Addis D.R. The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Philos. Trans. R. Soc. Lond. B: Biol. Sci. 2007, 362:773-786.
105. Hassabis D., et al. Patients with hippocampal amnesia cannot imagine new experiences. Proc. Natl. Acad. Sci. U.S.A. 2007, 104:1726-1731.
106. Keramati M., et al. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol. 2011, 7:e1002055.
107. Johnson A., et al. The hippocampus and exploration: dynamically evolving behavior and neural representations. Front. Hum. Neurosci. 2012, 6:216.
108. van der Meer M., et al. Information processing in decision-making systems. Neuroscientist 2012, 18:342-359.
109. Daw N.D., et al. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
110. van Bergen Y., et al. Nine-spined sticklebacks exploit the most reliable source when public and private information conflict. Proc. Biol. Sci. 2004, 271:957-962.
111. Barnard C.J., Sibly R.M. Producers and scroungers: a general model and its application to captive flocks of house sparrows. Anim. Behav. 1981, 29:543-550.
112. Vickery W.L., et al. Producers, scroungers, and group foraging. Am. Nat. 1991, 137:847-863.
113. Giraldeau L-A., Caraco T. Social Foraging Theory 2000, Princeton University Press.
114. Giraldeau L.A., et al. Potential disadvantages of using socially acquired information. Philos. Trans. R. Soc. Lond. B: Biol. Sci. 2002, 357:1559-1566.
115. Simons A.M. Many wrongs: the advantage of group navigation. Trends Ecol. Evol. 2004, 19:453-455.
116. Guttal V., Couzin I.D. Social interactions, information use, and the evolution of collective migration. Proc. Natl. Acad. Sci. U.S.A. 2010, 107:16172-16177.
117. Dussutour A., et al. The role of multiple pheromones in food recruitment by ants. J. Exp. Biol. 2009, 212:2337-2348.
118. Deneubourg J.L., et al. Probabilistic behaviour in ants: a strategy of errors?. J. Theor. Biol. 1983, 105:259-271.
119. Weidenmüller A., Seeley T.D. Imprecision in waggle dances of the honeybee (Apis mellifera) for nearby food sources: error or adaptation?. Behav. Ecol. Sociobiol. 1999, 46:190-199.
120. Galton F. Vox populi. Nature 1907, 75:450-451.