Recurrent neural network models for exploration in dynamic environments.

• Applicant: Professor Dr. Jan Peters
• Subject Area: Human Cognitive and Systems Neuroscience ; Experimental and Theoretical Network Neuroscience
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 496990750

Recurrent neural networks models have received increasing interest in cognitive and systems neuroscience. These models have recently been successfully trained on tasks from the human and animal neuroscience literature, and might yield insights into potential computational mechanisms underlying higher cognitive functions. In the first funding period, recurrent neural network models were used to shed light on a fundamental problem in reinforcement learning – the exploration/exploitation trade-off. A novel RNN architecture was identified that exhibited human-level performance on restless four-armed bandit tasks. Computational modelling revealed diverging mechanisms between human and RNN behavior, whereas aspects of neural representations in RNNs showed similarities to animal work. Further investigations revealed that computational mechanisms in RNNs increasingly converged with human data in case 1) networks had sufficient processing capacity and 2) test environments were sufficiently challenging (i.e exhibiting high volatility). The second funding period will examine mechanisms underlying adaptation to volatility and higher-order environmental structure in RNNs, develop novel exploration threshold models that may better account for human and RNN behavior in dynamic environments than standard reinforcement learning models, and test the diverging predictions of these different computational models regarding RNN hidden unit computations. Furthermore, individual differences in human exploration strategies are investigated identify potential mechanisms underlying behavioral heterogeneity observed in human learners during exploration.

• DFG Programme: Research Grants