Risk-sensitive choice and reinforcement learning under uncertainty

• Applicants: Professor Dr. Klaus Obermayer; Professor Dr. Dirk Ostwald
• Subject Area: Human Cognitive and Systems Neuroscience
• Term: from 2018 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 407012307

Many real-life decisions must be made in the face of risk that is due to uncertain information about the choice situation. At least two types of uncertainty can result in decision risk: (1) the uncertain consequences of the decision maker’s choice and (2) the decision maker’s uncertain knowledge about the choice situation. In effect, both types of uncertainty can impact the economic consequences of a decision and thus result in risk. We refer to these types of risk as “economic risk” and "perceptual risk". While the neural mechanisms underlying economic risk processing are fairly well established for static conditions and in the absence of perceptual uncertainty, the cognitive and neural processes underlying the acquisition of economic and perceptual risk-sensitive decision strategies are less clear. In the proposed project, we build on our recent computational work in the development of risk-sensitive Markov decision processes and reinforcement learning algorithms to investigate the cognitive and neural mechanisms of encoding economic and perceptual risk. At the theoretical level, this entails the adaptation of a risk-sensitive partially-observable Markov decision process (POMDP) framework as a behavioral model for human response times and choices. At the experimental level, it entails the empirical validation of the framework in two behavioral and one fMRI experiments. We will pursue these aims in three work packages. In the first work package, we will build on our previous theoretical work to derive novel model-free reinforcement learning algorithms that account for both economic and perceptual risk. In our second work package, we will test the hypothesis that the risk-sensitive POMDP framework will surpass previous approaches in capturing human response time behavior. In the third work package, we will use fMRI to identify the neural underpinnings of economic and perceptual risk processing. Overall, our project will extend previous work on risk-sensitive learning and decision making to the presence of perceptual uncertainty.

• DFG Programme: Research Grants