Neural and computational basis of dynamic material perception
Human Cognitive and Systems Neuroscience
Cognitive, Systems and Behavioural Neurobiology
Final Report Abstract
When we look at objects, we determine not only their identity and location but also their physical properties like softness or fragility, even without touching them. This ability is crucial for predicting and interacting with our environment. Yet, unlike object shape or orientation, physical properties, such as rigidity or friction, are not directly observable. How does the brain infer such properties by sight without access to the ground truth? Previous work has identified material motion (e.g., flowing, bouncing, deforming) as critical for disambiguating the visual signal and revealing the physical structure of the scene. This fellowship aimed at understanding the neural and computational basis of dynamic material perception. To investigate the neural basis of dynamic material perception we performed two experiments using functional magnetic resonance imaging. Specifically, participants watched video clips of dynamic interactions of four different types of materials (liquid, granular, non-rigid, rigid) and two control conditions while performing an orthogonal task. We found that the independently localized lateral occipital complex (LOC), previously shown to represent object shape, shows higher activation for all types of materials than control conditions, even for fluids with no constant shape. Furthermore, we found that an independently localized frontoparietal network of brain regions that has previously been implicated in visual intuitive physics, responds more strongly to all types of materials than control conditions, although responses were higher for solid objects than fluids. An adjacent rightlateralized region showed the opposite pattern (fluids > solid objects). Our results suggest that the cortical “physics network” does not just represent rigid-body physics (as previously tested) but physics more generally. Potentially, it has functionally distinct subregions that show higher responses for solids > fluids and vice versa. To investigate the computational basis of dynamic material perception, we studied different kinds of potential cognitive modes and constraints. Specifically, we investigated the role of heuristics in the visual estimation of material properties, here elasticity of bouncing objects. In particular, we experimentally brought several different models (23 heuristics and a rich combined feature model) into conflict and found that even for complex naturalistic motion trajectories, observers seem to rely on single heuristics such as the movement duration when estimating the elasticity of bouncing objects. Depending on the properties of the stimulus, observers can flexibly switch to another heuristic (e.g., when movement duration is not observable). This is consistent with the hypothesis that humans are computationally rational observers. In a further study, we found that when confronted with a different task, i.e., predicting the future path of a bouncing object, people can switch to another strategy not relying on heuristics. We found that human predictions are inexact, but show highly consistent error patterns suggesting an approximate, imperfect mental physics simulation. Finally, we further probed the mental physics model by presenting observers with incomplete scenes of material interactions in which only one of the interacting materials or objects was visible while the remaining scene was artificially rendered invisible. We found not only that material estimates were unaffected by this experimental manipulation, but importantly, that observers perceived the invisible objects and were able to select the underlying shape in a 2AFC. This finding suggests that the brain imputes the hidden objects in a physically plausible manner and is consistent with the hypothesis that people use an internal generative physics model in online perception.
Publications
-
Humans Can Visually Judge Grasp Quality and Refine Their Judgments Through Visual and Haptic Feedback. Frontiers in Neuroscience, 14.
Maiello, Guido; Schepko, Marcel; Klein, Lina K.; Paulun, Vivian C. & Fleming, Roland W.
-
Material perception for philosophers. Philosophy Compass, 16(10).
Ritchie, J. Brendan; Paulun, Vivian C.; Storrs, Katherine R. & Fleming, Roland W.
-
Distinct Neural Components of Visually Guided Grasping during Planning and Execution. The Journal of Neuroscience, 43(49), 8504-8514.
Klein, Lina K.; Maiello, Guido; Stubbs, Kevin; Proklova, Daria; Chen, Juan; Paulun, Vivian C.; Culham, Jody C. & Fleming, Roland W.
-
Efficient Visual Heuristics in the Perception of Physical Object Properties.
Paulun, Vivian C.; Bayer, Florian S.; Tenenbaum, Joshua B. & Fleming, Roland W.
