Linking Program Comprehension to Neural, Behavioral, and Psycho-Physiological Correlates

• Applicants: Dr. André Brechmann; Professorin Dr.-Ing. Janet Siegmund
• Subject Area: Software Engineering and Programming Languages
• Term: from 2016 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 285612080

• Final Report Year: 2023

Final Report Abstract

In this project, we observed program comprehension with fMRI, EEG, and eye tracking. This led to new interesting insights and demonstrated that these new angles allow us and other researchers to obtain more detailed insights on how programmers understand source code. As a first necessary step, we refined the methodology to reliably measure program comprehension with neuroimaging techniques. To this end, we evaluated different contrast conditions and baseline conditions, so that we can recommend suitable conditions depending on a study. Furthermore, we extended the single method measurement protocol with different modalities, such that we can combine the strengths of modalities (e.g., high spatial resolution of fMRI, high temporal resolution of eye tracking) and mitigate their weaknesses (low temporal resolution of fMRI). This allows us to link neuronal activation to the center of visual attention of participants, so we can better understand what aspects of a program trigger high mental load or (natural) language processing. In our tool CodersMUSE, researchers can directly analyze the temporal flow of how individual tasks trigger neuronal activation, visual attention, and even further modalities, such as respiratory rate. Based on the refined measurement framework, we could move further in our endeavor to better understand how programmers work with source code. We found that certain elements in code that are believed to influence program comprehension tend to actually have no effect. In other words, it does not seem to matter for program comprehension whether an algorithm is implemented in a recursive or iterative way, how many spaces are used for indentation, or whether code is complex according to complexity metrics. Instead, the most relevant factor that we identified in our studies is the experience of programmers. Experienced programmers comprehend code with less cognitive effort and focus on relevant parts of code that follow the execution order of code, not the linear order. These insights can inform programming education research, such that teachers can focus on teaching relevant aspects, especially identifying and focusing on relevant parts of source code. Interview in German National Radio (Deutschlandfunk: “Computer und Kommunikation”) and Article in “Spektrum der Wissenschaft”

Publications

• Simultaneous measurement of program comprehension with fMRI and eye tracking. Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 1-10. ACM.
  Peitek, Norman; Siegmund, Janet; Parnin, Chris; Apel, Sven; Hofmeister, Johannes C. & Brechmann, André
  
• CodersMUSE: Multi-Modal Data Exploration of Program-Comprehension Experiments. 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), 126-129. IEEE.
  Peitek, Norman; Apel, Sven; Brechmann, André; Parnin, Chris & Siegmund, Janet
  
• A Look into Programmers’ Heads. IEEE Transactions on Software Engineering, 46(4), 442-462.
  Peitek, Norman; Siegmund, Janet; Apel, Sven; Kastner, Christian; Parnin, Chris; Bethmann, Anja; Leich, Thomas; Saake, Gunter & Brechmann, Andre
  
• Community expectations for research artifacts and evaluation processes. Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 469-480. ACM.
  Hermann, Ben; Winter, Stefan & Siegmund, Janet
  
• Mastering Variation in Human Studies. ACM Transactions on Software Engineering and Methodology, 30(1), 1-40.
  Siegmund, Janet; Peitek, Norman; Apel, Sven & Siegmund, Norbert
  
• Studying programming in the neuroage. Communications of the ACM, 63(6), 30-34.
  Siegmund, Janet; Peitek, Norman; Brechmann, André; Parnin, Chris & Apel, Sven
  
• Program Comprehension and Code Complexity Metrics: An fMRI Study. 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 524-536. IEEE.
  Peitek, Norman; Apel, Sven; Parnin, Chris; Brechmann, Andre & Siegmund, Janet
  
• REyeker: Remote Eye Tracker. ACM Symposium on Eye Tracking Research and Applications, 1-5. ACM.
  Mucke, Jonas; Schwarzkopf, Marc & Siegmund, Janet
  
• Understanding Comprehension of Iterative and Recursive Programs with Remote Eye Tracking. In: Proc. Annual Conf. Psychology of Programming Interest Group (PPIG). 2021
  Arooba Aqeel et al.
  
• Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking study. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 120-131. ACM.
  Peitek, Norman; Bergum, Annabelle; Rekrut, Maurice; Mucke, Jonas; Nadig, Matthias; Parnin, Chris; Siegmund, Janet & Apel, Sven