This work investigated the effects of different audio contexts on software engineers when performing common software engineering tasks. The approach adopted was to observe (and eventually compare) the behaviour and performance of three cohorts of participants under different controlled experimental conditions.
A between-subject study design was adopted, where participants were placed into one of three experimental groups, namely: a control group that did not listen to any audio; participants who were made to listen to music; and participants listening to a podcast.
A controlled lab-based study was carried out to analyse the relationship between audio and task performance. A total of 30 undergraduate students pursuing degrees in information technology was presented with nine software engineering tasks to be completed in a controlled lab environment. These tasks fell into the following three categories: code comprehension; locating and debugging errors; and conducting a dry run of a given program. These tasks were designed with varying levels of complexity in order to investigate the behaviour of each group when performing increasingly difficult tasks.
During the series of tasks, participants were asked to wear a pair of headphones and were guided through a series of tasks using a custom-built application. An eye tracker was used to collect gaze data, which was used to analyse participant behaviour recorded through on-screen scanning and reading patterns while performing such tasks. A number of metrics were also collected, including: perceived workload levels, time measurements; task completion rates; and successful completion rates.
The experiment entailed monitoring a number of factors, such as audio levels and environmental conditions. This was done in order to maintain consistency throughout the study, allowing for better reproducibility, as well as to reduce any bias on participants that could affect the final results.
After each participant has completed the given tasks, an extensive analysis of each cohort’s performance and behaviour was carried out. Initial findings suggested that listening to podcasts tended to improve overall performance, possibly as a result of over-compensation, but also generated increased mental effort and frustration. On the other hand, music enhanced code comprehension, which grew as task complexity increased. Lastly, silence reduced frustration levels but without conveying a demonstrable benefit in task success rates.
It is believed that the proposed approach would provide the industry with insight into the potential benefits or drawbacks that listening to different audio sources could have on the performance of individual software engineers.
Figure 1. Heatmap generated on the basis of gaze data of one participant in the podcast group
Student: Robert Fenech Adami
Supervisor: Dr Chris Porter