Assessing cognitive workload during software engineering activities

Assessing the difficulty of a task as it is being performed can provide substantial value to an organisation. Fritz et al. [1] posit that such a capability would enable developers to, among other benefits, revise the estimates for tasks, reduce bug counts or even offer support to developers where needed. They proceeded to propose a methodology for assessing task difficulty among developers using physiological data from eye trackers, brain activity (electroencephalography or EEG), and electrodermal activity (EDA). Although the study by Fritz et al. yielded positive results, one cannot help but notice that the devices used in their research required a specialised lab setup and were arguably intrusive.

This project set out to investigate the extent to which the results obtained in the aforementioned study could be replicated by exclusively using sensors provided by a commercial off-the-shelf smart watch. The intention here was to explore the possibility of rendering such work more accessible in an industry setting.

Following a review of commercially available devices, the Fitbit Sense watch was chosen for the purpose of this study. The methodology followed by Fritz et al. was adapted to allow the of use heart-rate sensors on the said device. Twenty participants were asked to complete a set of software development tasks. The tasks were designed in a way that each successive task induced more cognitive stress on the participant. Time limits were also imposed to regulate the length of the study, and to induce additional pressure. Data was collected by means of a smartphone app during the exercise, as it extracted the data off the watch worn by the participants.

As with Fritz et al., a Bayes classifier was used to classify windows of heart-rate data as ‘stressful’ or ‘not stressful’. To train this classifier, participants were asked to complete the widely used NASA-TLX questionnaire about each task immediately after completing the respective tasks. This provided a subjective workload rating and perceived difficulty for each task, which was then used in conjunction with heart-rate windows as training data for the classifier.

Results varied but, as at the time of writing, it was possible to register a precision level of 73%. This demonstrates that real-time task-difficulty assessment using non-invasive commercial off-the-shelf devices is possible. This conclusion presents ample opportunities for future research in the area, ranging from improving the classification methodology, to encompassing the real time task-difficulty assessment as part of a set of real-world productivity tools that could support knowledge workers in their day-to-day jobs.

References/Bibliography

[1] T. Fritz, A. Begel, S. Müller, S. Yigit-Elliott and M. Züger, “Using psycho-physiological measures to assess task difficulty in software development”, Proceedings of the 36th International Conference on Software Engineering, 2014. Available: 10.1145/2568225.2568266.