Inclusive Musical Game Technology for Children

This research proposes a novel teaching aid aimed at music schools teaching music to students, including those with special needs. Previous research has successfully used modified or specifically built data gloves for this purpose. Data gloves worn by the teacher and the student were able to convey real time guidance and feedback by detecting the fingers’ state using strain gauges and conveying feedback to the student using miniature vibrating devices incorporated in the gloves.

This study proposes a flexible approach where the gloves could either respond to live performance by a teacher or use existing MIDI music files as a source. This could allow teachers to cater for more students simultaneously and facilitate practising at home. Furthermore, recording and playback facilities could allow the teacher’s and student’s performances to be recorded for repeated training or evaluation exercises respectively.

For the purpose of this study, a set of fabric gloves were equipped with strain gauges which measure the angle of each finger, thereby detecting which notes are currently being pressed. Similarly, small vibrating devices were attached to the fingers in order to give feedback to the student or direct him/her to which are the correct notes to press. A third device equipped with a loudspeaker is also included to give auditory feedback to the user.

The Espressif ESP32 was selected as the microcontroller platform for this project. The software was designed in C++ and developed using the Arduino IDE, a more user-friendly option than the native ESP-IDF environment provided by Espressif. Wi-Fi was chosen as the preferred communication medium due to its additional range and bandwidth over Bluetooth. This allows the various devices to communicate together as well as with user devices via a web interface which allows the users to control, monitor and upload MIDI files to the devices.