Inclusive musical game technology for children

This research proposes a novel teaching aid aimed at teachers within music schools and their students, including students of music with special needs. Previous research has successfully used modified or specifically built data gloves for this purpose. Data gloves worn by both teacher and student could convey real-time guidance and feedback by detecting the state of the fingers through strain gauges – and conveying feedback to the student using miniature vibrating devices incorporated into the gloves.

This study proposes a flexible approach, whereby the gloves could either respond to live performance by a teacher or use existing MIDI music files as a source. This would allow teachers to cater for more students simultaneously and facilitate practising at home. Furthermore, recording and playback facilities would allow the performances of teachers and students to be recorded for repeated training or evaluation exercises respectively.

For the purpose of this study, a set of fabric gloves were fitted with strain gauges to measure the angle of each finger, thereby detecting which keys were being pressed. Similarly, small vibrating devices were attached to the fingers in order to give feedback to the student or to indicate the correct keys to press. A third device equipped with a loudspeaker was also included so as to provide auditory feedback to the user.

The software was designed in object-oriented C++ and developed using the Arduino IDE, which is a more user-friendly option than the native ESP-IDF environment provided by Espressif. Wi-Fi was chosen as the preferred communication medium due to its additional range and bandwidth over Bluetooth. This would allow the various devices to communicate with each other, as well as with user devices through a web interface that would allow the users to control, monitor and upload MIDI files to the devices.