Teaching AI through Augmented Reality

Artificial intelligence (AI) has pervasively penetrated every aspect of our life, impacting the way society operates in general [1]. Such an emphasis on this technology highlights the importance that the general public could clearly comprehend and appreciate the nature and scope of AI. This awareness must be nurtured from a young age, as children today constitute tomorrow’s society, and are therefore required to be fully conversant  with AI and how it could render their lives easier [2], [3].

This final-year project seeks to address the challenge of teaching AI concepts to children, by offering a digital tool to help teachers/educators achieve this. This digital tool is an Android application that was developed using Unity which in turn uses the Vuforia Augmented Reality software development kit. The application also makes use of the Google Cloud Platform (GCP) and Google App Engine in order to expose a Python-based web Application Programming Interface (API) containing convolutional neural networks (CNN) developed using Keras. The application is in the form of a game in which children need to teach an augmented reality (AR) robot how to classify different colours by providing images to train the CNN. The application provides step-by-step instructions on what needs to be done by the user as well as simple explanations on the machine learning (ML) concepts being used. These concepts include: neural networks, training a neural network, and using neural networks for predictions and classifications. The gamified system is intended to encourage children to take interest in what they are being taught [4].

Whereas the interface of the application was designed using Unity, the CNN was designed using Python. The study discusses how the application transfers data between the Unity interface which uses the C# Programming language and the CNN, written in Python, through the means of a Google Web API. The expected outcome of this final-year project is to provide an entertaining teaching method to motivate children to learn important concepts of ML, this being a very important area of AI. The proposed method has been deemed to be more engaging, thus rendering the topic easier to understand than if it were taught through traditional teaching methods [5].

Figure 1. Flow of data between the application interface and the cloud- based API.

The effectiveness of the application was evaluated by comparing the level of knowledge gained after watching a video and using the application. The video was created specifically for the purpose of the study, and designed to convey the same information as the application. Target users (ie. children) were asked to answer the same questions after watching the  video  and  using the application. The quality of these answers has been reviewed in order to identify whether the participants gave better answers after having used the application.


[1] C. Baciu, D. Opre, and S. Riley, “A new way of thinking in the era of virtual reality and artificial intelligence,” 06 2016.

[2] A. Sangra and M. Gonzalez-Sanmamed, “The role of information and communication technologies in improving teaching and learning processes in primary and secondary schools,” ALT-J, vol. 18, no. 3, pp. 207–220, 2010.

[3] D. Kawade and S. Kulkarni, “Use of ICT in primary school,” Pioneer Journal, 2012.

[4] J. Lee and J. Hammer, “Gamification in education: What, How, Why bother?,” Academic Exchange Quarterly, vol. 15, pp. 1–5, 01 2011.

[5] A. Stott and C. Neustaedter, “Analysis of gamification in education,” Surrey, BC, Canada, vol. 8, p. 36, 2013.

Student: Francesca Silvio
Course: B.Sc. IT (Hons.) Artificial Intelligence
Supervisor: Prof. Matthew Montebello
Co-supervisor: Dr Vanessa Camilleri