In recent years, there has been a growing interest in the use of artificial intelligence (AI) when creating art. In response, many AI-based tools have been developed to assist artists. One frequently used AI tool is style transfer. This method can determine the artistic style of a reference image and convert another image to the same style.
The aim of this project was to apply existing style-transfer models to videos and games. The first task was applying style transfer to the background of videos. The project involved implementing a simple version of this, and explored different methods for refining the result. It built upon modern style-transfer and foreground-segmentation models.
The first step was to examine and compare many different style-transfer models. The output of the model was then processed to achieve the desired output. The second task involved applying style transfer in games, to be used by developers to quickly create custom, artistic filters for their games. Other implementations would require high-end graphics processing units (GPUs) in order to run in real time. The method implemented in this project would allow for achieving a visually similar result but in a potentially far more efficient method.
Previous implementations of this work by stylising every frame. This would require applying style transfer 60 times per second. The proposed method stylises the objects in the game once, instead, thus avoiding repeatedly stylising the same element. In addition, it would be possible to use this technique on lower-end hardware by pre-styling these elements. An additional script was developed in order to facilitate this process and would be applicable to both 2D and 3D games. In the case of 2D games, it would be simply a matter of stylising the images used within the games, whereas for 3D models, stylising the 2D texture wrapped around the model would achieve a similar result.
Figure 1. Stylising a segmented video based on Starry Night by van Gogh
Figure 2. Stylising a 2D game based on Starry Night by van Gogh
Student: Kyle Agius
Supervisor: Prof. John Abela