Listen by Looking: A framework to support the development of serious games for live music

This paper presents an ongoing project focused on the framework Listen by Looking, that includes several tools to support the implementation of digital games, allowing the interactive display of music sheets, piano rolls and dialog windows with contextual information. Using this framework, a serious game for mobile devices was designed to entertain and teach music to users by listening to songs and answering questions about the melodies and musical structures learned during the game. The usefulness of game was assessed in a controlled setup, by comparing the answers of the app users with those of people who listened the same music without the app. Furthermore, an external synchronization tool through a data-over-sound channel has been tested in order to simultaneously manage a wide network of devices. The results are encouraging for future developments of the framework, with the possibility of using it during a public performance and with different games.