Research Results

Drawing out the true value of digitized music

Music-understanding technologies will open up a new future for musical experiences FY2022 update

photo: Masataka Goto
Masataka Goto (Prime Senior Researcher, Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST))
PRESTO
Researcher: "Information and Knowledge" (2000-2003)
CREST
Creation of Human-Harmonized Information Technology for Convivial Society Research director: "Building a Similarity-aware Information Environment for a Content-Symbiotic Society" (2011-2016)
Trusted quality AI systems Building a Trusted Explorable Recommendation Foundation Technology Research Director (2020-2025)
ACCEL
Research director: "Building Foundations and Developing Applications for Next-Generation Media Content Ecosystem Technologies" (2016-2020)
ACT-I
Research supervisor: "Information and Future" (2016-2021)

Dr. Goto and his team have launched a series of new cutting-edge music services that would be impossible without music-understanding technologies

With the digitization of music, people' s everyday music-listening environments have been enhanced dramatically. Thanks to the popularization of music streaming and video-sharing services, vast arrays of music can be enjoyed anywhere at any time, making it easier to listen to music than ever before. However, this only represents a quantitative change, in which a large volume of music has been accumulated and made readily available.

Masataka Goto (a Prime Senior Researcher, ACCEL research director, and former CREST research director) and his team thought that the latent potential of digitized music has not yet been exploited and decided to work on bringing about a future qualitative change with which a variety of different values could be produced based on the content (elements) of music. To this end, they have taken on the challenge of developing cutting-edge "music-understanding technologies" to draw out the true value of digitized music by estimating musical elements from the audio signals of pieces of music and allowing people to enjoy music in more active ways.

In 2012, for example, they, with the aim of opening up a path to the future of musical experience and based on many long years of research and development work into music-understanding technologies, developed and publicly launched an active music listening service called "Songle", which visualizes the musical elements of musical pieces. They then developed and publicly launched a series of other services such as "Songrium", a music browsing assistance service that makes visual representations of large arrays of music content and allows users to take a panoramic view of the music through visual representation, and " TextAlive", a service that allows users to easily create "lyric-based animations" to their own liking. Each of the services can be used by anyone who accesses the websites. The services make it possible to add further value to the vast online collections of music and provide active musical appreciation/creation experiences that will deepen user's understanding of music.

Songle, a service that can analyze music on the web and visualize its content while enabling users to jump to the chorus (hook)

Songle is a world-first service that, by utilizing "music-understanding technologies" developed by Dr. Goto and his team, allows people to g ain a better unde rstanding o f music and makes it possible to control robots, lighting, and various types of computer-graphic animation in synchronization with music.

Songle helps users deepen their understanding of musical pieces by allowing them to listen to those pieces while looking at "music map" that visualizes the essential musical elements of each piece, such as the chorus sections, melody, beats, and chord progressions, instead of just listening to the sound in a conventional manner. It also allows users to jump to the chorus (hook) of a song with just a push of a button. This is particularly useful when listening for the first time to songs users are not familiar with, meaning they can skim through vast volumes of music to select songs that suit their tastes while appreciating music in a more active manner. More than 1 million pieces have already been analyzed automatically, and their music maps are made publicly available.

Moreover, a development framework called Songle Widget has been launched, and this allows any user to utilize the music maps to achieve music-synchronized control. The framework has already been used in a wide variety of ways such as making robots automatically dance to music, controlling lighting, and displaying computer graphics that change in coordination with changes in a piece of music.