BLACK_III_desktop-1.jpg

Listening Machines

Final Project

Idea:

I believe that the music you grow up with and how your musical taste develops over time has a huge impact on your life.  These songs can become somewhat of a portrait of yourself.  Through this project, I became interested in creating a type of “fingerprint” of a song in order to create a visualization that defines who I am.  Ultimately, I created a series of fingerprints that each visualized a different feature extracted from songs important to me.  

Process:

I began by extracting some features from songs that I wanted to graph in my visualizations.  Figure 1 shows this initial feature extraction.

From left to right this image shows data from the HPCP, chromagram, and MFCC algorithms. I used this test to understand what each algorithm looked like before building the final visuals. 

Before making the fingerprints, I found a series of reference images that resembled the images I hoped to create (Figure 2). These reference images are from here and here respectively.

For the first visual, I wanted to use the chromagram data to build an image that would resemble a record as the song played.  Figure 3 shows a sample of the first visual.

As the song plays, a ray of the image is drawn each frame.  The inner most circle represents the C note and the outer most circle represents the B notes.  The size of each circle in the ray represents the intensity of the respective note in that instance.  The opacity of each circle is mapped to the intensity of the hpcp algorithm for that pitch frequency in that instance.  Next I wanted to create a visual that was similar to the one shown in Figure 2AFigure 4 shows the resulting visual.

Each note in the chromagram was mapped to a different symbol.  At each instance of the song, the symbol that is mapped to the most prominent note being played is drawn to the screen.  In the end you get a canvas that represents the song as a whole.  I then created a third fingerprint shown in Figure 5.

This sample graphs the intensity of each mfcc band to the size of the circles being drawn.  Finally, I created one last visual, shown in Figure 6.

This sample is slightly different form the rest since it does not create a fingerprint but allows the viewer to experience the visual as the song plays. 

Final Output:

For the final output, I selected six songs that represent my life…

  1. Big Yellow Taxi - Joni Mitchell

  2. Sweet Baby James - James Taylor

  3. Alison - Elvis Costello

  4. Thunder Road - Bruce Springsteen

  5. Mr Redundant - Rainbow Kitten Surprise

  6. The Once and Future Carpenter - The Avett Brothers

I ran my visuals on each one of these songs and created a large portrait.  The following images show the result.

Visual 1:

Visual 2:

Visual 3:

Conclusion:

The resulting images create a unique portrait of myself and my musical taste.  In the future I would like to expand this idea and create more complex fingerprints in order to get a more interesting stamp of each song.  It could also be interesting to train a machine learning model on these patterns and then provide it with a new song.  The model could potentially match the closest fingerprint to the incoming song. 

Eva Philips