Reflections on Marvin Minsky and His Music

Xiao Xiao on 2016-08-10

Meeting Marvin Minsky at an early prototype of my project MirrorFugue, October 2010

I only came to know Marvin through music — or through the piano, to be precise. In undergrad, I had known of Marvin, had heard of him as a larger than life figure of myth and mystery. This was around the time that his book The Emotion Machine was published. I had attended a lecture at the Media Lab as a sophomore and returned starry eyed, with an autograph.

I first really met Marvin at the 25th anniversary of the Media Lab, in my sophomore year of grad school. During an open house, he strolled right up to my demo of MirrorFugue and started playing. I would later learn from his wife Gloria that Marvin was a “piano detective”. If there were a piano anywhere in a building, he was sure to find it. Unknowingly, I had set up a sure way to make Marvin’s acquaintance.

Through my friends at the lab, I had heard of Marvin’s ability to improvise contrapuntal music but just assumed that it was the stuff of myth. It’s hard enough to improvise a single melody, much less two or more at the same time. But he really did! Right in front of me! I even got to add a note here and there.

Later, through his daughter Margaret, I had the pleasure of joining a series of musical salons at the Minsky residence. Through these salons, Marvin kindly agreed to be recorded for MirrorFugue. That was all before his health began deteriorating. Then I didn’t see too much of him for a couple of years.

Illustrating Marvin’s essay Music Mind, and Meaning for one of the music salons, 2011

In Fall of 2014, I saw Marvin again at the symposium on Beethoven’s Improvisations organized by Margaret. Not long after the event, I had heard of an AI meeting at the Minsky residence and decided to attend. Somehow, I had gotten the date wrong and showed up at the house right before a meeting of the Brookline Women’s Commission organized by Gloria. Since I had made it all the way there, I sat down at the piano and began to play a set of variations by Beethoven. At the end of the final variation, Marvin came up to the piano and doubled the main melody in a higher octave. When I finished the piece, he sat down and once again began to improvise. But he couldn’t play for long. The Brookline Women’s Commission had assembled in the meantime and were waiting to start their meeting.

When Gloria came in to tell Marvin he had to stop playing, Marvin looked really, really sad. This was how I came up with the idea to come back every week to play the piano for and with Marvin. I would play my Beethoven or my Bach and when I ran out of pieces I knew, I’d sightread whatever was on the piano, which was more Beethoven and more Bach. Sometimes, Marvin would come and improvise something, and a couple of times near the end, we would each be on a piano and would take turns making up phrases.

Marvin and I didn’t really talk too much. When we did, it was usually about childish things—squirrels, ice cream sandwiches, wind-up houses that powered everything inside. Sometimes, I’d share some insights about music that I discovered that week, and he would smile knowingly in return. Once, I asked Marvin how he got started with the piano, and he said that he grew up with a player piano that he would “program” by punching holes at the blank beginnings of piano rolls. You could even correct your mistakes with sticky tape. It occurred to me that the piano was how Marvin got his start programming before modern computers existed. I wonder how much it’s all related — getting machines to think and feel and programming the piano to play Bach and Beethoven.

I don’t claim to speak for Marvin, not about music nor about anything else. I had intended to spend more time at the Minskies while writing my PhD dissertation to pick his brain about ideas that I had, but I guess I can’t anymore…

I’d like to think that he would have enjoyed hearing about these ideas — thoughts that certainly owe a great deal to his own thinking. Yes, Marvin was famous as the founding father of Artificial Intelligence, for inventing the confocal microscope, and for his thoughts on computation, but there was another side of Marvin, another voice in his fugue — perhaps the cantus firmus — that always wondered about the nature of these marvelous machines we call human beings, machines that can move, that can remember, that can feel.

I feel that Marvin’s intimate relationship with music had something to do with this line of thought. Many mathematically inclined individuals appreciate music, especially the music of Bach and especially the clever games he played. They would analyze the structure of canons and fugues, coming up with unifying principles connecting math and music. Sometimes they learn how to play, but their playing stays on the level of intellectual acrobatics. (Nothing wrong with that. It’s a fine way to appreciate Bach, and on some level music is math). It’s easy to chunk Marvin with the other mathematicians, but his musical inquiry was fundamentally different.

First of all, Marvin’s favorite composer was actually Beethoven, not Bach (though I’m sure Bach comes a close second). That says something. You cannot truly appreciate Beethoven if you cannot embrace emotion. When I was a teenager, I stayed away from Beethoven for that very reason, like many of those technically and mathematically inclined who fear or even vilify emotion. But not Marvin! He embraced emotions by really trying to understand them, and not just intellectually. In his younger days, Marvin took it upon himself to learn Beethoven’s Appassionata sonata and according to his son Henry played it quite well, with true feeling. I wasn’t there to hear it, but I believe it. Even though Marvin lost much of his technical dexterity in his old age, he still played with warmth in his touch. And you can’t play the piano with a warm touch if you don’t have warmth in your heart. I can’t ask for certain anymore, but I wonder whether Marvin went to the Appassionata as a way to understand human emotion, not just by passive listening, nor by pure intellectual analysis, but by embodying the music.

Beethoven was also known for his motivic development and his architecture. If you take apart any piece of his, any individual building block is not much more than trivial, just triads, a diminished 7th here and there. But Beethoven’s genius was in how all the small units, each in itself not so “intelligent”, came together. The 5th symphony is famously built up from just two ambiguous thirds in the “da da da dum” motif.

Also, Beethoven wasn’t known for his fugues but he wrote beautiful ones. In his youth, Beethoven was apparently criticized by his teacher Hayden for being bad at contrapuntal writing, so the fugues at the end of many of his later great works (9th symphony, the end of the Diabelli variations, the Opus 111 piano sonata) were his demonstration — his building of the tower — that he had mastered his art.

Perhaps Marvin learning to improvise fugues was his way of learning how to learn. You could “understand” a fugue through its rules. You could segment the voices, label the harmonies, and study the subject in all its very mathematical transformations. And I’m sure Marvin did do all of that, but he also learned how to embody the processes to create fugues. How was he able to do that? By building up smaller modules—inventing small motifs, transposing them into every key, then figuring out how to connect the smaller modules. To the musically naïve, Marvin’s ability appears as the product of genius because they were not there to see all the small steps it took and how they all added up.

An article about a talk Marvin gave at IRCAM that confirms my theory

Marvin himself never claimed to be a genius musician. In fact, he never claimed to be any sort of genius. It was always others who ascribed that quality to him. And even though he rushed to play on any piano that he found, he never played to show off. When I asked him about improvising fugues, he quickly corrected me that what he played weren’t real fugues, only two voices, sometimes another half if he was lucky. Marvin played in an eternal quest to better understand the mysteries of the human mind — especially how we learn and how we are able to feel. These insights then fueled his mission to create machines that could learn and machines that could feel.

My relationship with music is similar. For me, learning to embody music is also a vehicle to better understand the human mind, especially learning and feeling. And ultimately, I think this was what brought Marvin and me together through the piano even though my research has little to do with AI. But perhaps it is more related than I had imagined myself. Rather than creating machines that could learn and feel, I wish to create machines to help us learn and machines that make us feel. To attempt both requires unraveling the mysteries of the human mind and heart.