Digital harmony of sound and light

Bill Alves Harvey Mudd College Digital Harmony of Sound 301 E. 12th St. Claremont, California 91711 alves@hmc.edu and Light In the late 1980s and early 1990s I was privileged to The sounds and compositional resources available in Just intonation first attracted me as a composer. Though the Pythagorean tradition receded with work with the computer animation pioneer John Whitney Sr. and was profoundly influenced by his ideas on how to apply musical concepts of harmony to visual arts of motion. At the time of his death in 1995, he and I were planning composition software in which an artist could apply these concepts to create harmonic patterns simultaneously in sound and animation. Though this idea was never realized beyond certain tests, I have taken this step in my subsequent work in computer-generated music and video in ways inspired by, if distinct from, Whitney’s early artworks. This article examines ways in which Whitney’s ideas can be applied to musical composition, and in particular to ways in which I have extrapolated principles from these ideas to create an artistic correspondence between abstract animation and computer music. Visual Harmony We use the word harmony today to refer not just to the vertical dimension of music, but also a general sense of agreement or peace—the original meaning of harmonia to the ancient Greeks (Franklin 2002). Pythagoras was credited with defining this equilibrium in mathematical terms, as whole number proportions that represented an ideal not just in musical scales, but in sculpture and architecture. Indeed the same mathematical proportions formed the very fabric of Platonic universe (Cornford 1937). This close connection between music and visual art remained a vital force in European and Islamic art well into the Renaissance period. One does not have to subscribe to Pythagorean numerology to feel the arresting impact of the pure and crystalline harmonies created by musical tunings based on integer ratios, or Just intonation, as well as the beauties of proportional symmetries in visual design. Computer Music Journal, 29:4, pp. 45-54, Winter 2005 © 2005 Massachusetts Institute of Technology. the adoption of musical temperament and other compositional structures for visual arts and architecture, Sir Isaac Newton and others proposed a connection of sound and vision through the wave properties of color in light and pitch in music (Jones 1972). Arguably the first efforts to use technology to apply concepts of musical composition to abstract arts of motion came with a series of experimental devices from the 18th through the 20th centuries generally called “color organs.” Most famously, the composer Aleksandr Skryabin’s Prométhée includes a part for such an instrument which would project the colors of Skryabin’s synesthetic associations. However, the technology these innovators used allowed only poor and vague definition of visual form (Collopy 2000). The technology of filmed animation offered much greater control over form and direct synchronization to music. The great abstract animator Oskar Fischinger created a remarkable series of films in the first half of the twentieth century in which he represented the sounds of accompanying music often with delicately fluid forms (Moritz 2004). Many of his films are a kind of supple choreography distilled to abstraction, but relying on an intuitive sense of connection to musical form. Mapping Vision to Sound Distinct approaches to the correspondences between sound and light are at least as important as differences in technique in distinguishing the work of artists in this tradition. Many artists, from the 18th-century inventor of a “color harpsichord,” Louis-Bertrand Castel, to modern creators of music visualizer software, have attempted to directly illustrate music, mapping pitch to spatial height or color hue, for example (Klein 1937}. The Music Animation Machine (www.musanim.com/} of Stephen Malinowski is especially interesting in this approach. Alves

Bill Alves Harvey Mudd College 301 E. 12th St. Claremont, California 91711 alves@hmc.edu In the late 1980s and early 1990s I was privileged to work with the computer animation pioneer John Whitney Sr. and was profoundly influenced by his ideas on how to apply musical concepts of harmony to visual arts of motion. At the time of his death in 1995, he and I were planning composition software in which an artist could apply these concepts to create harmonic patterns simultaneously in sound and animation. Though this idea was never realized beyond certain tests, I have taken this step in my subsequent work in computer-generated music and video in ways inspired by, if distinct from, Whitney’s early artworks. This article examines ways in which Whitney’s ideas can be applied to musical composition, and in particular to ways in which I have extrapolated principles from these ideas to create an artistic correspondence between abstract animation and computer music. Visual Harmony We use the word harmony today to refer not just to the vertical dimension of music, but also a general sense of agreement or peace—the original meaning of harmonia to the ancient Greeks (Franklin 2002). Pythagoras was credited with defining this equilibrium in mathematical terms, as whole number proportions that represented an ideal not just in musical scales, but in sculpture and architecture. Indeed the same mathematical proportions formed the very fabric of Platonic universe (Cornford 1937). This close connection between music and visual art remained a vital force in European and Islamic art well into the Renaissance period. One does not have to subscribe to Pythagorean numerology to feel the arresting impact of the pure and crystalline harmonies created by musical tunings based on integer ratios, or Just intonation, as well as the beauties of proportional symmetries in visual design. Computer Music Journal, 29:4, pp. 45–54, Winter 2005 © 2005 Massachusetts Institute of Technology. Digital Harmony of Sound and Light The sounds and compositional resources available in Just intonation first attracted me as a composer. Though the Pythagorean tradition receded with the adoption of musical temperament and other compositional structures for visual arts and architecture, Sir Isaac Newton and others proposed a connection of sound and vision through the wave properties of color in light and pitch in music (Jones 1972). Arguably the first efforts to use technology to apply concepts of musical composition to abstract arts of motion came with a series of experimental devices from the 18th through the 20th centuries generally called “color organs.” Most famously, the composer Aleksandr Skryabin’s Prométhée includes a part for such an instrument which would project the colors of Skryabin’s synesthetic associations. However, the technology these innovators used allowed only poor and vague definition of visual form (Collopy 2000). The technology of filmed animation offered much greater control over form and direct synchronization to music. The great abstract animator Oskar Fischinger created a remarkable series of films in the first half of the twentieth century in which he represented the sounds of accompanying music often with delicately fluid forms (Moritz 2004). Many of his films are a kind of supple choreography distilled to abstraction, but relying on an intuitive sense of connection to musical form. Mapping Vision to Sound Distinct approaches to the correspondences between sound and light are at least as important as differences in technique in distinguishing the work of artists in this tradition. Many artists, from the 18th-century inventor of a “color harpsichord,” Louis-Bertrand Castel, to modern creators of music visualizer software, have attempted to directly illustrate music, mapping pitch to spatial height or color hue, for example (Klein 1937). The Music Animation Machine (www.musanim.com/) of Stephen Malinowski is especially interesting in this approach. Alves

However, though artists often speak of “color harmony,” attempts to create a direct mapping of color relationships to the immediacy of musical consonance and dissonance have largely failed. To Whitney, such a direct, synesthetic mapping of music’s most basic parameters (pitch, loudness, and so forth) failed to capture the expressive vision of great works of music, which, to him, depended more directly on multidimensional interplay of tension and resolution. Moreover, he advocated an approach in which animation, instead of being a direct representation of music, corresponds to this higher level of aesthetic intention, creating what he termed “complementarity” (Whitney 1984). Whitney’s Differential Dynamics ferent senses throughout history. In particular, we must be careful to distinguish between consonance as a psychoacoustical phenomenon, which may depend in large part on the closeness of the frequency proportions to ratios of relatively small whole numbers, my focus here, and consonance in a more general sense as harmonic stasis or resolution, which may be created in many different ways.) Though Whitney’s films brilliantly demonstrate how computer animation can effectively create a temporal sensation of attraction and repulsion, of tension and resolution, he was never able to take the next step of creating music to correspond directly to those visual images. To take a very simple example of what Whitney called “differential dynamics,” imagine a set of points going around a circle, the second traveling twice the speed of the first, the third three times the speed of the first, and so on, all starting together at the twelve o’clock position. By the time the slowest point reaches halfway around the circle, all the points will align with either the six or twelve o’clock position. If the slowest point is at either the one-third or two-thirds position, all points will line up at the thirds: twelve, four, or eight o’clock. When the slowest point is at a position around the circle which is not a very clear integer divisor of the circle’s perimeter, the points will appear scattered somewhat randomly, and the eye will not detect a clear pattern. John Whitney recognized that digital computers could uniquely and directly realize in animated form the same kind of harmonic movement found in music in ways never imagined by the Greeks. Beginning in the 1960s, Whitney created a series of extraordinary films of abstract animation that used computers to create a harmony not of color, space, or musical intervals, but of motion. In his 1980 book Digital Harmony: On the Complementarity of Music and Visual Art, Whitney hypothesized that Pythagorean harmony could be matched in visual art: “This hypothesis assumes the existence of a new foundation for a new art. It assumes a broader context in which Pythagorean laws of harmony operate. . . . In other words, the hypothesis assumes that the attractive and repulsive forces of harmony’s consonant/dissonant patterns function outside the dominion of music” (p. 5). More particularly, Whitney discovered that if he set a large number of elements into repetitive motion such that the motion of the second was two times the speed of the first, the third three times the speed of the first, and so on, the animation that would result would demonstrate beautiful patterns of symmetry at points corresponding to the same ratios that define musical consonances. (Tenney [1988] reminds us that the terms consonance and dissonance have been used in many dif- Of course points moving around a circle is a very simple example. In his films Matrix I and III (1970 and 1972), Whitney had points or other shapes move around parametric curves. In Permutations (1968), the points move in various rose curves (sine functions in polar coordinates), and in Arabesque (1973) points initially arranged in a circle move linearly but wrap around at the edge of the screen (see Figure 1). In all of these examples, the same points of resonance, of striking symmetry, occur at the same points of harmonic proportions. Each of these early films, created by photographing frames from a CRT monitor with an animation Computer Music Journal Whitney’s Works

dynamics suggest a different kind of relationship. In the example of points moving around a circle, when the first point reaches the one-third point in the circle, all the points are aligned to the 12:00, 4:00, and 8:00 positions. By treating the 12:00 position as the fundamental, the one-third point would represent a 3/1 frequency ratio and the two-thirds point the 3/2 frequency ratio. (Just intonation ratios are conventionally notated with the larger number in the numerator. I have adopted this convention for ratios of both frequencies and visual proportions, as in this example.) Similar patterns would occur at other wholenumber divisions, but with increasing complexity as that whole number increases. Thus a five-fold division of the circle would correspond to frequency ratios of 5/5 (that is, 1/1, the fundamental), 5/4, 5/3, 5/2, and 5/1. Unlike the harmonic series, this set of musical intervals has a constant factor in the numerator. This set defines what Partch called an utonality and what others have called a subharmonic series (that is, the inversion of the harmonic series). Subharmonic series occur with combinations of any integer divisions of a whole, as with divisions of string lengths or evenly spaced holes on aerophones, for example. The higher the numerator, or number of divisions, what Partch calls the numerary nexus, the further along the spectrum we are towards dissonance. The smaller this common factor or numerary nexus, the more stable but less dynamic the sound and image. Thus the frequency ratio of 2:1 is the octave, which, as with the differential dynamics example, results in a profoundly stable but not very interesting consonance. Somewhat more complex factors such as 5 or 7, which offer more interesting musical consonances, result in fascinating symmetrical patterns that catch the eye’s attention when applied the differential dynamics system. Figure 2 demonstrates examples of these points of consonance, first in the simple example of points moving in a circle, then in a more complex rosecurve pattern, and finally as a set of tones in the same proportions. The number in the first column represents the proportion that the slowest element has traversed through the entire cycle, and hence the common factor or numerary nexus of all the elements. At points where this number represents a ratio of relatively small whole numbers, symmetrical patterns emerge. The right-hand column of Figure 2 shows the chords that would result if a set of sixteen pitches went through the same differential cycle. The beginning position (12:00 in circular motion), the fundamental, is here set to C2. One quarter through the cycle, for example, would represent a 4/1 ratio, or a frequency four times C2. To represent some of the pitches poorly approximated by twelve-tone notation, I have used an accidental of an arrow up or down to represent a pitch inflection of about 33 cents and a plus sign to indicate raising the pitch about 15 cents. The final row is an example of how both visual and aural dissonance results when the common factor is irrational or sufficiently complex. Artists such as Ronald Pellegrino have discovered similar points of correspondence between Just intonation and visual symmetry through the use of analog electronics, especially the oscilloscope (Pellegrino 1983). When two tones of relatively simple whole number frequency relationships are connected to the x and y inputs of an oscilloscope (or the vibrating mirrors of a laser scanner), the resulting visual form will be a Lissajou figure of relative stability and symmetry. The algorithms of Monro and Pressing (1998) demonstrate similar relationships between Just intonation harmonies and visual symmetries in more extensible and elaborate ways. However, Whitney intended differential dynamics to provide a set of principles which could be applied compositionally in many different forms, rather than algorithms for visualization. 48 Computer Music Journal New Explorations In my first video based on these principles, Hiway 70 (1997), I extended the polar coordinate curves of Whitney’s Permutations to three-dimensional graphics. But the most important way in which my work was distinguished from his is that, approaching this work as a composer, I created a soundtrack in tandem with the visual composition, carefully synchronizing movement between points of tension and dissonance and points of stability and tonal

Figure 2. Correspondences between points of visual resonance in two examples of differential motion and musical pitches (see text). Common factor Differential dynamics example—spheres moving in a circle Differential dynamics example—spheres moving in a rose curve pattern Musical correspondence pitches are approximate 1 2 3 continued

Common factor Differential dynamics example—spheres moving in a circle Differential dynamics example—spheres moving in a rose curve pattern 6 7 8 9 10 8.45 50 Computer Music Journal Musical correspondence pitches are approximate

Moritz, William. 2004. Optical Poetry: The Life and Work of Oskar Fischinger. Bloomington: Indiana University Press. Partch, Harry. 1974. Genesis of a Music. 2nd ed. New York: Da Capo. Pellegrino, Ronald. 1983. The Electronic Arts of Light and Sound. New York: Van Nostrand Reinhold. Sethares, William A. 1999. Tuning, Timbre, Spectrum, Scale. London: Springer Verlag. Tenney, James. 1988. A History of Consonance and Dissonance. New York: Excelsior. Whitney, John. 1980. Digital Harmony: On the Complementarity of Music and Visual Art. New York: Byte Books. Whitney, John. 1984. “To Paint on Water: The Audiovisual Duet of Complementarity.” Computer Music Journal 18(3):44–52. Wilson, Ervin. 1989. “D’Alessandro, Like a Hurricane.” Xenharmonikon XII:1–39. Computer Music Journal