This dissertation has examined multiple aspects of the project from a variety of perspectives. My background and my motivation in creating ConDiS, the role of the conductor, and the evolution of gestural interfaces are all addressed in a historical context. Practical exercises and experiments conducted throughout the research process are explained. This is followed by an analysis of the conductor’s use of her new tool, the conducting glove “ConGlove.” This revolves around her musical interpretation of a written musical score during live performances.
The whole ConDiS process was a unique experience from beginning to end. From the first stage of fleshing out an idea for developing a tool to be used in a performance of one’s own compositions. Then composing a new piece specifically written for the use of the new tool “ConGlove.” Working with a conductor who practiced using this new device. Through to the final stage of performing Kuuki no Sukimaon six occasions. Each stage was a unique experience in and of itself. When looking back on the process as a whole, it is therefore natural to ask: What (generalizable) conclusions can be drawn from this process?
Musical Expectations and Physical Constraints
In my application to the Norwegian Artistic Research Program I wrote the following:
The expected result is that through musical material of a composer and the conductor’s musical gestures controlling digital technology, new sound worlds, new ways, new possibilities for musical performance will be explored. Practices that digital technology has to offer will be used to create a unique musical experience.
Expectations were high but not unrealistic. The ConDiS process addressed all of these factors, all of which were explored. Results have shown that the ConDiS system has:
- Opened up a new sonic world with unprecedented possibilities through composing, conducting, and performing interactive mixed music.
- Created new possibilities for composers through electronic notation and immediate synchronization of acoustic and electronics.
- Opened up new capabilities in the form of an extended role of the conductor.
- Created new possibilities through an uninterrupted interplay between the performers, the electronics, and the conductor.
Through ConDiS the composer can write the electronic part of a through-composed mixed music (ensemble music) composition with a precision and accuracy similar to that used in the notated instrumental score. This allows him better control of the expected sonic outcome. In spite of the fact that the ConDiS hardware—the ConGlove—affects and can limit the conductor’s natural conducting gestures, the conductor is in direct contact with the performers. Simultaneously, there is a direct connection with the electronics using conducting gestures and clicks of buttons on the glove. With the use of ConDiS, the “middleman” is cut out. There is no need for an additional “cue man.”
With ConDiS the performance, the interplay of instrumental and electronic sounds, gains additional musical meaning through the musicianship and musical interpretation on the part of the conductor. With the use of ConDiS, the coexistence between the composer’s score of electronic and acoustic sounds flows through the conductor’s musical gestures.
Experiments made during the development phase of the ConDiS research project proved that the desired outcome was achievable. Sound can be successfully and dynamically altered in an expressive way. Sound can convincingly be moved and conducting gestures can change tempo and sonority. There are limitations, however, and these limitations coalesce around the human ability to learn and memorize. How many different pieces of information can we process at once? Where are the boundaries? Was the volume control turned on or not; did I push the 2ndbutton or was it the 3rdor…?
With ConDiS the conductor not only has to deal with all the traditional information written in the score, but she must additionally learn and remember various instructions for controlling the new member of the band, the live electronics. With more practice, the human limitations will decrease, and this will be reflected in an increased control of the electronics. The ConDiS artistic research project has proven that ConDiS can successfully be built around traditional conducting gestures. I do not foresee a need to introduce any major changes in the way this style of music (through-composed) is conducted, though I am fully aware that more improvised styles would introduce entirely new and different scenarios.
Physical Consequences of the ConGlove
Using a device attached to the conductor’s body raises questions about physical consequences. It is evident that a supplement such as the ConGlove, despite being conceived as an extension to enhance the role of the conductor, is in some ways limiting and demanding. It limits the intuitive gestures of the conductor’s left hand, bringing with it not only physical consequences, but exercising a psychological impact on her concentration.
Closed/Open Hand for the on/off Function
When designing the ConDiS, it was fundamentally important to create a device that would have as little physical impact as possible. Thus various options were tested, including a baton, a glove with sensors that measured motion, and infrared technology (see table on page 41). For more on the results of these tests specifically, refer to the discussion above (See chapter 4). Based on the results of these tests I chose what I thought to be the most prudent direction. Using an x-OSC sensor enabled me to use a combination of high-performance analog/digital channels and onboard sensors (gyroscope, accelerometer, magnetometer) via OSC messages over Wi-Fi. This allowed me to use both gestures and physical touch-sensitive tools like bending sensors and buttons.
To minimize the constraints imposed by the ConGlove, the system is built to allow the conductor to turn the electronic control functions on and off by opening or closing her hand. That way when no action is required, the conductor does not affect the playback of the electronic sounds and can move her hand undisturbed, as long as she does not close her hand. Therefore, when the function is turned off, the conductor can use her left hand freely and in precisely the same way she does without ConDiS. Thus, there should not be any obstruction of conducting gestures involving the left hand.
In a conversation after the closing concert in Rockheim Halldis and I discussed, among other things, the success of this decision to allow an on/off function. It turned out that she felt that it was disturbing. The reason was that when she was able to conduct normally, her mind was still focused on remembering when to reconnect the system.
I had not considered this point of view and had actually believed that I had come up with a great solution. Following this conversation, I can better understand her perspective, and future work will involve finding a better solution, the specifics of which I will discuss below.
Use of Buttons
When the conductor needs to change the tempo or stop and synchronize playback, she needs to press different finger buttons. Pressing these buttons entails movements that are not natural to the conductor and, in that respect, new and initially unpleasant. I was aware of all of this from the very beginning and tried to find ways to mitigate or avoid these problems. Having the conductor push buttons was a simple, functional solution, but not 100% foolproof. What if the conductor forgets to press a button, or what if she does not push them properly? During the six performances of Kuuki no Sukima there were few incidents where pressing a button failed and the ConDiS system needed manual intervention and correction. Most of these incidents occurred during the first few performances, but they became less frequent, and during the final concert, only one incident took place which required manual assistance.
All of the failures are traceable to human error. The conductor forgot to press a button or failed to push firmly enough. At the Nordic Tour’s last concert in Copenhagen, the metronome key (second finger) began to emit incomprehensible tempo changes, and it was revealed after the concert that the soldering of the wire was broken.
Using buttons is therefore not a perfect solution, but it proved to be the most reliable. Other solutions tested were gesture recognition and learning (MuBuForMax – hhmm) and a means of measuring directional change, registering turning points (zero points) in vertical and lower movements of the right or left hands (Max/MSP). For further details on these, see Appendix A. They all proved to be less reliable than buttons due to the complexity of the conductor’s gestures.
When asked Halldis for her opinion on my solution using buttons, her answers were straight forward. Her experience was negative and she found it disturbing to have to press buttons that weren’t protected from accidentally being pushed. She also mentioned the difficulty involved in remembering which buttons to press and which buttons she pressed that would interfere with the operator. She suggested that I find a solution that reflects the reality of conducting practice, for example, having the conductor cue off or give a cue signal by pushing a pressure sensor with her fingertips.
Controlling Volume Value with Arm Movements
From the start, I believed the solution of having the conductor activate the volume level control by closing her fist and adjust the level with her arm was the best and simplest one. It is a well-known practice that conductors use their left arm to indicate volume changes—an upward motion for increased volume and a downward one for decreasing volume. This method proved satisfactory but, as Halldis stated in our conversation, the main drawback was that she could never be sure when she had reached the maximum or minimum level. The ultimate solution would be to build a system such that the maximum volume level is when the conductor has a maximum arm position—that is, the conductor’s arm held straight up—and minimum volume with the arm straight down.
Keep in mind that the position of the arm and volume value can vary. It is all based on the relative position of the arm meaning that the calculation relates to the latest arm position value. This relative arm position means that if the conductor deactivates volume control with 75% of its maximum volume value, it will stay at 75% when activated again — no matter [the] arm position. When the conductor activates the volume value control again, the arm is most likely not going to be in the same volume position as the last time the conductor deactivated the control. To have the computer remember the previous position is done to avoid an audible jump in the volume level.
The ultimate solution would be to build a system in which the maximum volume level is when the conductor has a maximum arm position, that is, the conductor’s arm is held straight up from body or (0°), and minimum volume with the arm straight down (180°).
The Weaknesses of the ConDiS system
Although the ConDiS system proved to be successful in that it fulfills my basic needs for expanding my compositional methods, it is far from being a perfect system. The following section provides my assessment of the factors that weaken the system and that need to be fixed in future updates.
The weaknesses of the ConDiS system can be summarized as follows:
- It is an object that needs to be attached to the conductor.
- It needs too many pre-performance operations, including turning on the system, calibrating fingers, and pressing buttons.
- It requires conducting actions that are disturbing and unpleasant for the conductor.
- It needs an additional inspector who manually intervenes if something goes wrong.
Having a hardware device that must be attached to the conductor is not a perfect solution.
When asked after the final concert about her experience using the ConGlove, Halldis openly expressed her opinion that the ConGlove was too bulky, used material that was too thick, and therefore hindered her conducting. She recommended a much thinner material be used that would include holes for the fingertips, which would make it easier to turn pages. She was also concerned about the size and weight of the battery. It interfered with and disturbed the way she naturally moved her arm.
The pre-performance operations that need to be completed before walking on the stage are disruptive and should be minimized. These operations include turning on the system, calibrating the fingers, and pressing various buttons. The perfect solution would be one that did not need any preparation and would turn on with an upbeat motion from the conductor at the very beginning of the performance. As mentioned above in the “Performance Preparation” section (p. 90), I managed to simplify these actions, but further development is needed.
The ConGlove requires specific actions during a performance that are not related to physical gestures intuitively used by a conductor. These actions include pressing buttons with different functions. Although the button solution was the most simple and reliable one that I could come up with, it is not ideal. It requires a certain amount of thought and abnormal motion. In my conversation with Halldis, she mentioned the button functions and recommended other solutions, such as pressing sensors (see “Use of Buttons” p. 120).
The ability of the conductor to synchronize the system by jumping to markers written in the DAW beforehand is not a perfect solution even if it worked well for this project. The best solution would be to synchronize the whole score according to the conductor’s movements. Another solution would be to have the conductor synchronize the music when she feels it necessary. Unfortunately, my attempts to implement these types of resolutions turned out to be too complicated or too unreliable to be of use in the ConDiS system. Despite its weaknesses, the ConDiS system did not prove to be too complicated, and Halldis and Arne Johansen (p. 74), who also experimented with the system, managed to use it after only relatively short periods of practice.
Conductors normally spend a substantial amount of time preparing with new compositions and in that context, the addition of ConDiS is less than ideal. The addition of ConDiS means they need to conduct an additional member of the ensemble, “the electronic sound extension,” or the DAW. They do so by following additional information written in the score. Preparation therefore mostly takes the form of practicing when to press which buttons and controlling the overall volume. They can easily incorporate this practice into their “traditional” preparation routine.
This does not mean that a conductor who is not familiar with the use of the ConDiS system would not need both instructions and practice before using it. But this process of preparation and practice is shorter than one might think. This of course depends on the musical style that is to be conducted since more improvised styles in which the conductor becomes a performer would allow or even require an entirely different way of conducting. Since the aim of the ConDiS research project focuses on through-composed mixed music, I will not explore that subject any further.
According to Halldis, she did not spend too much time practicing the functions, “perhaps a week or so,” with additional practice and learning taking place during rehearsals and concert performances. When asked about her preparation for the closing concert, held nine months after the five consecutive concerts of the Nordic Tour, she responded by saying that there had been no preparation time: “I was so used to the ConGlove that I just needed to put it on and start using it.”
The fact that the ConDiS system is not 100% reliable means that an additional person has to be available to manually “correct” any faults. This is mainly needed when the conductor forgets to press a button or does not press a button hard enough (see chapter 6). Then the “inspector” has to intervene and click the button manually via the DAW. This is a weakness of the system that needs to be solved in future updates.
One of the most critical challenges in the preparation of the ConDiS research project was to expand my compositional methods. Extended instrumental and electronic notation played a significant role in the process. I was fully aware that without expanding my notation language, my sonic world could not expand.
However, it is not enough to expand one’s collection of extended notation technique; one has to learn how to use it. I would need to be able to use it as clearly and fluidly as possible and use it often to make it second nature. This development can only be accomplished by exercising the musical language through music composition.
ConDiS and its electronic notation graphics opens up new possibilities for composers to write with more precision the interplay between live instruments and their accompanying electronics. That way the composer becomes more aware, more conscious of the electronic writing in the whole compositional process. In other words, the writing of electronic sound becomes a part of the whole or an integrated part of the orchestration. That integration by itself leads to new methods, opening up novel possibilities for written compositions as well as musical performances.
Kuuki no Sukima went further sonically than anything else I have done so far. My constant search for altered sounds, different sonic combinations, and ways to compose and perform mixed music gained a new dimension. In composing Kuuki no Sukima I felt I was getting closer to discovering the sonic gap that I believe is there somewhere “In Between the Air.” I realize that even though I have gained a lot from the success of the ConDiS artistic research project, there is much to be discovered still. Therefore, it is so important to continue my journey through the valleys of the endless spectrum of music. And it is so important that I continue my research into varieties of musical harmony. I know I have just opened up a new door and now I am eager to discover and further explore the musical spectrum of the valleys behind or “in between.”
Because the ConDiS project aimed to develop conducting tools for the performance of through-composed mixed music, I must emphasize here that all of the research work associated with the project was confined to that particular type of music. Other forms such as improvised or even semi-improvised music were not included. Different forms reveal different potentials for using ConDiS, as shown through my conducting of “Silence I Ask” at the closing concert in Rockheim. There I conducted a semi-improvised performance which does not require the accuracy of through-composed music. I was therefore able to freely improvise my interpretation of the electronic sounds that extended the performing voices. This kind of performance (use of ConDiS) is, by all means, more entertaining and to some extend more interesting for the audience. It has more of a visual relationship to what you are hearing, and it is definitely fun to conduct since you are also performing an instrument. Nevertheless, this type of performance was not what the ConDiS research project is about. “Silence I Ask” was added to the program in historical context to the first performance of the work in Rockheim 2013, and to show the potential of ConDiS
I believe that the ConDiS system has contributed in important ways to my artistic position at the national and international levels. This is particularly evident in the results of my research process, including my composition “Kuuki no Sukima” and its performances, and now this dissertation in textual form and its publicly available video components. The ConDiS system has broadened my composition potential. It has demonstrated new possibilities for music performance and it has proved it possible to extend the role of the conductor. The future development of the ConDiS system needs to be put in the hands of others—other composers, conductors, and performers. That way the ConDiS project will continue to contribute to the professional development of all types of mixed music performance, thus elevating new ideas and artistic innovation. I also foresee the ConDiS concept easily expanding its reach to cross over to other artistic fields, especially the field of multimedia.
While composing Kuuki no Sukimaand again when listening to the performances of the work, I constantly asked myself if I was exploiting the full potential of ConDiS. I can confidently say that I did. This is not to say that the end results exactly met my expectations at the beginning of the project, but the project was successful to the extent that it fully satisfied my intent to produce a compelling performance of my composition.
The ConDiS research project aimed to make a tool for use in conducting through-composed mixed music, meaning the conductor has to follow instructions provided in a written score. That alone significantly constrains other possibilities inherent in potential uses for the ConDiS system. This is especially evident in the fact that certain capabilities that I initially intended the conductor to have, such as pan and effects control, had to be eliminated for practical reasons.
As stated in the introduction, I am interested in writing through-composed mixed music, and so it was necessary for me to focus on that subject. The end effect of this is that the ConDiS research limited itself to the potential of the system within that specific framework. Hence the study did not address the full range of the possibilities for using the ConDiS system. As demonstrated at the closing concert in Rockheim during the performance of “Silence I Ask” for Voices, the possibilities for using the system are many and inspiring. And it is my hope and intent that all of these will be explored in future compositions.
At the beginning of the ConDiS project, I was aware of the fact that the potential capabilities of ConDiS could affect or change my compositional writing practices. I could even go so far as to expect a change in my compositional method. Early in the compositional process, it became clear to me that there was little need to change my style, I felt no obligation or desire to write a composition only to demonstrate the possible use of ConDiS. Instead, I felt I had a tool that was useful for fulfilling my “desire” to expand my sonic world. Therefore, Kuuki no Sukima was primarily written to fulfill my artistic ambition. Kuuki no Sukima is by no means a demonstrative composition; it is a serious composition fulfilling my artistic needs. This is in and of itself a successful result.
Achievements
Experiments made through various short compositions—“Etudes”—during the development phase of the ConDiS research project proved that it would be possible to achieve the desired results. There are several options for incorporating the electronic sound extension of the acoustic instruments graphically into the traditional musical score. Sonic changes can be written successfully into the musical score as graphical symbols similar to other traditional symbols such as volume change indications (crescendo/diminuendo p or f). Button-pushing symbols are easily understood, as are the panning symbols.
During a performance, sound can be successfully and expressively dynamically altered. Sound can be moved in a convincing manner and conducting gestures can change tempo and sonority. However, there are limitations in the form of the human ability to learn and memorize, the limits of the human brain. How many different types of information can we process at once? Where are the boundaries? How can the conductor avoid a rush of questions in her mind during the performance: Was the volume control turned on or not? Did I push the 2nd button or was it the 3rd, or…?
A variety of experiments and practical performance exercises provided invaluable information regarding these questions that informed the research and development process. They resulted in changes that were driven by technical limitations, human psychological congestion thresholds, and aesthetics. The project was partially reconstructed and simplified.
Due to technical limitations, indented features such as facial recognition and other gestural recognition features had to be left out. Given human limitations and the tradition of conducting, the final version of ConDiS eliminated the conductor’s control of both panning and effects. Even though I accept that these functions are not part of the conductor’s conventional job, my imagination is still tickled by the possibility of offering the conductor those capabilities. Another decision was made to exclude all innovative and exotic conducting gestures, this time for aesthetic reasons. This decision demonstrates to me that ConDiS can be built successfully within the limits of traditional conducting. Similarly, but for musical reasons, a decision was made not to include any extra-musical conducting show-off gestures.
When using the ConDiS system, the conductor not only has to deal with all the traditional information written in the score, but she must additionally learn and remember various symbols and graphic representations to lead the new member of the band, the live electronics. With more practice, the human limitations threshold will become lower, reflecting an increased control of the electronics, as was demonstrated during the five concerts of the Nordic Tour. With more practice, the conductor’s job can expand even further.
I am fully aware that ConDiS is not a complete system for live digital conducting of mixed media performance. I am also aware that ConDiS might never—perhaps should never—become a complete system due to the fact that constant technological innovation offers endless new developments and changes. But it is an important step to unify the interplay needed for mixed music performances. It is an important step to add the DAW as a new member of the orchestra.
With time the conductor becomes a part of the whole: the musical score, electronics, and the ensemble. With time, conducting electronics becomes as natural to the conductor as conducting the “other” instruments.
Future work
Cooperative Development
Further development of the ConDiS system is already planned, including a new design of the ConGlove. This will be based on the experience gathered over the course of the ConDiS project and the suggestions provided by conductor Halldis Rønning. Towards that end, I have already had meetings and correspondence on artistic and technical cooperation with persons related to innovation in the arts. In these, we have focused on our mutual interest in expanding the abilities of the glove technology in multimedia performances, including the visual arts and dance.
Cooperative Artistic Work
Not only will further development of ConDiS continue, so will the ongoing development of electronic notation and conducting instructions. The potential inherent in notation and instructional techniques will enhance and expand ConDiS’ potential, from writing ensemble works to use in multimedia performances. These developments will together reveal the capabilities or limitations of the conductor to control individual sonic groups of performers at a single time.
Thanks to a grant from the Norwegian-Icelandic Cultural Cooperation Fund, preparatory work on a new cooperative multimedia production (working title “Black Obsidian”) has already begun with conductor Halldis Rønning. In our application to the fund, the project is described as a “cooperative multimedia production of composer/conductor Hilmar Thordarson and conductor/composer Halldis Rønning. They will create a work together and cooperate with two excellent ensembles, the Icelandic Caput Ensemble[1]and the Norwegian IMNO ensemble to realize the end project “Sort Glimmer.” They will make use of the ConDiS technology glove to create and manipulate the electronic sounds and visual art design. The cooperation will consist of workshops with the two ensembles in both countries and eventually performances in both countries to be held at art festivals in 2020.”
The composition will be written in a more open form: a through-composed composition, semi-improvised and improvised. Workshops will be used to engage in experimentation with mixing the tradition of written composed music with improvisational instructions. This will further enhance and expand the potential of ConDiS, giving the conductor an excellent opportunity to test her ability and limitations. In this process, the conductor will be more involved in making her own artistic decisions. She will be able to “exform” them with a clear sense of perception or interpretation and creatively apply and sculpt real-time DSP to what is being played. This will allow the conductor not only to conduct the written instructions, but also to sculpt the performance and, by doing so, function as a co-composer and performer as well.
In order to further develop the ConDiS concept, i.e. to extend the conductor’s traditional conducting responsibility, it is necessary to involve more composers and conductors. Without this, a project such as this will end up like so many others have, fading away gradually or limited in its use to its creator alone. To avoid such a fate, workshops are scheduled in collaboration with ensembles and academic art institutions to offer young, talented composers the opportunity to compose and conduct music for the ConDiS system. The likelihood that the ConDiS concept and tools will be able to put down firm roots is significantly improved when it can be used and experimented with by the youngest generation of musicians.
[1]http://caput.is