The ConDiS project is an artistic research project, not a technical research or engineering project. The following chapter focuses therefore on a practical exploration of why the x-OSC was chosen and how the sensor can be used to capture conducting gestures, as is appropriate for my application.

Choice of Sensors

The types of sensors available now is more varied than ever before, and they are becoming more affordable. Given this fast-evolving technology, one must keep in mind that what is news today may be yesterday’s news tomorrow.

To select the type of sensor best suited to the idea behind the Conducting Digital System, the following criteria were proposed:

• Wireless technology;
• High speed and versatile communication —> OSC protocol for accuracy and flexibility;
• Simple and small enough to use but sophisticated enough to fulfill the needs;
• Reliable and not affected by external interference, such as stage lights or human sweat;
• Comfortable/natural for the conductor to wear;
• Low-cost technology.

The following sensors were tested: The Leap Motion Controller, Myo Gesture Control Armband, HotHand, Qualisys Motion Capture Systems, Xbox One Kinect 2.0 Sensor, and x-io Technologies’ x-OSC sensor.

Brand	Tech. type	Simple/Complex	Reliability	Comfort	Price
Leap Motion	USB	_	_	+	+
Myo Armband	Infrared	+	_	+	+
Hot Hand	USB	_	+	+	+
Qualisys	Infrared	_	+	_	_
Kinect 2.0	Infrared	+	? Not enough time to test	+	+
x-OSC	OSC	+	+	+	+

Table 1. Evaluation of sensors. + means positive result, – means negative result.

I decided to use the x-OSC for the ConDiS Conducting Digital System for the following reasons:

• Simple to use – The OSC offers various implementations, including real-time sound and media processing environments, web interactivity tools, a large variety of programming languages and hardware devices for sensor measurement.

• Reliability– It uses the Open Sound Control (OSC) protocol for communication between the sensor and the computer. The advantages of using OSC include interoperability, accuracy, and flexibility.

• Comfort– The x-OSC is a relatively small Wi-Fi I/O board that fits comfortably in the hand.

• Price– The x–OSC wireless I/O board is priced within the limits initially set out for creating an inexpensive, fully functional system.

Exploration of the sensor in terms of the project’s practical applications

As stated in the previous chapter, there are numerous existing studies on conductors’ movements. Nevertheless, it was very important for me at the start of this research to better understand in a physical way the response of the x-OSC I/O board to different conducting gestures. The best means of doing this was to physically conduct the electronic sounds myself and explore responses based on various (traditional, non-traditional) conducting gestures. By taking a “snapshot” of these gestures, I could better understand and feel the relevance and robustness of the gestures in conjunction with their conducting function. I was thus in a position to judge their potential for use.

Traditional conducting gestures

The basic traditional conducting gestures are gestures indicating the desired volume, meter, and tempo. With hand gestures, the conductor traditionally indicates an increase or decrease in volume by lifting her arm(s) or making larger (for louder) and smaller (for softer) gestures. The conductor uses different arm gesture patterns to indicate the written time signature—e.g., ,  —and variable speeds of the arm gestures to indicate the written tempo or metronome. In the next section I describe the test used to find out if it would be possible to use the x-OSC sensor to recognize and learn various traditional conducting gestures.

 

Up/down motion

Holding a hand open and raising an arm up and down (palm facing the ceiling on the way up and the floor on the way down) produced the following patterns:

Figure 13. Arm slowly up/down. X-axis peaks at turning points

 

Raising a hand slowly, as shown in Figure 13 the X-axis (red) indicator showed a gradual move in the direction of the arm, moving up and down. Turning points are also traceable in the form of high and low points. The Y-axis (blue) showed a small but gradual movement in the direction of the arm but apparent peaks at turning points. The Z axis (green) showed an up/down movement when the arm was raised slowly and a down/up motion when lowering the arm. Fast moves up or down were shown at the turning points.

Figure 14. Arm fast up/down. X-axis peaks at turning points

 

When the arm was raised quickly up and down (Figure 14), a slightly different picture was revealed, especially at the Y- and Z-axes, whereas the X-axis was mostly identical to the case for slow movement. The only notable difference was a minor deviation especially at high points, which was probably caused by computer latency. The Y-axis showed an up/down motion when the arm was raised and a down/up motion when lowering the arm with no peaks at turning points. The Z-axis showed up/down motion on the way up and down/up motion on the way down with no peaks or change of direction at turning points.

Assessment of sensor to capture up/down motion

The accelerometer worked extremely well for measuring the down motion of the hand. The result was very stable with respect to whether the arm was moved, especially the X-axis movement. This provided good enough resolution for use in calculating the arm position in space. Therefore, a decision was made to use the up/down gesture recognition for Volume and Effect control.

4/4 beat conducting gesture – Beat/Tempo

Conducting standard 4/4 beat gestures in two different tempos—fast and slow (metronome 60 for slow and 120 for fast)—resulted in the following patterns:

Figure 15. Metronome 60. Counting clear 4/4 pattern

 

As shown in Figure 15, all the axes X, Y, Z showed traceable patterns when conducting a precise 4/4 beat pattern at a relatively slow tempo, 60 bpm. The X-axis was peaking at every beat (1, 2, 3, 4) while the Y-axis showed reverse motion. The X-axis shows each beat as clear peak points. The Y-axis shows reverse motion with peak points at upbeat to first, second, third, and fourth beat and a definite low point on the first beat. The Z-axis is moderately clear, showing peak points at every beat, although the first and second beats are not especially clear:

Figure 16. Metronome 60. Counting less clear pattern

 

When conducting the same tempo with a bit more of a “natural” style (Figure 16), i.e. a very clear downbeat with the remaining second, third, and fourth beats not as strict or more flowing in style than in the first example, the patterns became less clear. The first beat in particular could easily be confused with the fourth beat:

Figure 17. Metronome 120. Counting clear 4/4 patterns

 

When conducting clear beats in a relatively fast tempo of 120 bpm, the patterns became much more blurred, as illustrated in figure 17. The upbeat to first beat was still evident, especially on the Y-axis. As before, the X-axis showed peak points at first and second beat, though these were much blurrier than before. Also as before, the Z-axis was somewhat obscure and irregular:

Figure 18. Metronome 120. Counting unclear 4/4 patterns

When conducting in more fluid gestures but still with strict upbeat and downbeat to the first beat, the first beat was clearer, but the whole pattern became blurrier (Figure 18). This happened since the up/down beat to the first beat was accented while the other beats (second, third, and fourth) were less accented or even not at all.

Assessment of sensor ability to capture metric motion

As expected, conducting a straight and clear 4/4 pattern provided acceptable results in the form of repeated patterns with peaks on each beat. When conducting more freely, the patterns started to get blurry. The same happened with increased tempo, with a faster tempo resulting in less predictable patterns. Using the MuBuForMax-hhmm program learn the conducting patterns resulted in precision rates between 50% and 80%, an unsatisfactory result in light of the artistic goal of the ConDiS system.

It is simply a fact that conductors like to use expressive conducting gestures for conducting tempo, meaning they insist on being able to use patterns that go beyond the strict metric gestures. They are musicians too and they need to express themselves freely, as conductor Halldis Rønning clearly states in the interview appended to this dissertation. For this reason, the results when conducting a 4/4 beat revealed that it would be impossible to use the MuBuForMax–hhmm learning feature. Another solution would have to be found that would allow the conductor to conduct tempo as freely and musically as possible.

Circular motion (non-traditional gesture)

Although circular motion is not a part of the conductor’s vocabulary in the way the up/down motion and metric patterns are, it was necessary to ascertain if this type of motion would be useful in realizing the original idea of having the conductor control the panning of the electronic sound. The notion was that if the conductor made a gesture significantly different from any traditional ones, there would be no confusion between the conductor and the performers. Circular motion probably best satisfied this requirement since it is hardly used as a traditional conducting gesture. One potential use for circular motion was to control spectral location of a sound.

 

The objective was to detect circular motion with the arm held up above the head and moving clockwise around the head:

Figure 19. Left arm overhead moving 90° per sec.

Circular motion with the left hand over the head, the palm facing right and moving one circle 360° per 4 sec. or 90° per 1 sec., or the equivalent of a tempo of 60 bpm. As can be seen in figure 19 the pattern is very clear and linear on all the axes. The X-axis shows the high point at 0° and the low point at 180°. The Y-axis has a high point at 180° and a low one at 0°, or a phase of 180° from the X-axis. The Z-axis point is in a +- 90° phase from the other axes with high points at 90° and low points at 270°.

Assessment of sensor ability to capture circular motion

Moving the arm in a circular motion provided a clear and reliable result, certainly good enough to be used in the early development stages of the ConDiS system. It was used, for instance, when Arne Johansen conducted the Jonsvatnet Brass (p. 74). Later I decided not to use it for the final version for aesthetic reasons. Having the conductor moving his arm in circles proved to be a very distracting gesture that seemed to have more to do visually with the rodeo than contemporary music performance. I later determined that holding the arm straight out and twisting it from left to right gave a similar result.

Circular motion with the left hand 90°out from the body moving left-right, right-left.

The ConDiS system and its Graphical Interface

Figure20. The ConDiS Graphical Interface

Figure 20 shows the graphical interface of the ConDiS system (red highlighted squares) as used in conjunction with the Ableton Live digital audio workstation (DAW). The system is originally written in Max/MSP and Max for Live.

1. The accelerometer of the x-OSC sensor. The blue column (Y-axis) is used to sense the position of the conductor’s left arm, which is in a high position in this illustration.
2. The “ConGlove” finger bending device. Senses finger signs to activate various functions of the system.
3. The synchronization and metronome device. Senses the button-clicking function of the “ConGlove.”
4. The volume control device. Senses the arm position of the conductor. If the arm position is high, the sliders are high. Conductor can select which instrumental group she activates by straightening out her finger or closing her hand to activate all.
5. The metronome of the DAW related to the metronome button function of the ConDiS system.
6. The Conducting track. If selected (as it is in the picture), the ConDiS device board is visible.
7. The instrument tracks showing instruments either as groups, e.g., woodwind, percussion, and strings, or as individual instruments. Fig. 20 shows the flute.
8. The “automated” effect track of the flute. It is showing the automated reverberation of the flute track.
9. The “automated” surround track of the flute. Showing how the flute sounds move in space.
10. The “markers” written in the DAW that relate to the same markers (numbers) written in the score. This is the backbone of the ConDiS synchronization function, as pressing the jump forward button would move the play head of the DAW to the next marker while jump backward takes it back to the previous marker.

The ConDiS Software

ConDiS is intended to be accessible to others without the presence or involvement of the developer. It is intended to be flexible and open for personal adjustments and/or individual experimental development, an essential component in the artistic philosophy behind the ConDiS project. Therefore the decision was made to use Max/MSP, Max for Live, and Ableton Live, a commercially available software that is user-friendly and yet flexible enough to fulfill the artistic needs of the project. This software is commonly used by composers, performers, software designers, researchers, and artists to create various forms of artistic performances and installations. This is an important factor in choosing appropriate software. The following is a brief description of the ConDiS system, its interface, and the Max/MSP graphical programming as it appears when used in conjunction with the Ableton Live digital audio workstation (DAW). For more detailed instructions on the practical use of the interface, see Chapter 6, Performance Preparation.