Recent sensor technologies provide a vast amount of new possibilities and developments in the field of music. But one area is nearly untouched so far: Musical instruments.
Although it might sound a little bit unromantic, playing and exercising a musical instrument and making music is a difficult task, similar to training in sports. In the latter, technology based training assistance is already common. To rise supportive technologies for musical instruments, several projects and experiments are carried out in our laboratories. With several technologies measurements are conducted to receive objective and individual data of musicians and live performances. With these, investigations regarding e.g. coordination, fatigue and general performance monitoring is possible.
Beside this, new interfaces, especially based on sensor technologies are developed and evaluated. This could lead to new musical interfaces and feedback methods or to augmented traditional musical instruments. (Contact and recent publications click here).
- Dr. David Banach
- Dr. Alberto Calatroni
- Matthias Flückiger
- Tobias Grosshauser
- Andreas Hösl
- Dr. Bernd Tessendorf
- Dr. Martin Kusserow
Sensor Technologies for Musical Instruments
From a technical point of view, instrumental music making involves mainly audible and visible playing parameters. Parameters like force, pressure and fast movements, happening within milliseconds are particularly much more difficult to capture. Our research is focusing on these parameters additional to movement and audio analysis. To achieve this, several sensors are integrated unobtrusively into and fixed on the musical instrument. The evaluation of these new sensor setups is done with amateur musicians, students of the ZHDK and professionals. A further goal of the use of sensors is a more simple integration of conventional musical instruments into electronic music environments by converting the data to MIDI and OSC.
Audio-Haptic Modalities in Musical Interfaces
- Acceleration measurements of bow-string interaction
- High-speed video analysis of bow-string interaction
- Classification algorithm for Cello sounds
Sensors for Stringed Instruments
- Left hand finger position and pressure measurement; bow postion, acceleration and speed measurement; orientation and position of the violin and bow measurement.
- Bow postion, acceleration and speed measurement.
- Orientation and position of the violin and bow measurement.
Sensor Technologies for Drums
- Sensor based high speed synchronization and coordination measurement
- Optical drum recognition
- Regularity and rhythmical precision and stability measurements
Sensor Technologies for Keyboard Instruments
- Finger position and pressure measurement on piano keys
- Key angle measurement
- Coordination and synchronization between left and right hand fingers and within music groups
Sensor Technologies for Wind Instruments
- Lip pressure and direction of pressure measurements
- Wind instrument position and orientation measurement, in general posture recognition
- Physical load while playing or holding the instrument
Supportive Technologies for Exercising, Teaching and Learning
Exercising is a daily routine in learning a musical instrument. It is one of the most important tasks but very little attention is drawn to it. But there is a high potential to increase the efficiency and to learn faster. Beside many others, this could be obtained/reached with new sensor and feedback technologies.
The following overview shows the latest of our developments in sensing different parameters for most musical instruments:
Research and Applications in Musical Instrument Playing
Fatigue and Malposition Detection
Musicians’ practice overly long placing themselves at the risk of performance and practice shortcomings and bodily pain. With a set of sensors consisting of 6 or 9 DOF IMUs fixed to the instrument, the orientation and movement of the violin was measured. We compared individual perceived fatigue to the motion of the body and the instrument. In short, professional musicians showed more stability regarding their posture and more endurance during a one hour practicing session.
Synchronization (Music Groups)
Several experiments were conducted with two or more musicians playing together. In a violin duo, the timing between the bow and finger changes for each player were captured. Similar measurements are done with trios and quartets. The synchronization in professional music groups was constantly below 15ms, amateur groups showed delays up to 100ms.
Multi-modal Real-time Feedback
Several feedback modalities are implemented to communicate with the musician while playing. Visual feedback is shown in Figure above, the background color changes, if a certain event occurs, but also haptic feedback is integrated into e.g. violin bows to indicate cramping or wrong finger positions.
New Music and Musical Instruments
Beside the pedagogical applications, the technologies enable musicians and composers to augment the traditional musical instruments. The main advantage is the low latency data capturing of the above mentioned parameters (finger/hand position and pres-sure, etc.) and sending them in real-time via osc or midi.
- New parameters for musical expression: Several parameters e.g. finger pressure, finger position can be mapped to any sound re-lated event like panning, volume, effects, MIDI..., or to any other objects like light or laser shows (e.g. via DMX).
- Low-latency MIDI or OSC-Interfaces on nearly any traditional musical instrument.
- All Technologies can be integrated into electrical instruments (e.g. E-guitar) or silent instruments.