Music Technology

Main content

Recent sensor technologies provide a vast amount of new possibilities and developments in the field of music. But one area is nearly untouched so far: Musical instruments.
Although it might sound a little bit unromantic, playing and exercising a musical instrument and making music is a difficult task, similar to training in sports. In the latter, technology based training assistance is already common. To rise supportive technologies for musical instruments, several projects and experiments are carried out in our laboratories. With several technologies measurements are conducted to receive objective and individual data of musicians and live performances. With these, investigations regarding e.g. coordination, fatigue and general performance monitoring is possible.
Beside this, new interfaces, especially based on sensor technologies are developed and evaluated. This could lead to new musical interfaces and feedback methods or to augmented traditional musical instruments. (Contact and recent publications click here).

Some reports about our music project are available at SRF, ETH News and a recent short docu with Prof. Z. Bron and E. Nick on 3SAT 3SAT.

Members

  • Dr. David Banach
  • Dr. Alberto Calatroni
  • Matthias Flückiger
  • Tobias Grosshauser

Former Members

  • Andreas Hösl
  • Dr. Bernd Tessendorf
  • Dr. Martin Kusserow

Sensor Technologies for Musical Instruments

From a technical point of view, instrumental music making involves mainly audible and visible playing parameters. Parameters like force, pressure and fast movements, happening within milliseconds are particularly much more difficult to capture. Our research is focusing on these parameters additional to movement and audio analysis. To achieve this, several sensors are integrated unobtrusively into and fixed on the musical instrument. The evaluation of these new sensor setups is done with amateur musicians, students of the ZHDK and professionals. A further goal of the use of sensors is a more simple integration of conventional musical instruments into electronic music environments by converting the data to MIDI and OSC.

Audio-Haptic Modalities in Musical Interfaces

  • Acceleration measurements of bow-string interaction
  • High-speed video analysis of bow-string interaction
  • Classification algorithm for Cello sounds
Violin

Sensors for Stringed Instruments

  • Left hand finger position and pressure measurement; bow postion, acceleration and speed measurement; orientation and position of the violin and bow measurement.
  • Bow postion, acceleration and speed measurement.
  • Orientation and position of the violin and bow measurement.
Drums

Sensor Technologies for Drums

  • Sensor based high speed synchronization and coordination measurement
  • Optical drum recognition
  • Regularity and rhythmical precision and stability measurements
Keyboards

Sensor Technologies for Keyboard Instruments

  • Finger position and pressure measurement on piano keys
  • Key angle measurement
  • Coordination and synchronization between left and right hand fingers and within music groups
Trumpet

Sensor Technologies for Wind Instruments

  • Lip pressure and direction of pressure measurements
  • Wind instrument position and orientation measurement, in general posture recognition
  • Physical load while playing or holding the instrument

Supportive Technologies for Exercising, Teaching and Learning

Exercising is a daily routine in learning a musical instrument. It is one of the most important tasks but very little attention is drawn to it. But there is a high potential to increase the efficiency and to learn faster. Beside many others, this could be obtained/reached with new sensor and feedback technologies.

Force sensors
Force sensors to capture pianists' finger position and pressure on the piano keys.

The following overview shows the latest of our developments in sensing different parameters for most musical instruments:

Table
Overview of the sensor technologies available for musical instrument sensing. x = tested, (x) in evaluation, 0 = in development.

Research and Applications in Musical Instrument Playing

Fatigue and Malposition Detection

Musicians’ practice overly long placing themselves at the risk of performance and practice shortcomings and bodily pain. With a set of sensors consisting of 6 or 9 DOF IMUs fixed to the instrument, the orientation and movement of the violin was measured. We compared individual perceived fatigue to the motion of the body and the instrument. In short, professional musicians showed more stability regarding their posture and more endurance during a one hour practicing session.

Sensor on violin
Wirless orientation sensor fixed on a violin to capture motion and orientation of the instrument.

Synchronization (Music Groups)

Several experiments were conducted with two or more musicians playing together. In a violin duo, the timing between the bow and finger changes for each player were captured. Similar measurements are done with trios and quartets. The synchronization in professional music groups was constantly below 15ms, amateur groups showed delays up to 100ms. 

Multi-modal Real-time Feedback

Several feedback modalities are implemented to communicate with the musician while playing. Visual feedback is shown in Figure above, the background color changes, if a certain event occurs, but also haptic feedback is integrated into e.g. violin bows to indicate cramping or wrong finger positions.

App and wireless sensors
App and wireless sensors for data visualization and real-time feedback.

New Music and Musical Instruments

Beside the pedagogical applications, the technologies enable musicians and composers to augment the traditional musical instruments. The main advantage is the low latency data capturing of the above mentioned parameters (finger/hand position and pres-sure, etc.) and sending them in real-time via osc or midi.

This means:

  • New parameters for musical expression: Several parameters e.g. finger pressure, finger position can be mapped to any sound re-lated event like panning, volume, effects, MIDI..., or to any other objects like light or laser shows (e.g. via DMX).
  • Low-latency MIDI or OSC-Interfaces on nearly any traditional musical instrument.
  • All Technologies can be integrated into electrical instruments (e.g. E-guitar) or silent instruments.
 
 
Page URL: http://www.ife.ee.ethz.ch/research/music-technology.html
Tue Apr 25 15:49:46 CEST 2017
© 2017 Eidgenössische Technische Hochschule Zürich