Nox Glove

In this post I present a prototype I’ve been working on in collaboration with saxophonist Remi Fox. It is a sensor-equipped glove aimed at tracking a saxophone player’s hand gestures. The position and motion of the fingers is sampled by a microprocessor via sensors tied to the glove, and this info is then transmitted wireless to a computer where it can be mapped to MIDI controllers or used to track musical notes. The player can then trigger sounds, control effects, play synthesizers, and so on, without fussing with a keyboard.

So, the constraints of the system are the following :

• The device must induce the least possible constraints on the player’s hand. It must be lightweight and comfortable enough to be worn for the duration of a concert, jam or studio session.
• It must produce position / motion data stable and accurate enough to infer musical notes and allow for precise continuous control of effects.
• The delay in the sensing and transmission chain must be low enough to produce a feeling of immediate reaction to the player’s gestures.
• It must cope with the constraints of live show : presence of high contrast and dynamic lighting, heat, magnetic interferences, noisy environment, physical obstacles on stage, etc…

We did a quick survey of the existing solutions. The vast majority of general-purpose solutions is in the field of video games and VR, and some more specialized products can be found in medical fields and robotics. Even if connected gloves date back to the late 80’s (with Nintendo’s Power Glove being a famous one), they generally lack some features required by our intended application:

• Widespread flex/bend sensors based solutions still suffer from a low accuracy and reproductibility. (IMU-based solutions are much more promising in this regard, but a little more complex to implement and use.)
• Most gloves are still relatively thick, some of them cover the whole fingers, leading to a loss of dexterity.
• Older solutions interface to a computer through serial link with a cable, while newer wireless ones are prone to Bluetooth or Wi-Fi lags.
• Typical game applications do not cover the whole range of possible musical gestures, focusing on more likely game interactions such as prehension and pointing fingers. Some complex movements, some joints, or even whole fingers might be ignored.

To get a more concrete grasp on the problem we built a first prototype using flex sensors glued between two layers of hook-and-loop fasteners. This way the sensors can be placed at our convenience on a glove, which allows to test for various joints combinations. The sensors are wired to a small PCB featuring simple voltage dividers, the output of which is fed to the ADC of a micro-controller. The PCB, the micro-controller, along with a lithium-ion battery are packed together in a plastic box which is attached to the back of the hand. The micro-controller can act as a WiFi access point, or try to connect to an existing access point, all of which can be configured from the host computer. Once the network is established, the micro-controller simply read the ADC at a fixed rate, perform minimal calculation to infer a ‘bend amount’ and broadcast the data in the form of UDP packets.

I wrote a small program to configure the glove and to plot the received values and ‘calibrate’ the sensors, which for now means essentially determining a high and low threshold used for musical notes detection, which is very rudimentary, but works well as a first guess.

I then wrote a Max/MSP external which is embedded in a Max for Live plug-in and can connect to the glove and receive the flex sensors data. You can specify a calibration file created from the configuration application, and map MIDI notes to fingers. Then you can plug-in a virtual instrument on the same track and off you go !

There are several lessons to be gained from that first prototype :

• Flex sensors are very noisy and you have to filter them to achieve good results. In the process you might loss rapidly changing information. Another problem with flex sensors is that it difficult to achieve good reproductibility of measures for the same angle of flexion. Also, the measure tend to over-react to quick changes : say we bend from angle 0° to angle 45°, the sensor will show a peak well over 45°, then it will slowly decrease to a stable accurate value.
• Some gestures of the saxophone player can’t reliably be detected with flex sensors (even if they were accurate). Fingerings involving the pinky finger are mostly executed with a small rotation of the wrist, pushing the finger on the key, instead of bending it. Abduction moves are also hard to sense with flex sensors because of their small variations and the fact that the amplitude of abduction required to reach a given key depends on the position of the whole hand, including the rotation of the wrist.
• Flex sensors do induce a small amount of rigidity in the glove, which causes some discomfort, although it was less than what we had expected.
• Wi-Fi may not be the best solution during live performance. Although the delay seems to be reasonable in most cases in our configuration, with a round trip of about 3 to 8ms, there is no guarantee against sudden spikes (and it did happen a few times), hanging the system for up to 100ms, which is way too much and would cause out of sync detection. However, prior experience with bluetooth seems to place it in a considerably worse position. Other wireless solutions might have to be tested for our purpose.

We are now working on a second prototype based on 9-DoF IMUs to overcome the limitations of flex-sensors, so, the story is to be continued…

Thanks for reading !

Martin