La Selva Virgen, en la Selva Oscura

A collaboration with Pedro García-Velásquez

I worked with composer Pedro García-Velásquez to help him design a robot ensemble for one his projects. The ensemble consists of a dozen of robotic arms holding mallets and sticks that can hit small percussion instruments and sculptures made of rocks and blown glass, designed by sculptor Marion Flament .

Five Raspberry Pi control the servomotors of several arms each, using PWM outputs. The Raspberries receive gestures commands (either from a remote laptop, or from one of the other Raspberries) as OSC messages, such as /marimbaG3 or /glockenspielF4, and then convert these commands into sequences of PWM changes to perform the desired gestures.

The ensemble is designed to be modular and easily installed over a fairly large area with little infrastructure, as it had to be suitable to several scenarios:

• A first concert at Fondation Singer-Polignac , where the ensemble was both played live from a MIDI keyboard, and from prerecorded MIDI sequences.

• A second concert at Basilique de Saint Denis, in a chamber music configuration with ensemble Le Balcon , where the robots were played live with other musicians.

• An autonomous installation at Fondation Singer-Polignac, where the robots were playing fragments generated on the fly at fixed hours of the day.

You can see here an excerpt from the concert at Basilique de Saint Denis:

And here is the installation at Fondation Singer-Polignac:

In order to ease the installation and allow experimenting with the spatial distribution of the robots, the robots are connected over a wireless network. Thus the two main problems we had to solve were the automatisation of the configuration process, and the handling of the latency, especially in live scenarios.

Autoconfiguration

The robot ensemble relies on a library I wrote as part of my research. This library offers support for service discovery, leader election, clock synchronization and a publish/subscribe message passing mechanism, as well as OSC messages serialization. The technical details of the library are described in this paper published at JIM2020 (in French), but here I will focus on how the library has been leveraged to help with this particular project.

Service discovery

The Raspberries use the python bindings of the library to be automatically discovered and subscribe to a named channel, which allows to dispatch messages to the robots using memorizable names that are stable across network configuration changes. The service discovery module is also exposed through a Max/MSP object, which is used to list all online robots and address them from a Max/MSP patch on a remote laptop. That patch converts MIDI messages from a keyboard or a prerecorded sequence to OSC messages and dispatch them to the robots using the same external.

The translation from MIDI to OSC is specified by a simple text file associating each MIDI note to a robot name and an OSC message, like this:

Clock synchronization

Upon joining the network, the robots also start a leader election to pick a clock leader. The robots can then send synchronization requests to the leader to estimate their time offset, and adjust their own clock accordingly. The synchronization protocol is inspired by NTP, and uses the same kind of latency and jitter mitigation algorithms, but leaves aside the inherent complications of the NTP infrastructure, such as multiple servers and layers or encryption. It also uses a user-level software clock, that allows the synchronization process to run without needing special privileges and without interfering with other system services. This auto-configured synchronized clock is used in the latency handling described below.

Latency handling

There are two main kinds of latencies that add up in the system:

• The network latency. This is the time during which OSC packets travel from the leader to a robot. It is subject to some variability over time, but more importantly, it is subject to some variability between two robots, which can degrade synchronicity between musical events. It was in the order of tens of microseconds in our use cases.

• The movement latency. This is the time that the arm takes to travel from its current position to the impact once a command has been received. This latency is in the order of a few hundreds of milliseconds, so it dominates the network latency. It also exhibits variability due to a number of factors, which makes it difficule for a musician to play in time.

These latencies can not really be eliminated, but the system can be designed to remove as much variability as possible, in order to both preserve synchronicity of musical events, and allow the instrumentist to anticipate a fairly stable delay.

Network Latency

The network latency is aligned to a known latency using the clock synchronization service. Messages sent to the robots can carry an execution date. Upon reception, the message delivery to the next processing stage (ie the layer responsible for moving the robot's arm) is delayed until the synchronized clock of this robot reaches the specified date. By setting the execution date to an appropriate time in the near future (say 10 or 15ms from the current date), one can absorb most of the network jitter and maintain synchronous delivery of simultaneous musical event to the gesture layer.

Movement Latency

The movement latency varies depending on the geometry of the instrument being played, on the characteristics of the servomotors and the arm, on the weight of the mallet, and on the previous gesture's end position. This last parameter is the most annoying, because it means that the latency of the exact same gesture depends on what other gesture has been played before. It is especially problematic in a live play situation, since it is harder for a musician to adapt to that kind of varying latency than to learn a fixed latency.

The gesture layer is a python script that loads a dictionnary of gestures specific to each robot. The dictionary keys are OSC messages addresses, and the dictionary entries are lists of commands describing the gesture to perform. Each command is a list starting by a string, followed by optional arguments:

The command specify the desired positions of the various servomotors, along with timing and alignment informations:

• ['servo', n, angle] is used to specify a servomotor desired angular position. It converts angle to the corresponding PWM duty cycle and sets pin n of the Rapsberry Pi to that value.
• ['delay', d] can be used to wait for a predefined amount of time of d seconds.
• ['complete'] is used to wait all previous 'servo' commands to complete. It uses the last known position of each servomotor and a very simple kinetics model of the servomotors to predict the time taken to reach all desired positions.
• ['align'] indicates the alignment point of the gestures, which is generally the time where the mallet should hit the instrument.

The gesture layer keeps track of the position of each servomotor. When receiving a gesture message, it looks it up in the gesture dictionary to find the command sequence. It then goes through the sequence, predicting the time for each 'servo' to complete, and converts each 'complete' command to a delay value. It also accumulate delays until the 'align' point, and uses this to compute the amount of time it needs to delay the execution of the whole sequence in order to align the mallet impact to a fixed latency.

This allowed us to set the total key-down-to-impact latency to a stable 200ms, which seems quite a lot, but but was no big deal for our experienced instrumentist to learn his way around, allowing him to play in time with other musicians.