Handmade Network Wheel Reinvention Jam

Last week I participated in the Handmade Network Wheel Reinvention Jam . The jam is a 7 day event organized by the Handmade Network community, to reinvent a wheel in software.

The Wheel

I chose to work on an audio live-coding tool. There are a number of existing wheels in that domain:

• Graphical dataflow environments like Max/MSP and PureData are used to prototype and build DSP effects and synthesizers by connecting processing 'boxes' together.
  I don't really like the mouse-heavy node graph UX where you have to constantly move boxes around (even if the layout of boxes is irrelevant to the computation!) and draw lots of cables between them. Furthermore, it imposes a dataflow paradigm that, although relatively convenient for expressing many DSP algorithms, is not well suited for writing more imperative control, or expressing temporality.

• SuperCollider, which is a sound synthesis server, controlled either by a client that uses a dedicated scripting language, or directly via OSC messages. I see it as kind of a 'headless' Max, which avoids the unnecessary box shuffling and cable drawing, but it still relies on building a node graph with pre-compiled black boxes.

• Another intersting DSP environment is FAUST: it is a programming language that allows expressing DSP graphs, and has a number of interesting features like inline GUI widgets, an automatic documentation and dataflow diagram generator, integrated spectrograms/meters, etc.
  But it still leans heaviliy towards a declarative, functional dataflow paradigm, and can sometimes be a bit too cryptic to my taste.

In both Max/MSP, PureData, and SuperCollider, it is also possible to write boxes in C, but there is a big abstraction gap between the environement and the way boxes are written, and a lot of friction due to the need of developping boxes using a separate toolchain, and compile and reload them each time a change is made.

My Jam Project

By contrast to these existing solutions, I wanted to have a bare-minimum live-coding environment in which I can quickly prototype DSP ideas in C. Think shadertoy but specifically for audio!

The environment handles the selection and configuration of input and output audio devices. It loads a user-provided dynamic library which contains the user's dsp and UI functions. The system calls the user's dsp function, passing it input and output buffers. It also provides a UI view and calls the user's UI function to allow them drawing UI widgets suitable for audio applications, like sliders, dial knobs, and curves. The user dll is seamlessly hot-reloaded when it changes, without interupting the audio. User code can also request blocks of memory that persist across reloads. On top of that, the environment exposes an API of dsp helpers to quickly create oscillators, envelopes, or simple filters.

Here is the basic window which allows selecting audio devices and configuring the audio parameters:

Here's a demo of non-interrupting hot-reloading:

And here is a demo of the curve UI used to create an envelope:

And here is a very simplistic synth demoing the basic oscillator and filter helpers:

I also made a series of more detailed devlogs to document my thoughts and my progress. You can find them here:

• Day 1: https://youtu.be/NFBSAP4n_NU — simple base application with audio input/output
• Day 2: https://youtu.be/vSofGnl6Jzk — configuring audio devices
• Day 3: https://youtu.be/Jhh3dM9XydY — seamless code hot-reloading
• Day 4: https://youtu.be/mwa3TOhPjO4 — persistent memory and basic UI system
• Day 6: https://youtu.be/DmmKoz8XiJo — UI auto-layout system and curve UI
• Day 7: https://youtu.be/eYZwwhwBDx0 — Wrap-up

Wrap-up

So how has it been going, and what did we achieve?

All in all I would say I managed to get the fundations I wanted working, and I also got to experiment and learn about topics I have less experience with, such as hot-code reloading or auto-layout. It has also been a lot of fun all along, so the experience was very satisfying.

I did spent too much time to my taste fiddling with CoreAudio and trying to scavenge useful info from the docs or from sample code, but I think I now have a better understanding of that area, so I could go back and make a more robust audio device layer later.

I also whish I had more time to spend on DSP-specific features, since that's more my area of expertise, and I feel like I barely just scratched the surface here. And I would have like to do some more interesting synthesis demos beyond simple ring modulation, but the infrastructure had to be done first!

As for the video series, you can tell I'm not really a youtuber, it was unusual for me to document my work in this way, and it took a lot of time putting it together, that I could have spent programming. On the other hand, it provided a motivation for each day, and it forced me to organize and prioritize my work more rigorously, so it also had some upside. Anyway I hope it's at least of some value to a few people.

So, I would say it has been an amazing overall experience!

I don't exactly know what the project will become now, but I would certainly like to go back to it now and then if I find the time. I think now would be a great time to do more serious synthesis stuff to really see what is needed.

For now I could see the project going in several directions depending on what we want to do with it.
For example, we would need a lot more features to use it as a live music making software, such as sampling and sequencing capabilities, and perhaps MIDI and OSC communications. We could also provide a large library of very simple reusable DSP block and push it towards a prototyping environment for quickly testing out DSP algorithms, before making them into plugins. Or we could add more full-featured modules, with associated UI, and bring the environment closer to a modular synthesizer.

Those different sets of features could be added as libraries that a user can choose from and import in their own module, allowing them to tailor the environment to their needs.

So let me know what you think!

This has been an amazing week, be sure to also check the other jam projects on the handmade network website, or come discuss with us on the HMN discord! I'll see you there!