“Trece años dedicò a esas heterogéneas fatigas, pero la mano de un forastero lo asesinò y su novela era insensata y nadie encontrò el laberinto.”
J.L. Borges, El jardìn de senderos que se bifurcan.

Babbler – Wheel Reinvention Jam

Last week was the 2024 Wheel Reinvention Jam! My entry is a little toy computing system based on my (mis)understanding of Dynamicland’s Realtalk.

Dynamicland is a lab lead by Bret Victor, whose research aims to create what they call a “humane dynamic medium”. They recently released a new website which is a treasure trove of orientation papers, progress reports, demos, and archives, and it got me really excited about the work they’re doing there.

The most visible aspect of Dynamicland is that it is a real place, where people can walk and talk and collaborate in authoring ad-hoc programs using tangible objects organized in the physical space. Entangled with this striking difference with how we commonly experience computing, there is a more profound proposition, that of a paradigm shift from personal computing to communal computing. This is, perhaps, the truly revolutionnary vision of Dynamicland. Unfortunately, this is also the point I decided to entirely miss during that Jam.

Don’t get me wrong! I vibe with this vision and I’d like to see more of it, but I had no projectors, no cameras, no space and no time, and didn’t want to fuss with computer vision and fiducial systems. I also wasn’t going to whip up a communal experience in a week. So what’s left?

Another aspect that got me intrigued is the operating system powering the whole place, called Realtalk. Although this is the thing people have to interact with in order to author new behaviours attached to physical objects, it does not catch as much light as the tangible and communal aspects of the project, and is comparatively under-represented in Dynamicland’s public material. One could likely argue it is by design, since one goal of the system is to be pretty unremarkeable to its users, who should focus their attention on the social interactions and the dynamic organization of the space.

However, it is not unremarkable to the technical eye, and I think its programming model actually contributes a lot to what makes Dynamicland, well, dynamic. Conceptually, Realtalk is organized around a shared database of facts, that objects can query and modify to interact with each other. This database can contain pretty much anything objects claim or wish to be true about the world, as well as spatial relationships and anything that can be derived from the vision system. Objects can declare game-like rules to apply when a given set of facts is present in the database. Applying a rule can in turn add facts to the database or apply more rules, etc. This is how individual objects can become aware of their surroundings, react to or modify the behaviour of other objects, and be flexibly re-combined to solve complex tasks as needed by the human participants.

Although these basic principles are simple to grasp, I had some trouble piecing together more technical details from demos, emails, and contributor blog posts, and trying to get a better understanding of how the system works. So in good Handmade spirit, I decided to create my own pocket version of Realtalk, in order to at least gain some insight about the shape of the problem it is solving.

So unless mentionned otherwise, what I will describe in the following applies to Babbler and only reflects my second-hand understanding of Realtalk. I don’t know how close it matches the real thing (besides the obviously smaller scope), but the point is more to explore the space and hopefully hit some of the same design questions as the Dynamicland’s team faced.

Detail from Olaus Magnus’ Carta Marina. Babbler is probably as far to Realtalk as this depiction of a whale and orca is from the actual animals.

The Interface

I first made a simple interface, where you can drag/resize cards on an infinite canvas and write code on them. I immediately got sidetracked into writing a little structure editor for the cards, re-using some code I had from my work on Quadrant. This avoids having to write a parser and maintain a mapping between the source and the parsed representation, and also simplifies auto-layout and syntax highlighting.

The column on the left is intended for putting away unused cards, that won’t be taken into account by the sytem. If I had more time playing with more complex programs I would probably have added a second column on the right to store cards that are active, but don’t need to be put on the canvas (e.g. “rulebooks” whose spatial location and graphics are irrelevant).

It is kind of funny that it already highlights one advantage of the benefits of Dynamicland’s approach (once you have a fiducial system in place): you simply don’t have to do all this user interface work. You can just put pages on a table, rearrange them however you like, stack them in any order, put away those you don’t need or file them in a binder, etc.

Claims, Wishes and Whens

Contrary to Realtalk, which uses a superset of Lua, I chose to make a custom Lisp-like mini-language which maps directly to my structure editor. There are three forms that allow querying and modifying the facts database, which represents the shared state of the system.

The fact database has no “world model” and does not attach any meaning to facts, nor enforces any consistency between them. Facts are only given a meaning by matching when rules. The database itself just stores value trees, which are composed of lists and atom values, which can be numbers, symbols, card identifiers, or strings.

The syntax trees of facts in claim, wish and when forms undergo an evaluation step that transform bound names and expressions into their values to produce a value tree. Unbound symbols and unevaluated forms are simply mapped to identifiers and list values. Notably, claim, wish and when forms inside those syntax trees are not evaluated, so these words can be part of a fact.

Patterns in a when clause are matched term to term with facts in the database, using a depth-first traversal. Atom values match if they are equal, while list values match if their children match one-by-one.

A when clause can use placeholders of the form $name inside its fact pattern. A placeholder matches any sub-tree at the same position, and if the whole pattern is matched, the name name is bound to the value of the matched sub-tree.

Below is an example of using facts and rules to alter the graphical aspect of cards and link the behaviour of different objects.

The resolution of rules uses a simple (if inefficient) fixed point method. You start each frame with an empty database, and you just keep executing claim, wish and when forms over and over until no new fact is added to the database. Then you update the UI and move on to the next frame.

Care has to be taken to not put the same fact multiple times in the database. Luckily we can reuse the fact-matching code used by when forms to check if a fact is already in the database. We also need to make sure that other side-effectful forms are run at most once per frame. Facts are numbered in the order they are added to the database. When clauses remember the last fact they matched and only run side-effectful actions for newly matched facts. Note that when forms are still re-checked, because they could match new facts even if their parent matches an old fact.

I’m pretty sure this simplistic scheme breaks down as you add more ways of matching things, and down the line you probably need a smarter “solver”. In particular this model doesn’t handle fact deletion at all, and doesn’t provide a way to detect when no fact is matched. These two features seem tricky because the first one can invalidate previously held facts, whereas the second can be invalidated by later facts. Both have rippling effects when you consider facts that depended on the invalidated facts through when clauses. Realtalk seem to have the ability of detecting unmatched facts through a When ... Otherwise ... End construct, but I have found no information on how it deals with invalidation.

Helpers

In the above example I glossed over the built-in rules that allow labeling or highlighting cards. I call them “helpers” because they listen to some predefined wishes and help them come true. They could just as well be user-level when clauses if the language had a direct way to do graphics, but for the purpose of the jam they’re using hard-coded actions. However they do use the same matching mechanism as when clauses, with the following patterns:

The built-in actions basically just set some data and update a frame counter in the cards structure. When the frame counter matches the current frame number, the UI systems applies the custom decorations on top of the card.

Responders

One of the core features of Realtalk is the ability to relate objects to each other using their positions and orientation in space. Even if our card live on a 2D canvas, it is still a powerful way of combining cards to build more complex programs.

In keeping with the spirit of Realtalk, these spatial relationships are expressed as facts and queried with when clauses. For example, in Babbler you can detect if a card is pointing up at another card using a when clause like this:

(when (self is pointing up at $p) ...)

The direction can be up, down, left, or right, and both the pointer and the pointee can be any card identifier. These three elements can also be placeholders, so if you wanted to know when any card is pointing at the current card in any direction, you would write:

(when ($p is pointing $d at self) ...)

Here’s an example of using such spatial queries:

It would be impractical to populate the facts database with all possible spatial relationships between each pair of objects at the beginning of each frame. We need rules to generate new facts on-demand, only when they are queried by a when. I call these rules responders, since they’re responding to a query by generating new facts. Again, they could eventually be user-defined, but I kept them as built-in constructs for now. They also use the same matching mechanisms as whens, although in a bit different way.

Responders have a pattern and a callback. Conceptually, our pointing responder looks like this:

(respond-to ($p is pointing $d at $q) callback)

When a query is not matched, we try to match each responder pattern to the query pattern. For each responder that matches the query, we run its callback, passing it the match bindings. The callback can then use values bound to names p, d, and q to determine which pointing relationships hold and create new facts accordingly. Keep in mind that in thise case bound values can be placeholders themselves. So for example, the query (#6 points up at $x) will match the responder with p = #6, d = up and q = $x, and will thus create a fact (#6 points up at #n) for each card #n that points up at #6.

Nesting Whens

Realtalk can express multiple patterns in its When constructs, separating them with commas (i.e. When p1, p2 : ... End). In this case the when is triggered only when all patterns match. I don’t have this construct in Babbler. The “lispy” way would probably be to have a list of patterns, but then single patterns whens look awkward. I could also just treat the comma as a separator and have (when (p1 , p2) ...), but I did not bother.

You can however achieve the same end with an admitedly more verbose nested construct:

(when (p1)
    (when (p2)
        actions))

This executes actions only when both p1 and p2 have matches.

You can obviously execute an action when either p1 or p2 have matches by just having two clauses:

(when (p1)
    actions)
(when (p2)
    actions)

Although with this construct you can’t enforce that placeholders with the same names in p1 and p2 match the same values.

Anyway this allows us to implement logic gates in Babbler:

Persistent State

So far our programs are purely reactive: they can only react to the current state of the canvas and can not carry state from frame to frame. So for example you can’t implement a latch, or a counter that stores a value, or a clock, or really anything that needs some persistent state.

I suspected that I could use the facts database to store persistent values, or to query facts in the past, by essentially copying the state of the database to a log each frame. But I didn’t really have time to figure out how to make a simple and intuitive syntax for it, so I fell back to using good old variables declared by each cards. (I also had all the machinery from binding match placeholders to values, so it was the path of least resistance).

You can declare a variable and initialize it with the (var name init) form. The init only gets executed when the init element is created or changed. You can then use name in expressions and set its value with the (set name value) form. Variables are only accessible within their scope, and top-level variables are persisted across frames.

Here’s an examples of a click counter using variables:

And a stopwatch of sorts (it’s actually counting frames, not milliseconds!):

You can see that I have to make what feels like redundant claims about the values of variables, in order to make them accessible to other cards.

Thanks to Andy Matuschak, who pointed me to the much nicer Realtalk solution, called Memories, that I had missed in the archives. This essentially introduces a Remember fact. construct which adds a persistent fact to the database. The fact is also attached to a particular card, so you still get some namespacing, but it is also accessible to all objects through queries, so unlike variables it really clicks with the Realtalky way of doing things.

Closing Thoughts

Overall I’m pretty happy with how this jam project went.

Of course I wish I had more time to add interesting graphics helpers (e.g. hooking into my vector graphics renderer) and spatial responders (e.g. proximity, closest neighbors, or clustering detectors), as well as allowing user-defined responders. The compute capabilities of the language are also, hum, pretty limited (you can add things!).

But despite (thanks to?) its obvious limitations, I feel like this jam project helped me appreciate what’s compelling about Realtalk and Dynamicland in more depth. I hit a number of interesting design issues that I had not anticipated, which gave me a better understanding of the design space, and sometimes raised more questions about how Realtalk works. Here are some of them:

If you happen to be in the know, I’d love to hear some answers to these questions!

Anyway, with this new appreciation for Realtalk and Dynamicland, I’ll sure keep an eye on the lab’s research. I also hope to continue playing with Babbler as time allows.

Thanks for reading, and see you soon~

^ Jungle Babbler picture by J.M Garg, CC License