The Influence of the Actor Model (Praxis architecture 101)

No, not a post about my salacious exploits with a C-list Hollywood celebrity (that’s for a different blog 🙂 ), but a technical overview of a key aspect of the Praxis architecture. Absolutely essential to Praxis’ media neutral architecture, as well as the ability to edit everything live, is an asynchronous, shared-nothing, message-passing system. This is loosely inspired by the Actor Model and other solutions for concurrent / distributed programming (without following any particular concept to the letter).

What’s the problem?

Much of my work involves working with multiple media (primarily audio and video), or maybe multiple video streams with a low frame-rate webcam controlling events in a high frame-rate projection.  Sometimes you might be working with video working at 60 fps and audio processing at ~689 fps (Praxis uses an internal 64 sample buffer).  How do we process events in those pipelines and allow them to talk to each other without interfering with each other?  Now add in the principle of edit everything live where we want to be able to insert new components, live compile code, or load resources, all without skipping a frame or a beat.

Solving this problem was probably my primary reason for developing Praxis.  There’s some great software out there for working with audio or with graphics / video.  However, there’s not a lot that I consider works well with both at the same time.  For example, I think Processing is a great project (and the live-code API in Praxis is based on it for a reason!), but as a system for working with multiple media I find it frustratingly lacking. This is not due to a lack of libraries for audio, etc. but because the underlying architecture is fundamentally tied to the single frame-rate based model.

What’s wrong with synchronization? … Everything!

Working with different media in this way is analogous to working with Threads. In fact, in most cases that is exactly what we’re doing. Now the common way of dealing with communication between threads in a Java program is to use a synchronized block or other form of lock (mutex). Unfortunately, this is a terrible way of dealing with this problem.

Time waits for nothing

There’s a great article by Ross Bencina (author of AudioMulch) called Real-time audio programming 101: time waits for nothing. It discusses in some detail the reasons not to use mutexes (among other things) when programming audio. I consider it a must-read article for anyone doing audio coding, and I also feel that many of the principles are equally relevant to working with other media (ie. real-time video).

Remember, your media thread is for processing media! You need to be aware of the code that is running in that thread and the maximum execution time of that code to ensure that video frames or audio buffers are processed on time. If you start adding in locks, then besides the overhead of synchronization and putting your code at the mercy of the scheduler, you add in the possibility of contention with code that cannot meet your processing deadline.

Optimization

There’s another problem with locks, and that is working out how coarse-grained it is – how much code it covers. You’d expect that for best performance you’d want to keep the lock as fine as possible, only covering the minimal required code in an individual component. Well, besides this still not being a good idea for the reasons stated above, it also inhibits a more important performance optimization that happens in the Praxis code. Both the audio and video pipelines switch off processing of sections of the graph where they can prove the output is not required.  To achieve this, they need to be sure that changes can only happen at the beginning of each processing cycle.  However, to achieve this using mutexes would massively increase the chances of contention.

Atomicity

The final problem with synchronization is simply one of misapprehension, but one I’ve seen on multiple occasions.  Imagine for a moment you have a Sample object type, and on that type is a synchronized play() method.  From your control thread (maybe Processing’s draw() method) you call play, and the sample plays fine and all is well.  Now imagine you decide to play 4 samples in sync, so you call play() 4 times in succession and all is well .. actually, no it isn’t, because there’s no guarantee that those 4 samples will play together.  This is another issue with using synchronized methods – how can the user ensure that a series of events happen atomically?

The Praxis architecture

One (probably the) way to deal with the issues above is to pass events into your media process using a non-blocking queue or ring-buffer. I’ve written a variety of code in the past that used variations on these methods, and I’ve used a variety of other non-blocking concepts (such as Futures) for dealing with resource loading, etc. Praxis began life as an experiment to build an architecture based entirely around a single, uniform message-passing model. This architecture is loosely inspired by the Actor Model. I won’t go into detail about this here (this post is getting long enough!), but would recommend this Wikipedia page, or this great introduction to Actors in Java by Fabrizio Giudici (need to fix up those code examples, though, Fabrizio!).

A forest not a tree

Individual components within Praxis (such as a video effect, sample player, button, etc.) exist within a tree of components, which can be multiple levels deep.  All components have an address, which follows a familiar slash-separated syntax (eg. /audio/delay1). The first level of this tree is a special type of component called a Root.  Roots generally provide the media context for the components that exist within them, and there are currently 4 types of root available within the standard Praxis LIVE install – audio, video, MIDI and GUI. While you would commonly have one of each type you require, it is perfectly possible to have as many roots of the same type as your system can handle.

All roots exist within the Root Hub. The root hub is a container for roots, but it is not itself a component. When a new root is installed into the hub, it is the hub which provides it with the (primary) Thread with which it will run. All roots are sand-boxed from each other – there is no way for any root or component within it to directly access another root or its components. Instead, the hub acts as a router to pass messages between different roots.

Hub Manager in Praxis LIVE

Here you can see the Hub Manager within the Praxis LIVE application, which gives you a visual representation of the root hub.  You will notice that there are 3 user roots running (audio, video and GUI), but that the button is toggled to also show you the system roots.  This is another key part of the architecture – all of the system code is equally sand-boxed and confined to using the message-passing system. In addition (in case you’re wondering about the weird ID’s of system roots), the hub provides a mechanism for looking up the addresses of various services, similar to the ServiceLoader mechanism in the Java core libraries.

If you’re using Praxis LIVE, one of those system roots is the gateway by which the NetBeans Platform-based application communicates with the Praxis core.

Ports and Controls

Components within Praxis have two ways of communicating – Ports and Controls. Ports are used for synchronous communication with sibling components – components that share the same parent. When you connect components by drawing lines in the Praxis LIVE graph editor, you are connecting ports together. They are a lightweight mechanism for sharing control data, video frames, audio buffers, etc. Ports also have an address consisting of their component address and ID (eg. /audio/delay!in).

Controls are the basis of Praxis’ message-passing system.  They receive, react and respond to messages usually coming from another component within another root.  All communication (without exception) between components in different roots ends with a control receiving a message. Controls have an address consisting of their component address and ID (eg. /audio/delay.time) – the dot syntax is a deliberate parallel with method calls.

Ports and Controls of an audio:sampleplayer in Praxis LIVE

Above you can see an example of an audio:sampleplayer component from within the Praxis LIVE editor – ports can be seen on the graph component and controls within the property dialog – when you manipulate values within the dialog, you are sending messages to controls on the component. There are three different types of controls – actions, properties and functions. Actions are represented by the buttons at the top of the dialog, and properties within the table. Functions are not currently reflected within the editor, but are primarily used internally.

All components provide an .info control that returns a data structure outlining the configuration of ports and controls.  This is analogous to the BeanInfo mechanism used by JavaBeans, and is how Praxis LIVE can configure its editors.

NB. The ready and error ports reflect the fact that samples cannot be loaded immediately without the audio breaking up.  Therefore, this request is passed to one of the system roots that handles background loading of resources.  Once the request is completed then a message is sent from the ready or error port depending on whether the sample could be loaded or not. Other components that load resources use the same methodology.

Calls and Arguments

The basis message sent within the Praxis environment is called a Call. All call objects are immutable (as per the actor model, there should be no shared mutable data) and consist of the following immutable fields –

• To Address – the address that the call is being sent to.
• From Address – the address of the control that initiated the call, and to which any response should be sent.
• Type – one of Invoke, Invoke Quietly, Return or Error. Calls of type Invoke require a response, whereas calls of type Invoke Quietly do not – useful for efficiency if the sender cannot make use of the response (eg. MIDI bindings).
• Time – a time stamp for the call.
• Arguments – 0 or more Argument type.

Praxis defines its own type system, all of which subclass the type Argument. This helps to define certain semantics required by the system, including immutability. Arguments should also offer conversion to and from a defined string representation (for saving to file, etc.) The main core types are String, Number, Resource (URI), Array and Map. Info structures and addresses are also arguments. There is also a Reference type, which holds a reference to any Java Object – it is used for handing over newly built components or loaded resources from one root to another, and should be used sparingly and carefully as it allows you to break with the concept of no shared mutable state!

Unlike the standard actor model, components do not maintain their own queue for incoming messages.  Instead, each root component maintains a (non-blocking!) queue, and passes messages down to its descendant components at the correct time. Time stamps of calls are currently related to system clock time, though calls may be scheduled in advance (to alleviate jitter), and roots are free to run slightly before or ahead of system time as may be needed. The time stamp is not intended to be a general mechanism for scheduling events in the future; nor does the system (as per the actor model convention) ensure the order of message delivery, though some controls do use the time stamp to ensure the priority of later messages eg. in the setting of property values.

Praxis Script

It is perfectly possible to control the Praxis system directly from Java code using callbacks, in a similar method to that described in another great post by Fabrizio Giudici (A Cleaner MVC Inspired by Continuation-Passing Style). This is the mechanism used in the Praxis LIVE application to connect elements of the NetBeans Platform to the Praxis core. However, this method can become verbose, and requires compiling so is no use at runtime. Therefore the Praxis core can also be driven by its own simple script language (if you’re using Praxis LIVE you can ignore this fact if you want to, though if you open up a .pxr patch file from a Praxis LIVE project you’ll notice they’re actually saved as valid Praxis scripts).

Praxis script is a simple text based way of controlling components within the Praxis environment.  It uses a syntax loosely based on Tcl. Two common commands used are @ (at) and ~ (connect). @ takes a component address and an optional component type (used to construct a new component) and script (that will be run within the context of the address, allowing nested hierarchies). This simple script language hides the complexity of working within the Praxis message-passing system, automatically handling the sending and receiving of messages.

This is a simple script used to create a simple graph identical to the Hello World example in the examples download – it is almost the same as the output of the video.pxr file from the graph editor.  There are actually 5 different roots (threads) involved in the execution of this script!

@ /video root:video {
    .width 400
    .height 300
    .fps 20

    @ ./noise video:test:noise {}

    @ ./image video:still {
        .image [file "resources/hello world.png"]
    }

    @ ./window video:output {}

    ~ ./noise!out ./image!in
    ~ ./image!out ./window!in
}

/video.start

In Practice (in Praxis)

What began initially as an experiment has matured into a robust system that I have been using in my own work for almost 2 years now. Defining a standard architecture in this way has made it incredibly easy to wrap pretty much any library quickly and get it to talk to any other elements of the Praxis system safely. (eg. I’ve got an Arduino module I wrote in about 20min a while back and hope to tidy it up for release soon!). The distributed architecture also has other benefits – it should be fairly simple to expand a Praxis hub to run across multiple machines in the near future.

Up until now, Praxis has primarily existed as a project that I can use, that I’m happy to tweak at the last minute and leave to run in confidence for the hours, days or weeks it needs to.  I put a lot of this stability down to my quest to banish synchronized from its code base! 🙂 Hopefully with the development of the Praxis LIVE visual editor and some better documentation, this code and concept can now be of use to others.

If you’ve made it this far, thanks for your perseverance.  If you feel anything needs clarifying or explaining better, please comment below and I’ll fix it.  And now it’s time for a beer (on me, if you’re lucky! 😉 )

Advertisement