Can AU Sampler Survive Dynamic Chain Changes?

Josip Ćavar — Wed, 18 Jan 2023 14:24:40 +0000

Apple’s AVAudioEngine documentation opens with a brave statement:

You can connect, disconnect, and remove audio nodes during runtime with minor limitations. Removing an audio node that has differing channel counts, or that’s a mixer, can break the graph. Reconnect audio nodes only when they’re upstream of a mixer.

AVAudioEngine header docs state the same in slightly different words:

The engine supports dynamic connection, disconnection, and removal of nodes while running, with only minor limitations:

all dynamic reconnections must occur upstream of a mixer

while removals of effects will normally result in the automatic connection of the adjacent nodes, removal of a node which has differing input vs. output channel counts, or which is a mixer, is likely to result in a broken graph.

In the first blog post in our sampler series, we laid the foundations for exploring the world of software audio samplers, while the third part reveals the secrets of AUSampler’s missing documentation. In this article, we’ll put AUSampler to the test and see if the promise of AVAudioEngine’s dynamic connection abilities holds true.

Dynamic chain configuration

Before we dive into the specifics of AUSampler, let’s explore what dynamic chain configuration even means.

We will start with a simple case of an oscillator. Our goal is to replace the Distortion audio effect with the Equalizer audio effect dynamically while the oscillator is rendering audio without interruptions.

We can do this in four steps:

The code would look something like this:

func reconnectDynamically() {
    engine.attach(equalizer)
    engine.connect(equalizer, to: engine.mainMixerNode, format: nil)
    engine.connect(oscillator, to: equalizer, format: nil)
    engine.detach(distortion)
}

We can take a shortcut here and only detach the node that we want to remove. The engine will take care of the rest.

To verify that everything is in the correct state, we can print the engine’s description. This will print out the underlying audio graph:

print(engine.description)

AVAudioEngineGraph: initialized = 1, running = 1, number of nodes = 4

 ******** output chain ********

 node 0x6000005b0280 {'auou' 'ahal' 'appl'}, 'I'
 inputs = 1
  (bus0, en1) <- (bus0) 0x6000005bdd00, {'aumx' 'mcmx' 'appl'}

 node 0x6000005bdd00 {'aumx' 'mcmx' 'appl'}, 'I'
 inputs = 2\n  (bus0, en0) <- (bus0) 0x0, {}, 
  (bus1, en1) <- (bus0) 0x6000005bcf80, {'aufx' 'nbeq' 'appl'},
 outputs = 1
  (bus0, en1) -> (bus0) 0x6000005b0280, {'auou' 'ahal' 'appl'},

 node 0x6000005bcf80 {'aufx' 'nbeq' 'appl'}, 'I'
 inputs = 1
  (bus0, en1) <- (bus0) 0x6000019bc870, {'aufc' 'conv' 'appl'},
 outputs = 1
  (bus0, en1) -> (bus1) 0x6000005bdd00, {'aumx' 'mcmx' 'appl'},

 node 0x6000019bc870 {'aufc' 'conv' 'appl'}, 'I'
 outputs = 1
  (bus0, en1) -> (bus0) 0x6000005bcf80, {'aufx' 'nbeq' 'appl'},

You can check what these abbreviations map to by running auval -a in Terminal. This will print out all audio units in your system.

This graph is mostly consistent. The number of nodes is correct, and everything is wired up correctly. The only anomaly is an instance of `0x0` input in the main mixer. This is an artifact of the removal process. On successive connections, the mixer will reuse this bus. And while it looks suspicious, it seems that it doesn’t do any damage.

Sampler dynamic chain configuration

Now, let’s try to do the same with Sampler. Our goal is to:

Play and sustain a note with Sampler

Change the effect dynamically while the note is still playing

We can use the same reconnect procedure that worked for the oscillator:

engine.attach(equalizer)
engine.connect(equalizer, to: engine.mainMixerNode, format: nil)
engine.connect(sampler, to: equalizer, format: nil)
engine.detach(distortion)

However, we’ll hear that the sound is interrupted. Moreover, if we try to play some notes again, we will hear the sine wave instead of our instrument. What happened here?

In the previous article on AUSampler we introduced some tools to help us debug and understand Sampler behavior. We can use what we’ve learned to investigate this problem. The process will be the following:

Start the engine

Stop the debugger

Add a breakpoint

Perform dynamic node connection

Observe what is happening

Since we don’t have much knowledge about AUSampler’s internals, we will start with a very general breakpoint:

rb Sampler:: --command "frame info" --auto-continue 1

The Sampler is implemented in C++ and this will print out every invocation of the method that matches this pattern.

Make sure to run this on an iOS simulator. As mentioned in the previous article, macOS doesn’t expose any symbols, and the debugger will not be able to find them.

If we try to change a node now, we’ll see that several things are called:

Sampler::CleanupMemory()
Sampler::SetProperty(unsigned int, unsigned int, unsigned int, void const*, unsigned int)
Sampler::SetProperty(unsigned int, unsigned int, unsigned int, void const*, unsigned int)
Sampler::SetProperty(unsigned int, unsigned int, unsigned int, void const*, unsigned int)
Sampler::SetProperty(unsigned int, unsigned int, unsigned int, void const*, unsigned int)
Sampler::GetPropertyInfo(unsigned int, unsigned int, unsigned int, unsigned int&, bool&)
Sampler::InitializeMemory()
Sampler::GetInitializeNoteCount() const
Sampler::GetPropertyInfo(unsigned int, unsigned int, unsigned int, unsigned int&, bool&)

We should take special note of the following:

Sampler::CleanupMemory is called (which in turn calls: AudioStreamer::Destroy)

Sampler::InitializeMemory

We don’t know exactly what these methods do, but it is a strong indication that the Sampler is reinitialized and that the audio stream is destroyed.

AUSampler’s default preset will produce a sine wave. If you hear it, it means that the Sampler has been reset.

What can we do about it? The good news is that the AUSampler is able to handle dynamic reconnections, but before we dive into that, let’s take a tour of the AVAudioEngine audio splitting.

AVAudioEngine audio splitting

The AVAudioEngine first introduced splitting support in iOS 9 (see Session 507: What’s new in Core Audio from WWDC 2015).

Before that, only one-to-one connections were supported:

SOURCE: APPLE WWDC 2015 KEYNOTE

Then a new API was introduced:

- (void)connect:(AVAudioNode *)sourceNode toConnectionPoints:(NSArray<AVAudioConnectionPoint *> *)destNodes fromBus:(AVAudioNodeBus)sourceBus format:(AVAudioFormat * __nullable)format

SOURCE: APPLE WWDC 2015 KEYNOTE

The engine transparently handles internal connections for us, but it’s interesting to see what happens under the hood.

When we have one-to-many connections, the engine will create an instance of the AUMultiSplitter audio unit. This can be confirmed by looking at the underlying graph that we can print:

AVAudioEngineGraph 0x600002c25c00: initialized = 0, running = 0, number of nodes = 7

 ******** output chain ********

 node 0x600003e23280 {auou rioc appl}, 'U'
<em> em>inputs = 1
  (bus0, en1) <- (bus0) 0x600003e28b80, {aumx mcmx appl}

 node 0x600003e28b80 {aumx mcmx appl}, 'U'
 inputs = 3
  (bus0, en1) <- (bus0) 0x600003e20400, {aufx dist appl}
  (bus1, en1) <- (bus0) 0x600003e01780, {aufx dist appl}
 outputs = 1
  (bus0, en1) -> (bus0) 0x600003e23280, {auou rioc appl}

<em> em>node 0x600003e20400 {aufx dist appl}, 'U'
 inputs = 1
  (bus0, en1) <- (bus2) 0x600003e26300, {aumx mspl appl}
 outputs = 1
  (bus0, en1) -> (bus0) 0x600003e28b80, {aumx mcmx appl}

 node 0x600003e01780 {aufx dist appl}, 'U'
 inputs = 1
  (bus0, en1) <- (bus0) 0x600003e26300, {aumx mspl appl}
 outputs = 1
  (bus0, en1) -> (bus1) 0x600003e28b80, {aumx mcmx appl}

<em> em>node 0x600003e26300 {aumx mspl appl}, 'U'
 inputs = 1
  (bus0, en1) <- (bus0) 0x600002200240, {aumu samp appl}
 outputs = 3
  (bus0, en1) -> (bus0) 0x600003e01780, {aufx dist appl}
  (bus1, en1) -> (bus0) 0x600003e00b00, {aufx dist appl}

 node 0x600002200240 {aumu samp appl}, 'U'
 outputs = 1
  (bus0, en1) -> (bus0) 0x600003e26300, {aumx mspl appl}

Note an instance of the mspl audio unit. It represents an instance of the AUMultiSplitter audio unit responsible for audio splitting.

These are the unit’s header docs:

kAudioUnitSubType_MultiSplitter
    An audio unit that sends its input bus to any number of output buses.
    Every output bus gets all channels of the input bus.
    This unit's implementation is lighter weight than kAudioUnitSubType_Splitter
    even for two output buses, and is recommended over kAudioUnitSubType_Splitter.

This audio unit turned out to be the key to keeping AUSampler alive while dynamically reconnecting it.

Correct setup

With this knowledge, we are now able to find the correct setup.

Ideally, we would like to do this:

However, we realized earlier this wouldn’t work. Instead, let’s do this:

These extra two dummy effects give us a fully working sampler. The reconnection process would look like this:

Step 1: Create a Sampler with two static dummy nodes and connect it to them.

engine.attach(sampler)
engine.attach(staticDistortion)
engine.attach(staticDistortion2)
engine.attach(distortion)
  
engine.connect(
    sampler,
    to: [
           AVAudioConnectionPoint(node: staticDistortion, bus: 0),
           AVAudioConnectionPoint(node: staticDistortion2, bus: 0),
           AVAudioConnectionPoint(node: distortion, bus: 0),
    ],
    fromBus: 0,
    format: nil
)
engine.connect(distortion, to: engine.mainMixerNode, format: nil)
engine.connect(staticDistortion, to: engine.mainMixerNode, format: nil)
engine.connect(staticDistortion2, to: engine.mainMixerNode, format: nil)

Step 2: Reconnect.

engine.disconnectNodeOutput(distortion)
engine.detach(distortion)
engine.connect(equalizer, to: engine.mainMixerNode, format: nil)

engine.connect(
    sampler,
    to: [
          AVAudioConnectionPoint(node: staticDistortion, bus: 0),
          AVAudioConnectionPoint(node: staticDistortion2, bus: 0),
          AVAudioConnectionPoint(node: equalizer, bus: 0),
    ],
    fromBus: 0,
    format: nil
)

If we now add the same breakpoint as in our initial setup, we will see that nothing gets called and the Sampler is not touched on reconnection.

Finding this out was pure luck. We had a more complex setup that worked by default. After making some changes to our chain, we found that things got broken and then went down the rabbit hole that brought us to this realization.

We need two dummy nodes because the splitter node is very sensitive to the number of connections. If we leave it with only one connection, we will get the following error:

[avae] AVAEInternal.h:76 required condition is false: [AVAudioEngineGraph.mm:3135:GetOutputConnectionPointsForNode: (numSplitterConnections == 0 || numSplitterConnections > 1)]

Tricky, yet important

While quite tricky, we’ve seen that the AUSampler in combination with AVAudioEngine is capable of picking up dynamically added or removed nodes. This use case is very important for some apps, and while going to the AudioUnit level is always an option, it also requires us to sacrifice the convenience of the AVAudioEngine API.

Please note that the AVAudioPlayerNode is not immune to this problem either. If we try to connect new nodes to the AVAudioPlayerNode during playback, the playback will stop and we will need to schedule buffers/files again.

Furthermore, it’s puzzling that introducing a splitter prevents the Sampler from cleanup; it would seem that shouldn’t be necessary.

We have filed both of these as bugs to Apple.

You can find the associated source code here.