AVAudioPlayer Causing Frame Rate Drops in SpriteKit

nev-clock

It started with a trip to Singapore.

This was a natural deadline to get AdventureKit to 1.0.

It was going so well. In the last week, I had added my first animated walk cycle and even included some reach animations for my main character.

Then I realised what was missing—footsteps! I wanted to hear the click of shoes on concrete as my character walked the streets of Lecce.

I’ll Just Do This One Last Thing…

This was a straightforward problem to solve.

I already had a sound effects component set up. It had a simple API—send in the name of the audio clip to play and it will play it.

All I needed to do was trigger the footstep sound effect at the start of every walk cycle animation loop. The work of a couple of lines of code.

I put this in place on the Friday. We were due to leave Saturday morning.

One more build and run, I thought, and I would be good to go. Perfect timing, really. The audio enhancements would be a nice, tidy little bow.

The game launched just fine. It looked good. I heard the footsteps.

I noticed that it was dropping frames.

My heart sank.

Diagnosis

Singapore was fun! The engine dropped frames the whole time!

Writing it out here, the problem is obvious. The last thing I added was the audio changes.

The problem at the time was that I hadn’t really been paying attention to the framerate counter. It could have been the audio, but it also could have been that these drops were happening for weeks and I only just noticed them at that moment.

Trying to outsmart myself, I also considered that it was highly unlikely that small sound effects would cause a significant drop. It was much more likely to be graphics related (which would really suck, as the idea of having to rewrite my animation engine filled me with dread).

Low-Hanging Fruit

Not considering the audio and afraid of tackling the visuals, I first checked the engine without having the debugger attached.

This was a vain hope, but AdventureKit writes a lot of log messages and, although it does use Apple’s new super-fast os_log, I thought that Xcode’s display of these logs in the debugger might be introducing a small delay.

Of course it wasn’t that.

Diving Deeper

The SpriteKit frame rate counter would drop from 60fps to 57fps at the start of every walk, with a fluctuation between 58 and 60fps as the walk continued.

The weird thing was, though, that when I ran it at the iPad’s native 120fps, it still only dropped one or two frames.

This didn’t make sense—if the issue was that some process was taking 3 frames worth of time at 60fps (~48ms), then the same amount of time should show a drop of 6 frames at 120fps. The drop remained 2-3 frames, even at the increased frame rate.

This actually gave me some hope, as it probably meant that it wasn’t anything related to graphics after all.

AdventureKit makes changing the size of sprites easy so I was able to quickly switch to using half-sized sprites to confirm this. The exact same number of frames were being dropped, even though memory usage plummeted.

This brought back around to audio which, as the last thing I worked on, was probably where I should have looked first.

Of course, I don’t work like that because I’m a maverick.

The Sound of Silence

During set up, my audio component took an array of sound files and prepared some AVAudioPlayer instances with the sounds loaded and the prepareToPlay method called on each. These were all stored in a dictionary where the key was the name of the sound to be played. They were as ready as possible.

Playing a sound was as simple as sending the key to the component which looked up the corresponding AVAudioPlayer and called play on it.

At least it should have been this simple.

Something about playing the sound using these players was causing the frame rate to drop.

SKAction

There is an SKAction instance method called playSoundFileNamed(:) that is designed to be used for instantaneous sound effects.

Replacing the AVAudioPlayer set up with a dictionary of these SKAction objects worked perfectly with no lag and no dropped frames.

So Then Use That, Dingus!

This would, of course, be the easy solution.

This is me, though.

This SKAction does not work with asset catalogs. It only works with audio assets that are loose in the main bundle.

I have set up an extensive workflow that automatically sets up asset catalogs with all of my assets, including audio and data files. It gives me the confidence that the latest build always has the most up-to-date assets.

I really didn’t want to have to break this workflow without doing some further research first.

I also don’t know what SKAction uses under the hood, but the documentation recommends using AVAudioPlayer for longer sound files (e.g. background music) which suggests that using AVFoundation alongside SpriteKit shouldn’t be a problem.

SKAudioNode

After the SKAction failed, I looked at SKAudioNode which seemed to be a more robust way of managing audio in SpriteKit. Some initial Googling about it gave me hope.

This SO post suggested that using audio assets in asset catalogs would work with SKAudioNode. Perfect! Problem solved!

All I needed to do was address the assets correctly.

For the life of me, I could not get this to work. Things I tried:

  1. Making sure the names of the assets were as simple as possible (no hyphens, no underscores).
  2. Trying different asset catalogs.
  3. Trying different audio formats.
  4. Making sure the underlying file was named the same as the asset catalog alias.
  5. Trying all of these options with and without the file extensions.

In my fugue state, I even considered initialising the SKAudioNode with my own custom AVAudioNode (which is an initialisation option for SKAudioNode) but the simplest one I found to use was AVAudioPlayerNode. This could only be initialised with…a URL. No good for asset catalogs!

All the other AVAudioNode options at that level are about managing audio buffers and encodings and required hooking up buses and managing data streams and oh my!

Positional Audio

However, in playing around with SKAudioNode, I did get hooked on its positional audio capabilities.

In my model, individual entities declare the sounds that they want to use so all of my sound effects are already tied to on-screen elements. Every entity has an audio component and so they have an easy way of adding an SKAudioNode to their node hierarchy:

public override func didAddToEntity() {
    for (name, url) in self.soundURLs {
        let node = SKAudioNode(url: url)
        node.name = nodeName(for: name)
        node.autoplayLooped = false
        self.entity?.component(ofType: NodeComponent.self)?.add(node)
    }
}

Everything was in place and all that was needed was to set the scene’s listener to the camera:

self.scene?.listener = self.scene?.camera

That’s it! As my character walked to the far edges of the screen, the footstep audio would move to the far edges of the stereo field.

There was no way I was leaving this awesomeness behind.

I just needed to solve the problem of not being able to use asset catalogs.

The good news was that, unlike the SKAction, the SKAudioNode can be initialised with a URL. The bad news was that assets in a catalog do not have URLs.

You Dirty, Dirty Boy

Reader, I hacked around it.

Now, when the audio component is initialised with a list of sounds that reference data assets in the asset catalog, it goes and reads that data then writes a copy of it to the caches folder. It then uses this cache URL to initialise an SKAudioNode and everything works as expected.

There are many reasons why this is a bad idea. Here are a few:

  1. It duplicates data.
  2. If a player has a storage-constrained device, it might not be able to write the data to the cache meaning No Audio For Them!
  3. File IO is a more expensive operation than just reading the data from a bundle URL.

All of these downsides just to protect my asset catalog based workflow? Preposterous! The prosecution rests.

The defence rises.

Consider these mitigating factors, your honours:

  1. The audio data is relatively small, and does not include the background music which are the largest of the audio files.
  2. Apple’s mobile operating system has systems to clear up space as needed in extremely constrained environments.
  3. This file writing operation happens once during scene set up and adds no more than a fraction of a second to the load times which include the much slower operation of preparing hundreds of megabytes of textures.

It’s not ideal and maybe in future I’ll change my workflow to copy the audio data into the bundle instead of into the asset catalog, but it’s Good Enough for Now (this should be my tagline).

Conclusion

AVAudioPlayer does weird things with SpriteKit. It’s fine for background music, but probably don’t use it for sound effects.

SKAudioNode is amazing (the positional stuff, which is so easy to set up, is awesome by itself but it has other cool atmospheric effects too).

When coding, the last thing I changed was probably the thing what broke it. Maybe look there first next time.