The First Step to Optimize a Video Game? Measure!
For sure the title for this story can be generalized to all software. It is too common in software development to see people making changes to their code trying to get it to go faster without really knowing what they are doing.
This is very problematic for several reasons. Of course, the first one is that it is hard to really appreciate if the changes have really helped or not. We appreciate time relatively, waiting for things to happen will make them seem longer and even sometimes we’ll be too optimistic and think something improved when really that was not the case. Having an automated way to test changes and measure them properly will always make the improvements faster and better.
However, the biggest reason why it is absolutely necessary to do proper time measurements is that the intuition of where the real problem lies is often completely wrong. Software is counterintuitive this way, looping through endless arrays can be almost instantaneous thanks to modern compiler and hardware optimizations, whilst seemingly innocuous practices, like keeping the lifetime of variables to a minimum will completely thrash your performance.
Why am I referring specifically to video games? Because I really like programming them. Also, because you’ll find specific problems when dealing with performance issues in gaming. Video games need to draw each image from scratch on every frame, which ideally happens 60 times a second. In some cases, like Virtual Reality, the fact that at any one point your software is not updating 60 times in a second might prevent you from selling your game altogether. Not only do things need to happen fast, but also consistently; it is better to have a game running at stable 30 frames per second than sometimes at 60 and slightly stuttering every few seconds.
The story
A while ago I spent a few years developing a video game, Space Pizza. As with most of my video game developments, I am usually more interested in learning about game development than in improving the gaming experience. In this case, I decided to use quite a low-level framework (monogame) and develop as many features as I could from scratch. As soon as I had a usable engine to efficiently create new levels and experiences I became a bit bored of it.
I didn’t pay a lot of attention to the game’s performance whilst developing it, being quite a simple game and running it in my desktop computer everything seemed to work well. Everything became very noticeable once I ported the game to my Android phone. In a way, I was a bit surprised of such a poor execution since I had followed numerous optimization strategies and good practices:
- Frustum culling: only draw elements shown on the screen at the moment
- Ordering draw calls depending on each object’s materials and textures
- Cache supposedly expensive calculations over frames
- Use data-oriented design for optimizing CPU cache usage
No wonder all this is fundamental when working on a AAA game or where you’ll have some technical challenges like lots of particles or objects. However, when you are a one-man developer making by hand most of your systems, the performance drops will reside in very stupid mistakes or, at best in bad practices you are not completely aware of.
After seeing the performance problems, I spent quite a long time trying to improve lots of random things that I was certain would fix the issues. Nothing really worked, so to start properly measuring the game’s performance to have some certainty before improving things willy-nilly.
Problems measuring video game performance
When I used to work for a professional development studio I had lots of tools at my disposal. Especially with consoles, I could see the usage of each CPU core in real-time or see how long it took for the GPU to receive the draw commands, execute them, and notify the CPU back.
Working home, in a Linux setup, and with mostly custom-made software I had none of those luxuries. I needed to develop a custom way for properly measuring how much time was spent in each part of my engine.
The game I was writing using the Entity Component System (ECS) architectural pattern. Explaining it properly would require a separate article, but for our current scope let’s just say that it allows you to define a series of Systems like 3d drawing, 2d drawing, physics update, logic update… that are completely independent of each other and where each of them gets called once for each game step. Thanks to this, I could just take the current time before and after calling each system and print the difference out in the console. Done.
Alas, not really done. As already mentioned, and probably common knowledge, each iteration of the game loop happens at least 30 times a second; my console log was filled with print statements, and was impossible to make any sense of it. I tried logging the highest values, which would happen when loading the game, which ended up being useless. I would be interested in seeing spikes and average values depending on what I was doing on-screen, and just stopping the execution would not be good enough. Pausing the debugger on demand had more or less the same issues.
Some inspiration
One of my hobbies is reading game development blogs and stories. I remembered reading about a problem developing a classic game and how it was fixed. Sadly I forgot what game it was. Said game randomly crashed, and the development team had no proper debugging tools. In order to find in which part it failed, the team showed a colored border whilst playing it where the color depended on which part of the code was being executed. When the game crashed, it displayed the last frame, so the developers could have some idea where the problem resided.
This may seem quite a trivial technique, but it did give me a starting point for where to start with. It prompted me to assign a color to each system and show graphs on the screen with varying sizes depending on how much time it takes for each of them. The end result :
In the above image, you can see a screenshot of the loading screen with the debug menu in the top left corner. It does not look very fancy at all, but it ended up being quite useful. The left bar shows how long it takes for the drawing and the logic part of the game in general, the next two show how much it takes for each of the systems related to it and the last one how much memory is being used.
Below these lines, I also drew some horizontal ones, which would be the maximum time that could be taken for 30fps and 60 fps. This way I would have an idea where to stop optimizing.
The main issue found
Contrary to my initial intuition, most of the time was spent in the physics system. Of course, I had decided to write it myself, and it turned out to be too simple even for this easy use case.
What I was doing is, for each object, check against every other if it intersected and take the appropriate measures. Since my scene was composed of lots of small objects, this would take a really long time.
Real physics engines have a list of “active” objects and only those will be checked at any point in time. If any one of them has not moved for a while and it is not in the vicinity of any big change, it will be removed from the “active” list and put to sleep.
I would not implement such a complex change. I decided to simplify the common procedure by only checking the objects that moved since the last run. I was still checking that object against all others, but that was more than acceptable. This was good enough since at the start of the game no object would overlap. At any point in time, it would be possible for two objects to overlap if any of those moved on the previous frame.
This ended up being good enough for making all lines small enough to fit inside the 30 frames per second line during the gameplay. I originally thought this would be good enough, however…
More measurements were needed
Every few seconds, the game would stall for a moment and then continue. All the operations I did at any point in time were already measured, so it became evident that the issue was somewhere out of my control.
Most of my software development experience up until that point was with compiled languages like C++, and the few times I used something like Javascript performance was not important. It looked like the problem would have something to do with the C# garbage collector, but that was uncharted territory for me.
Since the game was loading and unloading lots of assets, I feared I was at some point not managing them correctly. Maybe I was unnecessarily loading them halfway through the execution, or they became dangling from previous stages and needed freeing.
I added an extra line to my homemade debugger, that would show the total amount of memory my game was using. I could see this growing considerably when loading, and then shrinking appropriately when my assets were not needed anymore.
However, a small amount of space was taken every on a drawn frame, only to go down every so often when the garbage collector freed it up, causing the noticed stutter. It was time to find better ways of measuring memory. I found such tools to be available for the C# language. I could just play the game for a while whilst the tools were keeping track of where objects were allocated and freed and just look at the places where this happened most often.
I didn’t need to create any specific tooling for this because of the nature of the software, since it was enough to see the accumulated values throughout the execution.
And the answer was quite unexpected in this case. The problem came mainly for constructs such as this:
In there, both lightPosition and lightColor would be created and then deleted from the heap. The inner part of the loop would get executed thousands of times on every update, creating the stutter.
The fix was as simple as changing the code like follows:
The variables would only be created and disposed of once per update.
Summary
Optimizing software performance is hard. In some cases, like games, it becomes harder. In any case, the only proper way of achieving good results is to start with proper measurements. Any time spent on this will soon payback.
