I hope you’ve all been enjoying Shamus’ series on the art of programming, which I understand is writing special words that make games happen. Sometimes you don’t write the words good enough and the game isn’t good; John Romero did this one time and he’s been working as a garbage man in Tulsa ever since. I’m the lead writer, so programming isn’t really my department, but having accidentally opened the source code while Arvind was explaining TortoiseHg again I can see why Shamus has been having so many problems–to be frank, the grammar was terrible and almost three quarters of the words were misspelled. I did an editing pass which I assume fixed most of the problems; Shamus has assured me this will be the topic of posts #43-129.
But I think we’ve all got the basic idea: coding is “hard” and “interesting” and “requires technical skill” and “can be objectively assessed.” But is it really the most fundamental part of a videogame? Shamus and Arvind say “yes,” repeatedly, at progressively louder volumes–but I’m not convinced. If you take away the code I’m sure a videogame will still run, but can you say the same about its story? What would Killzone, Rocket League, and Neko Atsume be without their rich internationally beloved canons? And if it wasn’t for Final Fantasy villains, how would you know which of your old forum accounts to be slightly embarrassed by?
My point is that my job writing Good Robot (or more precisely, writing a couple hundred headlines that display when interacting with vendors, plus some names to go with level bosses) is exactly as critical as the stuff Arvind and Shamus do all week. I’m guessing. They keep forgetting to tell me when their meetings are.
So in the vein of the rest of this series, here’s a few days in the life of the Lead Writer (!).
Good Robot has a problem. It’s a strange, goofy, inexplicable problem and I’m pretty sure (60%-ish) that it’s not my fault. Here is what’s up:
Our game is capped at 60fps. That’s fine, except the cap isn’t self-imposed. Oh, I have a frame-limiter in the game, but it doesn’t do anything. If I disable it, the game is still limited to 60fps. Even if I render nothing more than a blank screen, I can’t get the framerate to go above 60. Under those conditions, the framerate should be in the thousands.
Good Robot is almost done, and we are on course to finish the remaining tasks in the next couple of weeks. We’ll release the game in the first week of April, which should give us some breathing room for testing and polish.
However, there is another reason we are launching the game two months after we’re done making it – promotion. This is the part where you email every single Game Journalist / YouTube Personality / Twitch Streamer / Person with a Blog in an illegal-substance-fueled-frenzy and hope they play your game and tell others about it.
You have to cover our game! It has an exploding Frisbee that bounces off walls in it!
“Why do you need to promote your game, Arvind?” I hear my friend Manny Straw exclaim, “If your game is any good, surely you can just put it up on Steam and people who see it will tell their friends about it, and then those friends will tell their friends, and soon you’ll sell a million copies! That’s how Minecraft did it!”
My to-do list grows and shrinks as the project rolls on. I’ll have 20 items on my to-do list one week. I’ll get 13 of them done. Then at our next weekly meeting, those 13 are reviewed. Some are marked as done. Some end up back on the list because my solution was too narrow, or didn’t work in all cases, or I misunderstood the problem. Then a few new issues will get piled onto the list.
So after the meeting my to-do list will be back up to 25 or so items and we’ll begin again. So it goes.
But some items have never been touched. They’ve haunted the bottom of the list, never getting done, never getting looked at. The oldest item on my list now is actually a collection of bullet-points that can all roughly be summed up as “performance problems”. To wit: The game runs too slow.
Not on my machine, mind you. It’s fine on my machine. But on craptops (i.e. really old/slow computers) it runs at about half the framerate it should. So what’s going on?
Some other indie developers were nice enough to play the game and send us their feedback. A common theme in the feedback was that things were too chaotic. Or too random. Or unfocused. Or too busy.
Looking at the game, it’s easy to see why, and it’s easy to see how we slipped into this state. We made a system that let you make wildly divergent robots, simply by tweaking a text file. Since creating robots is easy and variety is good, then more robot types = more good, right? Aren’t games always bragging about how many enemy types they have? Isn’t this a selling point? “Fight over 12 different enemy types!” it says on the back of the box.
Only 12, AAA game? Pshaw. We have that many in the first 15 minutes of the game!
It made sense at the time, but when the feedback rolled in it was a forehead-slapping moment for all of us. Of course this will seem like random chaos to someone who hasn’t played the game constantly for 6 months.
It’s like a version of Half-Life 2 where your first fight is against a group of foes with the behaviors of an antlion, a zombie, two soldiers, a metrocop, a strider, and a gunship. It’s not about being “too hard”, really. Even if you lower the hitpoints and damage output on the gunship and the strider to bring them into line with (say) a metrocop, the player still can’t be expected to parse all this chaos. They’re not going to appreciate the differences between the soldier and the zombie when both foes die in the same panicked volley of weapons fire.
When you’ve got more than one person working on a complex bit of software, you generally need a specification (spec) for new features. The bigger the team, the more you need a spec. The more complex a feature, the more you need a spec.
According to stereotypes, big firms usually lean too hard on specs, to the point where they might spend more time writing the specs than coding the feature:
“The button will be ten pixels from the left margin and will conform to the usability guidelines sheet 201-a. It will be labeled “Join Game” and will – after a confirmation popup as outlined in the interface framework – begin polling the designated server in request for an open slot. If no slot is found, then the fallback behavior […]”
Meanwhile, little indie houses have a slightly less formal approach:
Bruce: Can you add a button that will let players join the game? Barbara: Sure.
Stuff gets done either way, but sometimes indies are a little slapdash and sometimes big firms are a little too bureaucratic. On Good Robot, our spec is usually a sentence or two in the shared Google doc that we use as a universal to-do list.
But this week I ran into something that I realized was too complicated for that. It was one of those features that sounded obvious and simple in the meeting, but then became mysterious when I sat down to write the damn thing. (This is the point of a spec: To reveal the unknowns BEFORE coding begins. This is important in big firms, since once you’ve begun coding you’ve ALREADY been allotted a fixed time budget, which means this is a bad time to begin figuring out what you need to do.)
Let’s talk about little things that make a big difference.
The game was lacking something. It was just too static. The world didn’t move, didn’t animate, didn’t react. The screenshots looked good, but when you were playing the game it felt like you were flying past a painting or something.
So my first solution is dust particles. Here is a shot of them, zoomed in so everything looks terrible:
The little specks are the thing we’re interested in here. In still frame, they just look like part of the background. But in gameplay they drift around. They get brighter when you hit them with your flashlight beam and they’re obscured by walls.
Like a lot of “field” effects, this one is pretty simple. There are only 400 dust motes in the game. They drift slowly, so the game doesn’t even have to move all 400 every frame. Each one of them moves in a different (but constant) direction. If it drifts off one side of the screen (either because you’re moving, or it is) it simply re-enters from the opposite side. If you could zoom out, you’d see your character has a rectangle of white particles following them around.
The more I work with a team of people, the more I’m convinced that having open, accessible game data is the path of least resistance. Why make buggy, lame, proprietary in-house tools when you can just stick all the data into text files and let people use their text editor of choice? Why spend time and effort packing simple data into binary files when you can leave it in plain text? As long as the data isn’t binary in nature (text-based 3D models and sound would probably not be a good idea) then open files are a win for everyone: Easier for coders, more comfortable for the artists, and more mod-friendly for enterprising players.
Of course, I’ve always thought this way, but I assumed it was bias from all the years I worked at Activeworlds, which focused on user-generated content, similar to Second Life or Roblox. I often wondered if I’d gravitate towards obscured data if I ever found myself working on a “proper” game.
But no. But if anything, I’m more pro-“open data” than ever.
But what if the users edit their data files to cheat and give themselves a billion hitpoints?
Yeah. Not a concern. Stopping single-player cheating is a lot like stopping pirates: It can’t be done, but if you’re really creative and determined you can waste a lot of time and money trying.
Early in the project, I had a lot of stuff hard-coded. Certain gameplay parameters were set in stone, and you couldn’t change them without changing the source code and patching the game. That’s basically fine for a one-person team. When I’m working alone, it’s just as easy to change a bit of source code as it is to change some text file of game data. But once Pyrodactyl joined, more and more of the game migrated out of code and into text files the artists controlled.
The only downside is…
It takes HOW LONG to launch the game?
The game used to launch in well under two seconds. Over the past few months, that number has been creeping upwards. This week I noticed it topped ten seconds. That seems unreasonable. We’re certainly not processing five times as much data at startup.
If anything, the game should be loading a little faster than before. Originally all of the level geometry drew from a single gargantuan atlas texture, which contained the tilesets for every level in the game. This texture was always read at startup, which probably had some modest time cost. But one of the problems Arvind ran into with Unrest is that some craptops can’t handle textures over 2048×2048. So, we chopped up the scenery textures so it only loads one at a time.
So what’s causing this slowdown?
That’s a Really Good Question.
I’ve always said that Microsoft makes horrible software, except for their software to make software, which is absolute excellence. When I left Activeworlds, I lost access to the corporate license of Professional Edition of Visual Studio and had to switch to the Hippie Freeloader Edition. (Called “Express”. Because nothing speeds your progress like missing features.)
It’s still great, but it was missing a good profiler. Profiling tools analyze the program as it runs, and show you where all the CPU cycles are going. It’s not something you need every day, but when you need it you really need it.
This year – in a mad fit of behavior so un-Microsoft it’s borderline suspicious – they began offering the super-premium version to the public, for free. So I have access to profiling tools again. Let’s see what it says about Good Robot:
I guess I sort of gave it away in the intro, but the result was actually a surprise to me. I would have expected the time to go into loading bulky texture data. (Nope. Nearly instant.) Maybe loading the 70 different audio files that comprise the sound effects for the game? (Nope. Trivial.) Or maybe the bit of code that examines the individual sprites a pixel at a time so that it can do per-pixel hit detection? (Eh. That’s a little heavy, taking around 11% of the startup time. But it’s not the problem.) Launching the engine? (Trivial.) Loading the shaders? (So fast you basically can’t measure it.) Loading the interface? (Not really.)
But no. The long load times were the result of reading text files.
Almost 40% of the startup cost is spent in EnvInit (), which is nothing but text parsing. And it’s not even all the text parsing. There are other text parsers at work elsewhere. They’re doing more in less time, because those other parsers are working on XML!
For those that don’t know: XML files are like text files, except they’re huge and cumbersome and barely readable. My two favorite XML quotes are:
“XML is a classic political compromise: it balances the needs of man and machine by being equally unreadable to both.” – Matthew Might
“XML combines the efficiency of text files with the readability of binary files” – unknown
But I probably shouldn’t be chucking stones at the XML camp while I’m living in the glass house that is my own text parser. XML might be huge and ungainly, but the game is loading it far, far faster than the rest of our game data, which is in a far simpler format. A great deal of time is spent reading the weapons in do_projectile_inventory (). The weapons file is a minuscule 25k, and holds just 48 weapons. It’s ludicrous that it should take 17% of the startup time reading that file. In the time the game spends laboriously reading that file, you could probably read it in from disk and write it back out again a hundred times over.
In an ideal world, there would be a single clog somewhere, one bit of inefficient or malfunctioning code that’s being called hundreds or thousands of times. But this is not an ideal world, and it seems like the problem is smeared all over the place. The problem is that I’m using strings poorly.
Way back in 2010, I had a post about what a pain in the ass it is to have to manually manage the memory of strings of text. You have to do lots of fussy work to allocate memory and make sure you handle it just right. Something that could be done in 2 lines of code in another language might take six or seven to do properly in C. And if you messed up you’d waste memory, slow down your program, or crash. So it was a lot of work for a lot of danger.
But that’s not how today’s hip youngsters do things. No! They have C++, which comes with sexy new ways to handle text.
I’ve been gradually transitioning to the “new” way of doing things over the last few years. (“New” is a really strange term in programming. I might mean “since last week” or “since the mid-90’s”, depending on context.) But there’s a difference between learning to do something and learning to do it well. I haven’t learned – or even bothered to investigate – the common pitfalls of the new system.
As a result, my text parser was doing a ton of needless work. Something like this:
It’s time to compare two strings!
But we don’t care about upper / lowercase. So take the first string and MAKE A COPY OF THE WHOLE THING, then convert the individual characters to lowercase, then take the contents of the copy and overwrite the original.
We also don’t care about whitespace. If we’re comparing “Apple” and “Apple “, they should match, even though the second one has a bunch of spaces after it. So MAKE A COPY OF THE WHOLE STRING.
Now take that copy AND MAKE ANOTHER COPY, and pull the spaces off the end. Now MAKE ANOTHER COPY and pull the spaces off the beginning. Now give the new, cleaned-up version of the string back to the original.
Now do ALL of that to the second string.
THEN compare them.
That doesn’t sound that bad, right? Sloppy, yes. But, copying a few extra bytes a few extra times should be harmless on a modern computer.
Except, when you need to look for thing A in group B. If thing A is a zucchini and I need to look for it in the list of 48 different vegetables, then I need to do that whole block of actions for every veggie until I find a match. As the list grows, the amount of extra copying can quickly get out of hand. The time cost can ramp up quickly, particularly in my implementation where I was frequently searching through lists within lists.
Does This Matter?
Well, not really. Not in the long run. This was something wrong, and it was wrong in an interesting and unexpected way, which is why I wrote about it. A ten-second load screen is not a horrible sin, and there are many games that take much longer to do much less.
But I’m also kind of picky when it comes to loading times, because loading slows down iteration and iteration is important. People love to say that “premature optimization is the root of all evil.”, but I think there’s a really good case for keeping the startup times snappy, particularly as team size grows.
Most of the people working on your game are probably not programmers. They’re artists. They’re making constant changes. Change the color. Add a few hitpoints. Swap out a new wall texture. Drop in a new sound effect. Tweak a particle effect. Replace a 3D model.You can’t make every part of the game editable while the game is running, which means your artists are going to be restarting the game often. In some cases, they might spend a majority of their time sitting and waiting for the game to start.
If you’ve ever tried to photoshop a large image on a crappy computer then you probably know what I’m talking about. It’s frustrating to have to wait a few seconds to see the results of every action. Just a two-second delay can be maddening. A ten second delay is even worse. It’s long enough to break your flow, but not long enough that you can (say) check Twitter or look for cat pictures. It’s a gap of dead time and it can kill creativity. I don’t know how other people respond, but when I have a lot of creative latency like this it really increases the odds that I’ll stop working as soon as it’s “good enough” instead of polishing things until it’s “great”.