Our RNG Git Hash Bug | Tim Mendez Blog

Background

I work on the live services game Dragon Prince: Xadia. We have iOS/Android clients, hosted gameservers, and a platform. Our gameservers and clients need to be in sync, so we have a field in our platform’s YAML configuration:

gameServerVersion: <gitHashOfDeployedCommitGoesHere>

This way, we can tell clients when they are out of date and need to update. This value is set dynamically by a TeamCity deployment job.

The Bug

One day, a good long while after we put this system in place, our QA lead pings me in the #game-ops channel on Slack:

@tim_m @bug there’s something wonky with Sandbox2. Deployment is finished for platform/server/Android and I downloaded the new Android client/patched and it’s still reporting that there’s an update

Clients for our Sandbox2 shard are blocking access with a “you gotta update” message.

A Brief Investigation

I’m assuming the deployment didn’t actually succeed, or either the platform or client was overwritten with the wrong git hash by accident. So, I pull down the Sandbox2 shard’s yml config and look at the gameServerVersion value.

gameServerVersion: 556474e378

I jot that down and take a look at our TeamCity deployment history. The git hash used in both client deployments matches this value.

I’m a bit perplexed here, so I download the iOS client myself to make sure it’s not an Android caching bug (of which we’ve had a few). Our QA lead, of course, has already gone through all the cache clearing steps for Android, with no success. I boot up iOS and see the same “you gotta update your client” message.

Our QA lead posts the client log for me to comb through. I’m expecting to see the client pointing to a different platform, or something to that effect. Instead, I see:

"gameServerVersion": "Infinity",

as part of the GameData payload, which we compare to the version bundled with the client.

Well that’s probably it. HMMMMM.

The Cause

I look back at the git hash and immediately see what’s wrong this time: quotes are missing.

556474e378 also happens to be scientific notation for, well, a very large value. It’s also expressed as 556474 × 10³⁷⁸.

YAML parsers will interpret values as strings even if quotes aren’t present, as long as the value is definitely not something else, like a number, boolean, etc.

Every short git hash that we had rolled before now happened to not be able to be interpreted as a number, so gameServerVersion was always being interpreted as a string.

The Solution

I wrap the git hash in quotes and, sure enough, the error message goes away.

gameServerVersion: "556474e378"

I also update our TeamCity code to wrap the git hash in quotes to fix this bug for future git commit hashes.

A git hash that can also be interpreted as mathematical expression was not on my initial list of causes for the bug.