Nothing But Data

This post is about a forum post on teamliquid.net. The rest of my post won't make sense unless you've read the post, so I suggest you read it and come back.

I think the teamliquid post neatly illustrates the necessity for both empiricism and analysis - you generally can't get far with one if you ignore the other. However, I should preface what I say: I think the post is a good post, and I'm glad it was written. I just don't think it can support the weight that is being placed on it (for instance, commenters went so far as to try to find a way to calculate "SQ" in SC2 Gears - alas, apparently this isn't feasible).

So let's get to my main criticism of the post: it derives a metric called the Spending Quotient (SQ) and implies that SQ measures "macro skills" - the ability to obtain and spend resources efficiently. But the post offers only tenuous support for the idea that a player should specifically try to maximize SQ.

Here's an analogy that will apply to the rest of my post. Arthur Okun described a metric called the "misery index," intended to measure the harm caused by unemployment and inflation. The misery index is calculated by adding the unemployment rate to the inflation rate. Now, imagine that Okun had published a forum post on teamliquidity.net, describing the misery index. Should central bankers respond by minimizing the misery index as a matter of policy?

Well, no, because the misery index has little normative weight. It combines two things that are generally undesirable above a certain level and calls for minimizing them. But the weight that it attributes to unemployment and inflation is arbitrary.

SQ has this quality as well, although the forum post goes to some length to justify the relative emphasis that it puts on unspent resources and average income. The question is whether the post gets from point A to point B, and I think the answer is no. But quickly, without referring back to the teamliquid forum post, see if you can connect the dots yourself: what is it about SQ that is supposed to be meaningful, beyond the fact that unspent resources should generally be kept low and average income should generally be kept high?

I think the answer, if you tease it out, is that within each league there seems to be a relationship between unspent resources and average income, and SQ essentially represents movements that are perpendicular to the resulting isoquant (changes in SQ represent shifts between leagues, while movements along the isoquant represent different games within a league). Increasing SQ is therefore supposed to be the most efficient way to climb from one league to another.

Now bear in mind, it's uncontroversial that it's generally best to minimize unspent resources and maximize average income. If I post a graph like this:



and I tell you to generally move up and to the left down and to the right [oops!] in order to improve, that is not interesting. What is interesting is that whatthefat (author of the forum post) has derived a mathematical formula that purports to quantify the relative importance of improvements in unspent resources and average income.

So, what's the matter with drawing the arrow that seems like the most direct path from one league to another? Well, this depends on the assumption that there is a causal relationship between SQ and winning. Causation is always a slippery concept that leans heavily on intuition, but there are several problems with the causal story that whatthefat is implicitly telling (whatthefat never get beyond the observation that higher-league players have higher SQs, so any causal connection is left to the reader to sort out):

1. The measure of SQ will depend not only on the player's performance but on the duration of the game and the course of events within the game. Thus, SQ could vary between leagues entirely as an environmental factor and we wouldn't be able to detect this from whatthefat's data.

2. In a very odd omission, whatthefat neglects to use the available data to test the causal effect of SQ. After all, whatthefat has the win/loss data for each game. Does a player win more often when his SQ is higher? whatthefat knows (or could know), but we don't. (It would also be interesting to see if the player with the higher SQ in each game won. whatthefat didn't compile data to calculate the opponent's SQ - why not? - so this is unkknowable.)

3. Note that, if you take SQ to be a measure of macro skills, a bronze player will regularly out-macro a diamond player, and a bronze player will out-macro a grandmaster league player a non-negligible amount of the time:



What this implies is that the variance in SQ is quite high, which probably indicates that it fails to isolate macro skills and is instead picking up a lot of noise. Whether game-to-game changes in SQ are meaningful is not clear.

So to sum up, I think whatthefat gets credit for compiling useful data but has not persuasively shown that SQ is a useful derivation of that data. Any formula that tells you to minimize unspent resources and maximize average income is going to be "useful." To treat SQ as being useful, beyond the fact that it increases with respect to average income and decreases with respect to unspent resources, would require a demonstration that it contributes to winning. whatthefat doesn't provide any such demonstration, and so we are left with our own personal intuition on the subject. Mine is that SQ is the StarCraft analogue of the misery index, and that StarCraft players would be well-advised to ignore it and focus on the known contributors to winning.