Pur Autre Vie

I'm not wrong, I'm just an asshole

Saturday, September 17, 2011

Nothing But Data

This post is about a forum post on teamliquid.net. The rest of my post won't make sense unless you've read the post, so I suggest you read it and come back.

I think the teamliquid post neatly illustrates the necessity for both empiricism and analysis - you generally can't get far with one if you ignore the other. However, I should preface what I say: I think the post is a good post, and I'm glad it was written. I just don't think it can support the weight that is being placed on it (for instance, commenters went so far as to try to find a way to calculate "SQ" in SC2 Gears - alas, apparently this isn't feasible).

So let's get to my main criticism of the post: it derives a metric called the Spending Quotient (SQ) and implies that SQ measures "macro skills" - the ability to obtain and spend resources efficiently. But the post offers only tenuous support for the idea that a player should specifically try to maximize SQ.

Here's an analogy that will apply to the rest of my post. Arthur Okun described a metric called the "misery index," intended to measure the harm caused by unemployment and inflation. The misery index is calculated by adding the unemployment rate to the inflation rate. Now, imagine that Okun had published a forum post on teamliquidity.net, describing the misery index. Should central bankers respond by minimizing the misery index as a matter of policy?

Well, no, because the misery index has little normative weight. It combines two things that are generally undesirable above a certain level and calls for minimizing them. But the weight that it attributes to unemployment and inflation is arbitrary.

SQ has this quality as well, although the forum post goes to some length to justify the relative emphasis that it puts on unspent resources and average income. The question is whether the post gets from point A to point B, and I think the answer is no. But quickly, without referring back to the teamliquid forum post, see if you can connect the dots yourself: what is it about SQ that is supposed to be meaningful, beyond the fact that unspent resources should generally be kept low and average income should generally be kept high?

I think the answer, if you tease it out, is that within each league there seems to be a relationship between unspent resources and average income, and SQ essentially represents movements that are perpendicular to the resulting isoquant (changes in SQ represent shifts between leagues, while movements along the isoquant represent different games within a league). Increasing SQ is therefore supposed to be the most efficient way to climb from one league to another.

Now bear in mind, it's uncontroversial that it's generally best to minimize unspent resources and maximize average income. If I post a graph like this:

and I tell you to generally move up and to the left down and to the right [oops!] in order to improve, that is not interesting. What is interesting is that whatthefat (author of the forum post) has derived a mathematical formula that purports to quantify the relative importance of improvements in unspent resources and average income.

So, what's the matter with drawing the arrow that seems like the most direct path from one league to another? Well, this depends on the assumption that there is a causal relationship between SQ and winning. Causation is always a slippery concept that leans heavily on intuition, but there are several problems with the causal story that whatthefat is implicitly telling (whatthefat never get beyond the observation that higher-league players have higher SQs, so any causal connection is left to the reader to sort out):

1. The measure of SQ will depend not only on the player's performance but on the duration of the game and the course of events within the game. Thus, SQ could vary between leagues entirely as an environmental factor and we wouldn't be able to detect this from whatthefat's data.

2. In a very odd omission, whatthefat neglects to use the available data to test the causal effect of SQ. After all, whatthefat has the win/loss data for each game. Does a player win more often when his SQ is higher? whatthefat knows (or could know), but we don't. (It would also be interesting to see if the player with the higher SQ in each game won. whatthefat didn't compile data to calculate the opponent's SQ - why not? - so this is unkknowable.)

3. Note that, if you take SQ to be a measure of macro skills, a bronze player will regularly out-macro a diamond player, and a bronze player will out-macro a grandmaster league player a non-negligible amount of the time:

What this implies is that the variance in SQ is quite high, which probably indicates that it fails to isolate macro skills and is instead picking up a lot of noise. Whether game-to-game changes in SQ are meaningful is not clear.

So to sum up, I think whatthefat gets credit for compiling useful data but has not persuasively shown that SQ is a useful derivation of that data. Any formula that tells you to minimize unspent resources and maximize average income is going to be "useful." To treat SQ as being useful, beyond the fact that it increases with respect to average income and decreases with respect to unspent resources, would require a demonstration that it contributes to winning. whatthefat doesn't provide any such demonstration, and so we are left with our own personal intuition on the subject. Mine is that SQ is the StarCraft analogue of the misery index, and that StarCraft players would be well-advised to ignore it and focus on the known contributors to winning.


Blogger Dongdong Weng said...

michael kors outlet online sale
true religion sale
chanel handbags
ray ban sunglasses
ralph lauren outlet
michael kors
christian louboutin outlet
celine outlet
hollister clothing store
michael kors handbags
jordan 9
burberry outlet
jordan 6 cigar
kate spade outlet
coach outlet
ghd hair strighteners
michael kors handbags
cheap jerseys
marc jacobs
toms shoes
prada uk
burberry sale
louis vuitton
toms outlet
ralph lauren polo shirts
replica watches
kate spade outlet
ray ban outlet
gucci shoes
polo ralph lauren
ray ban uk
louis vuitton outlet
louis vuitton purses
coach factory outlet
hollister clothing

3:31 AM  
Blogger Dongdong Weng said...

oakley sunglasses cheap
nike air max
michael kors handbags
michael kors outlet online
celine outlet
michael kors
tory burch outlet
christian louboutin
chi flat iron
toms outlet
prada outlet
cheap jordans
football jerseys
ray ban sunglasses uk
michael kors
polo ralph lauren
michael kors outlet
toms outlet
timberland boots
insanity workout
michael kors
abercrombie fitch
hollister co
air jordan 13
replica watches
michael kors outlet
gucci shoes
kate spade handbags
louis vuitton handbags

9:55 AM  
Blogger Jian Zhuo said...

nike flyknit
coach factory outlet
ugg canada sale
michael kors outlet store
gucci outlet online
kate spade uk
adidas nmd runner
cheap jordan shoes
coach outlet store online clearance
polo ralph lauren outlet

4:24 AM  
Blogger 郑佳颖 said...

adidas gazelle sale
true religion outlet
ferragamo sale
longchamp online shop
adidas shoes
yeezy sneakers
kyrie irving shoes
nike air force 1
adidas neo
adidas tubular UK

7:58 PM  
Blogger 5200 said...

coach outlet store online
michael kors outlet clearance
giuseppe zanotti
pandora jewelry
coach outlet
coach outlet online
rayban sunglasses
michael kors outlet
true religion jeans
san francisco 49ers jerseys

2:13 AM  

Post a Comment

<< Home