Response to The Civics of Potential Video Games

This post mostly serves as a methodological criticism of the article from the Games for Change Festival titled, “The Civic Potential of Video Games.” Before I begin on this exercise, I wanted to say that the article serves as potential for public discourse on video games in a positive and constructive light. It is not very often that this happens. For years, video games have been assaulted as the cause of rising violence in children. A summary and critique of this literature can be found here entitled, “Video Game Violence: A Review of the Empirical Literature”.

There are several important concepts in this paper that make it very worth reading from a topical standpoint:

Youth, particularly teenagers, need opportunities to develop and structure their own civic/political experiences through peer-to-peer learning and reflection. Incorporating these kinds of spaces into game design could potentially contribute to teen civic identity development. (forward pg 3)

There has been a tremendous amount of literature on the educational potential of video games and video game simulators. That they mention this and incorporate it into their study is a big plus.

Although public debates often fram video games as either good or bad, research is making it clear that when it comes to the effects of video games, it often depends. Context and content matter. (8)

This is a great statement and I hope that future studies along these lines develop this idea. Looking at the violence paper, one can see that most studies revolve around violent video games when they involve a specific game. Otherwise they try and correlate violent behavior with video games to other media, like television or movies or music.

We therefore chose to examine whether the digital divide applied to civic gaming experiences…In short, we wondered whether the distribution of civic gaming experiences in video games might propagate (or perhaps redress) the inequalities in civic learning opportunities that exist elsewhere in society (18)

Their data backs this statement up. Prosocial behavior in adolescents is something of a hobby to researchers, much like altruism. The effects of poverty on prosocial behavior is well documented. While this study has its share of issues, it is worth reading: “Helping Others?: The Effects of Childhood Poverty and Family Instability on Prosocial Behavior”

Related to the quote above, the authors of the video game study say:

…fewer than 10% of teens frequently experience many of the civic gaming experiences we found strongly related to civic outcomes.

The digital divide and money required to teach in this manner, plus the difficulty in playing games that would actually foster these outcomes are all part and parcel of both the generational differences of teachers and student (teachers might not have grown up with video games) and student household’s income / school district average income. This is a much larger, broader problem but giving people with money something to spend it on is an excellent step.

Criticism
Any social science article is open for disagreement. Social Science calls itself a science because we are supposed to never take anything for granted until it is so tested and so concrete there can be no question as to the concept’s validity (SEE: Matters of Concern/Fact, Latour). This particular article has only a few methodological issues.

First, telephone studies are on their way out. Without any way to purchase huge lists of cell phone numbers, it is impossible to reach a significant amount of the population (around 12-13%). Further, the typical composition of land-line households is usually white, middle class, older families. This is the reason that they use a weighting variable for their data.

A weighting variable makes a sample seem more balanced by giving non-white respondents more mathematical power. Typically, it is an attempt to make a dataset worth using as, without it, it would not be. It is common practice, but is a reminder that cold dialing is becoming less and less worth doing.

A spurious relationship in a quantitative study revolves around the idea that a concept, while correlated or related (regression is a correlation statistic), might not be actually causing anything. The typical example that is used is (wikipedia / any stats book):

An example of a spurious relationship can be illuminated examining a city's ice cream sales. These sales are highest when the rate of drownings in city swimming pools is highest. To allege that ice cream sales cause drowning, or vice-versa, would be to imply a spurious relationship between the two. In reality, a heat wave may have caused both. The heat wave is an example of a hidden or unseen variable.

This is important to this article as it is never mentioned. Are prosocial behaviors actually influencing these behaviors or does the math just check out? Speaking of the math, let’s look at the regression statistics.

The normal regression equation is: Y=A+bX1+bX2+bX3….. This formula is basically saying that Y, the dependent variable (in this case, prosocial behavior), is being caused by the independent variables (in this case, Demographic variables, Parental Involvement, Frequency of Game Play, Type of Game). What this variable says, when calculated is just how much of an effect are these variables having on Y. R2 is the amount of change in Y that all of the variables explain.

So, for each of these, looking across, you see .046, .102, .051, .119, .065. This is how much all of the X’s account for the change in Y. So, 4.6%, 10.2%, 11.9%, 6.5%. The change is not a whole lot.

Moving on from here, we also do not have an alpha level. Alpha is for how significant data is. Typically, this is set at the .05 level. This means that if the same test these researchers did was done 100 times, this data would be wrong 5 of those times (wrong being not representative). This has to do with sampling. When you sample, the mean of your sample, the average of your sample, for any given score, will fall within a reasonable distance from the average of the entire population (in this case, the American Population). In any case, this number, while not that important, is not provided. The reason this is important has to do with substance vs statistical significance.

In the spurious relationship example, those concepts were mathematically significant. This is statistical significance. It just so happens, when data is thrown into a model, that the numbers add up and the program used, SAS or SPSS, STATA, spits out tables that show there is a significant relationship.

Looking at this data, I can see that their data was probably significant, even without their alpha levels or any sort of post-test, but what I can’t see is substantive significance. The best example of substantive relationships would be Education level and income. The more education you have, the higher your income (traditionally) will be. This data relationship is extremely strong and will reflect that in the data. For example, if you were to run these data through without the game play variables, I’d venture to guess that the models would be stronger.

Finishing it up
Overall, this data is interesting but fails to really give enough information to warrant any sort of conclusion. What the data does say is that these ideas are important enough to study and important enough to try and look at. Unfortunately, reaching respondents to do these sorts of studies is growing increasingly more difficult as represented by their 80%+ non-response rate.

I’d love to see their dataset so that I could run a few tests and see if their questions were accurately measuring their concepts. It would be nice to see if their significance (alpha) levels were all .000 or if they were closer to .05. I believe they would be closer to the later than the former. In the notes they say they are doing a panel study to look at the information here more closely and I’m excited to see what they have to say.

In the end this paper serves very well as an initial exploratory look at the effect of video games on behavior that isn’t violence and that is excellent.