IFComp 2009: Correlation between rating and the number of testers

Juhana — Wed, 25 Nov 2009 13:33:30 +0000

Everybody’s always talking about how important it is to have your game tested (or at least I’m always talking about it). But does it really matter? Surely if you have a great idea and enough enthusiasm you can do without?

Well, no. People’s tolerance of broken or flawed games seems to go down every year. On the other hand it is delightful to see that those broken games are getting more and more rare. This year only 5 games didn’t have any testers. Yes, five. That means 79 % of games had at least some testing. Compared to the last year’s number, which was about 50 %, this is a huge improvement. I feel happy and warm inside.

So, how did the number of testers affect the final rating? Here’s a handy scatter plot chart.

RATING (from 1 to 24)

The correlation looks quite clear, doesn’t it? One game had 15 testers, three games had 16. Those all ranked from 2 to 5. Only the winner had a little bit less — 9 (but it did have two authors, when the design and testing process might arguably be a bit different).

Conversely, the bottom three had no testers at all. The other two games that had no testers ranked 18th and 19th, only two games separating them from the other untested works.

Here’s the same with bars:

1	Rover’s Day Out	9
2	Broken Legs	16
3	Snowquest	16
4	The Duel That Spanned the Ages	15
5	Earl Grey	16
6	The Duel in the Snow	7
7	Resonance	2
8	Interface	8
9	Byzantine Perspective	10
10	Grounded In Space	5
11	Yon Astounding Castle!	4
12	Condemned	1*
13	Eruption	2
14	Beta Tester	1*
15	The Ascot	1
16	Spelunker’s Quest	3
17	The Believable Adventures of…	4
18	The Grand Quest	0
19	Star Hunter	0
20	GATOR-ON, Friend to Wetlands!	1
21	Gleaming the Verb	4
22	zork, buried chaos	0
23	Trap Cave	0
24	The Hangover	0

The correlation becomes even more clear when we compare the number of testers to the average score of each game:

SCORE (from 10 to 1)

I don’t think I have to spell out what the lesson is to be learned from this data.

* Note that it wasn’t easy to get an absolute number of testers for some games. Yon Astounding Castle named one tester and mentioned “anonymous testers”, but not their amount. Condemned and Beta Tester did not credit any testers in-game (boo) but evidence elsewhere on the Internet reveals that they had testers. Both have been marked down as having one tester, but in truth they could have had much more. Many other games credited people for tools they had made, inspiration, technical support or other help. These are not counted in the numbers. If you have more accurate information, please let me know so I can update the charts.

Update: The author of Yon Astounding Castle confirms that there were 4 testers total.

Undo Restart Restore » Testing

IFComp 2009: Correlation between rating and the number of testers