IFDB statistics, part 3: IFComp

The IFComp has just ended so it's a good time to compare historical IFComp results with scores given to games in IFDB. Are there any differences?

Here are the charts from IFComps 1999–2012, based on IFDB data from November 1st, 2013. I couldn't find IFComp scores from before 1999 and there wasn't enough data for 2013 yet.

The games are placed in the chart with horizontal axis showing the IFComp score and vertical axis showing the IFDB score. Hovering the mouse over the dot displays the title. The blue diagonal line is the trendline. If the dot is above the trendline, it means it has been ranked higher in IFDB than in IFComp (in relation to other games in the same comp). Conversely if the dot is below the trendline the game did better in the comp than in IFDB. The distance from the line tells how big the disrepancy is.

Chart with arrows pointing at outliers

Only games with at least 5 votes in IFDB are counted. That includes everything from 2008 on. Older bottom-tier games tend to have fewer votes. The only disqualified highly-ranked game is A New Life with only 4 votes which tied for second place in 2005.

(The charts might not show up if you're reading this from the RSS feed or Planet IF.)

If we look at how IFComp winners are rated in IFDB compared to games released the same year we notice that the best games aren't always found from the competition. In fact based on IFDB ratings the comp winner is the best-rated game of the year in only two cases, and more than half the time the comp winner isn't even the best rated IFComp game in IFDB.

The XYZZY award for the best game of the year goes to an IFComp winner roughly one third of the time, and one third of the time it isn't even nominated. Only Lost Pig in 2007 hit the jackpot (IFComp winner, highest rated game of the year, XYZZY award).

In the table below the first number column is the ranking among the competition games and the second is the ranking among all games that year. The last column shows if the game was nominated or won the XYZZY award for best game. For example in 1996 The Meteor, The Stone And A Long Glass Of Sherbet won the comp, was nominated for XYZZY and in IFDB is rated the 5th best comp game and the 7th best game of 1996.

IFComp winner rankings in IFDB
IFDB ranking
Year Comp winner comp year XYZZY
1995 A Change in the Weather 2 9
1995 Uncle Zebulon's Will 1 4
1996 The Meteor, The Stone And A Long Glass Of Sherbet 5 7 nom.
1997 The Edifice 3 3 nom.
1998 Photopia 1 1 nom.
1999 Winter Wonderland 4 10 no
2000 Kaged 8 12 no
2001 All Roads 5 13 won
2002 Another Earth, Another Sky 5 10 nom.
2003 Slouching Towards Bedlam 1 2 won
2004 Luminous Horizon 5 10 no
2005 Vespers 1 2 won
2006 Floatpoint 5 11 nom.
2007 Lost Pig 1 1 won
2008 Violet 1 2 won
2009 Rover's Day Out 1 5 no
2010 Aotearoa 1 7 won
2011 Taco Fiction 3 6 no
2012 Andromeda Apocalypse 3 16 nom.

(Data retrieved 2013-11-21.)

Here are the "best of"s of each year. The number in the "IFDB best of comp" column is that game's actual placing in IFComp that year.

Best of year in IFComp, IFDB and XYZZYs
Year Comp winner IFDB best of comp IFDB best of year XYZZY best game
1995 A Change in the Weather
Uncle Zebulon's Will
Uncle Zebulon's Will 1 Christminster
1996 The Meteor, The Stone And A Long Glass Of Sherbet Lists and Lists 11 Lists and Lists So Far
1997 The Edifice Babel 2 Babel I-0
1998 Photopia Photopia 1 Photopia Spider and Web
1999 Winter Wonderland Exhibition 5 The Mulldoon Legacy Varicella
2000 Kaged Metamorphoses 2 Metamorphoses Being Andrew Plotkin
2001 All Roads Heroes 3 First Things First All Roads
2002 Another Earth, Another Sky Janitor 5 Savoir-Faire Savoir-Faire
2003 Slouching Towards Bedlam Slouching Towards Bedlam 1 City of Secrets Slouching Towards Bedlam
2004 Luminous Horizon All Things Devours 3 All Things Devours Blue Chairs
2005 Vespers Vespers 1 Tomorrow Never Comes Vespers
2006 Floatpoint The Primrose Path 2 Bronze The Elysium Enigma
2007 Lost Pig Lost Pig 1 Lost Pig Lost Pig
2008 Violet Violet 1 Blue Lacuna Violet
2009 Rover's Day Out Rover's Day Out 1 Make It Good Blue Lacuna
2010 Aotearoa Aotearoa 1 Flexible Survival Aotearoa
2011 Taco Fiction Kerkerkruip 8 Mentula Macanus: Apocolocyntosis Cryptozookeeper
2012 Andromeda Apocalypse howling dogs 11 Endless, Nameless Counterfeit Monkey

(Blue Lacuna is listed as a 2008 game in IFDB, but it was eligible for XYZZYs in 2009. If you count Blue Lacuna as a 2009 game then Violet is the highest rated in 2008.)

What's obvious here is that people have different criteria for rating games in IFComp and IFDB. Lists and Lists, which teaches how to program in Scheme, is technically impressive but hardly something that people would want to win the IFComp.

There's also an inherent bias with this kind of scoring where you obviously need to play the game before you can rank it. When you're participating in the IFComp as a judge you often play the games without knowing anything about them beforehand, or play and rate them all even if you do. Outside the comp you're likely to not even pick up a genre that doesn't interest you; if roguelikes are not your thing you're unlikely to play and rate Kerkerkruip.

Moral of the story? Firstly, even if your game doesn't rank well in IFComp it might still be highly appreciated in other contexts or as time passes. Secondly, releasing in IFComp is not (and has never been) required for a good game to get the appreciation it deserves.

And finally here are the best IFComp games ever according to IFDB.

Highest rated IFComp games in IFDB
Game Year Comp ranking
1 Lost Pig 2007 1
2 Violet 2008 1
3 Photopia 1998 1
4 Slouching Towards Bedlam 2003 1
5 Metamorphoses 2000 2
6 Heroes 2001 3
7 Babel 1997 2
8 Moments Out of Time 2001 2
9 The Best Man 2000 15
10 Exhibition 1999 5

3 thoughts on “IFDB statistics, part 3: IFComp

  1. Woo, I was waiting for this! I was even thinking of asking if you ever wanted to write a stats article for SPAG. Guess I'm too late!

  2. There are a few interesting things at work here...

    since the comp is explicitly about shaking down a limited subset of games, most-liked to least-liked, there's an emphasis on rating them that is not present when given a list of all works of IF ever -- indeed in the grander scheme of things a game is likely only to stand out enough for someone to bother giving it a rating at all if it's exceptional, that is if it warrants being recommended or rewarded or similarly to warn people that it's unredeemable. No one ever logs in just to give a middle-of-the-road game a 5/10.

    That said, the games in a given comp are only calibrated to weigh against each other, so eg. a game that places #3 in one year might actually have an overall higher score than a game placing #1 in a different year. Is a game that places 25th out of 25 games actually better than one that places 31st out of 31?

    It might be interesting or useful to count the comp votes -- either the overall numbers or the comp ranking -- as a rating source in the IFdb itself. Mobygames did all sorts of interesting things with scoring sources (including, for a time, SPAG), so you could see what the critics thought and also what the site users thought, though it's true that with a niche such as ours, the two groups have a lot of overlap... and if the site incorporated comp votes it would be a pain to prevent doctoring such as someone who voted once in the comp voting again as a site member. (This is not an insurmountable problem, if the comp website were to be integrated with the IFDB.)

Comments are closed.

Did you find this article useful? By subscribing to the blog's mailing list you will receive an email whenever a new blog post is published. The address won't be used for any other purposes.