The IFComp has just ended so it's a good time to compare historical IFComp results with scores given to games in IFDB. Are there any differences?
Here are the charts from IFComps 1999–2012, based on IFDB data from November 1st, 2013. I couldn't find IFComp scores from before 1999 and there wasn't enough data for 2013 yet.
The games are placed in the chart with horizontal axis showing the IFComp score and vertical axis showing the IFDB score. Hovering the mouse over the dot displays the title. The blue diagonal line is the trendline. If the dot is above the trendline, it means it has been ranked higher in IFDB than in IFComp (in relation to other games in the same comp). Conversely if the dot is below the trendline the game did better in the comp than in IFDB. The distance from the line tells how big the disrepancy is.
Only games with at least 5 votes in IFDB are counted. That includes everything from 2008 on. Older bottom-tier games tend to have fewer votes. The only disqualified highly-ranked game is A New Life with only 4 votes which tied for second place in 2005.
(The charts might not show up if you're reading this from the RSS feed or Planet IF.)
If we look at how IFComp winners are rated in IFDB compared to games released the same year we notice that the best games aren't always found from the competition. In fact based on IFDB ratings the comp winner is the best-rated game of the year in only two cases, and more than half the time the comp winner isn't even the best rated IFComp game in IFDB.
The XYZZY award for the best game of the year goes to an IFComp winner roughly one third of the time, and one third of the time it isn't even nominated. Only Lost Pig in 2007 hit the jackpot (IFComp winner, highest rated game of the year, XYZZY award).
In the table below the first number column is the ranking among the competition games and the second is the ranking among all games that year. The last column shows if the game was nominated or won the XYZZY award for best game. For example in 1996 The Meteor, The Stone And A Long Glass Of Sherbet won the comp, was nominated for XYZZY and in IFDB is rated the 5th best comp game and the 7th best game of 1996.
| IFDB ranking | ||||
|---|---|---|---|---|
| Year | Comp winner | comp | year | XYZZY |
| 1995 | A Change in the Weather | 2 | 9 | |
| 1995 | Uncle Zebulon's Will | 1 | 4 | |
| 1996 | The Meteor, The Stone And A Long Glass Of Sherbet | 5 | 7 | nom. |
| 1997 | The Edifice | 3 | 3 | nom. |
| 1998 | Photopia | 1 | 1 | nom. |
| 1999 | Winter Wonderland | 4 | 10 | no |
| 2000 | Kaged | 8 | 12 | no |
| 2001 | All Roads | 5 | 13 | won |
| 2002 | Another Earth, Another Sky | 5 | 10 | nom. |
| 2003 | Slouching Towards Bedlam | 1 | 2 | won |
| 2004 | Luminous Horizon | 5 | 10 | no |
| 2005 | Vespers | 1 | 2 | won |
| 2006 | Floatpoint | 5 | 11 | nom. |
| 2007 | Lost Pig | 1 | 1 | won |
| 2008 | Violet | 1 | 2 | won |
| 2009 | Rover's Day Out | 1 | 5 | no |
| 2010 | Aotearoa | 1 | 7 | won |
| 2011 | Taco Fiction | 3 | 6 | no |
| 2012 | Andromeda Apocalypse | 3 | 16 | nom. |
(Data retrieved 2013-11-21.)
Here are the "best of"s of each year. The number in the "IFDB best of comp" column is that game's actual placing in IFComp that year.
| Year | Comp winner | IFDB best of comp | IFDB best of year | XYZZY best game | |
|---|---|---|---|---|---|
| 1995 | A Change in the Weather Uncle Zebulon's Will |
Uncle Zebulon's Will | 1 | Christminster | |
| 1996 | The Meteor, The Stone And A Long Glass Of Sherbet | Lists and Lists | 11 | Lists and Lists | So Far |
| 1997 | The Edifice | Babel | 2 | Babel | I-0 |
| 1998 | Photopia | Photopia | 1 | Photopia | Spider and Web |
| 1999 | Winter Wonderland | Exhibition | 5 | The Mulldoon Legacy | Varicella |
| 2000 | Kaged | Metamorphoses | 2 | Metamorphoses | Being Andrew Plotkin |
| 2001 | All Roads | Heroes | 3 | First Things First | All Roads |
| 2002 | Another Earth, Another Sky | Janitor | 5 | Savoir-Faire | Savoir-Faire |
| 2003 | Slouching Towards Bedlam | Slouching Towards Bedlam | 1 | City of Secrets | Slouching Towards Bedlam |
| 2004 | Luminous Horizon | All Things Devours | 3 | All Things Devours | Blue Chairs |
| 2005 | Vespers | Vespers | 1 | Tomorrow Never Comes | Vespers |
| 2006 | Floatpoint | The Primrose Path | 2 | Bronze | The Elysium Enigma |
| 2007 | Lost Pig | Lost Pig | 1 | Lost Pig | Lost Pig |
| 2008 | Violet | Violet | 1 | Blue Lacuna | Violet |
| 2009 | Rover's Day Out | Rover's Day Out | 1 | Make It Good | Blue Lacuna |
| 2010 | Aotearoa | Aotearoa | 1 | Flexible Survival | Aotearoa |
| 2011 | Taco Fiction | Kerkerkruip | 8 | Mentula Macanus: Apocolocyntosis | Cryptozookeeper |
| 2012 | Andromeda Apocalypse | howling dogs | 11 | Endless, Nameless | Counterfeit Monkey |
(Blue Lacuna is listed as a 2008 game in IFDB, but it was eligible for XYZZYs in 2009. If you count Blue Lacuna as a 2009 game then Violet is the highest rated in 2008.)
What's obvious here is that people have different criteria for rating games in IFComp and IFDB. Lists and Lists, which teaches how to program in Scheme, is technically impressive but hardly something that people would want to win the IFComp.
There's also an inherent bias with this kind of scoring where you obviously need to play the game before you can rank it. When you're participating in the IFComp as a judge you often play the games without knowing anything about them beforehand, or play and rate them all even if you do. Outside the comp you're likely to not even pick up a genre that doesn't interest you; if roguelikes are not your thing you're unlikely to play and rate Kerkerkruip.
Moral of the story? Firstly, even if your game doesn't rank well in IFComp it might still be highly appreciated in other contexts or as time passes. Secondly, releasing in IFComp is not (and has never been) required for a good game to get the appreciation it deserves.
And finally here are the best IFComp games ever according to IFDB.
| Game | Year | Comp ranking | |
|---|---|---|---|
| 1 | Lost Pig | 2007 | 1 |
| 2 | Violet | 2008 | 1 |
| 3 | Photopia | 1998 | 1 |
| 4 | Slouching Towards Bedlam | 2003 | 1 |
| 5 | Metamorphoses | 2000 | 2 |
| 6 | Heroes | 2001 | 3 |
| 7 | Babel | 1997 | 2 |
| 8 | Moments Out of Time | 2001 | 2 |
| 9 | The Best Man | 2000 | 15 |
| 10 | Exhibition | 1999 | 5 |
This is really, really cool. Thanks for doing it!
Woo, I was waiting for this! I was even thinking of asking if you ever wanted to write a stats article for SPAG. Guess I'm too late!
There are a few interesting things at work here...
since the comp is explicitly about shaking down a limited subset of games, most-liked to least-liked, there's an emphasis on rating them that is not present when given a list of all works of IF ever -- indeed in the grander scheme of things a game is likely only to stand out enough for someone to bother giving it a rating at all if it's exceptional, that is if it warrants being recommended or rewarded or similarly to warn people that it's unredeemable. No one ever logs in just to give a middle-of-the-road game a 5/10.
That said, the games in a given comp are only calibrated to weigh against each other, so eg. a game that places #3 in one year might actually have an overall higher score than a game placing #1 in a different year. Is a game that places 25th out of 25 games actually better than one that places 31st out of 31?
It might be interesting or useful to count the comp votes -- either the overall numbers or the comp ranking -- as a rating source in the IFdb itself. Mobygames did all sorts of interesting things with scoring sources (including, for a time, SPAG), so you could see what the critics thought and also what the site users thought, though it's true that with a niche such as ours, the two groups have a lot of overlap... and if the site incorporated comp votes it would be a pain to prevent doctoring such as someone who voted once in the comp voting again as a site member. (This is not an insurmountable problem, if the comp website were to be integrated with the IFDB.)