Starborn is a short interactive sci-fi story that uses a simple keyword interface instead of a standard IF parser. It was made as a speed-if for this year's New Year's Speed-if event on ifMUD. It was released to the public about a week ago and at first gained the usual amount of interest: the logs show 7 online plays on the release day and less on the next two days.

Last Wednesday however C.E.J. Pacian wrote a blog post about it which was picked up by the IndieGames.com blog. From there a couple of people tweeted about it and suddenly the play count jumped to about 600 plays per day. The main Parchment site reported similar amounts of traffic.

I'm running the online version of the story on a modified version of Parchment that saves the transcripts to the server. I now have 1557 transcripts consisting of 23 896 turns (and counting, but the traffic is slowing down considerably). To my knowledge this is more play data collected than for any other single IF game so far.

So here's a random bunch of analysis from those transcripts. There are no spoilers other than keywords used throughout the game, but if you haven't played yet the data would probably make more sense if you at least checked out a couple of turns just to see what the parser looks like. Online version and downloads are available.

Parser errors

Since the gameplay is just typing single keywords given by the story, the amount of understood input was substantially high (93%). Most of the errors were typos (53.3% of all invalid input).

Standard IF commands were disabled so they count as unrecognized commands. The "sign of confusion" category contains commands like HUH, ?, WHERE, or verb commands that don't follow the standard IF convention (I GO OUT). A "poke" is the player testing the limits of the parser by trying to communicate with it (HELLO, BYE, OH NO) or trying to do things out of character, mostly involving trying to kill poor Kevin (DIE, KILL COUSIN), and even trying if the prompt recognizes Unix commands (LS, CD ..). "Gibberish" is a random string of letters or arbitrary words that don't appear anywhere in the story text (WALRUS, AWESOME). Non-keywords are words that appear in the story text but aren't keywords that you can choose (MOM, HOME).

Sticking with it to the end

About 11% of players quit without typing a single command, the same amount after the first command and a quarter kept playing until the game's conclusion.

What happens when the story tells the player that the input wasn't understood? In this case the game tells up front what kind of input it accepts. The parser can complain about the input only if the player disregards or misunderstands the instructions, makes a typo, or if the game has a bug that doesn't parse valid input correctly. There is a bug in Starborn where commanding MAGELLAN in one room gives a frustrating "Which do you mean, Magellan or the Magellan?" disambiguation question, but other than that the keyword system is immune to implementation issues where input consistent with the game's rules is disregarded because the author didn't implement a correct response.

Player made errors like typos can affect the player's experience just as much as the game's shortcomings. If you aren't able to make progress in the game, no matter the reason, you're likely to become frustrated and quit. I took a look at what happened when the player gave invalid input at the first turn. I expected that those players would be more likely to quit immediately than those who gave a valid command, but it turned out that there was no significant difference in that regard. They were still more likely to stop playing later on before reaching the end.

The reason for this might be that people who are prone to making typos are more likely to become frustrated when the story keeps stalling. This theory is supported by the fact that people who didn't complete the story gave almost twice as much invalid commands than people who completed the story (proportionate to the total amount of input; transcripts with less than 10 moves not counted).

The first turn

The story starts with short instructions on how to play.

After pressing space the screen is cleared and the game starts.

At this point the player has four keywords to choose from. It was expected that as the first command the first keyword was picked the most often and the last keyword the second most often and the transcripts show that the assumption was correct.

What wasn't as expected was that some people (20 out of the total, that's more than one percent) typed every available keyword at the same time ("MUSIC COUSIN WEIGHTLESSNESS HALLWAY"). Other typical mistakes were typos and assuming that the keyword in "...my COUSIN's sleep" is "COUSIN'S" including the apostrophe and the "s". Presumably out of habit the players would also try the usual IF commands (X ME, TURN MUSIC ON).

Category "other" includes, among others, choosing words from the text that weren't keywords, swearing, and trying to communicate with the parser (HI, OK, LOL).

Leading the player

I assumed that if choosing a keyword leads to a description that contains exactly one new keyword, the player is likely to pick that keyword instead of any of the previous yet unexplored ones. For example, if the player chooses the keyword MUSIC in the intro, the description will be:

The transcripts reveal that the most common command was indeed MAGELLAN, although I expected the figure to be much higher.

By default the story shows available keywords in the story text in upper case, so about one sixth of the players typed them in upper case as well.

("Other" includes empty input, numbers, and other input where there are no characters.)

As the game never instructs the player that they can type only one of the words in a two-word keyword, it wasn't surprising that the players were generally inclined to type the entire name of the keyword.

Typos

How many ways can you write WEIGHTLESSNESS? At least 20 different ways:

Because you can never take into account every possible way a word could be typoed, it'd be great if the parser could just go through the available vocabulary and choose the closest match to the player's input. You could then have the parser understand the most outrageous misspellings the players could come up with. This game is an exception of sorts because all available keywords (and therefore all acceptable commands) are at all times visible to the player so there's no need to be wary of the parser choosing a word that the player hasn't encountered yet, but even in a traditional IF game you could have the parser go through only the words that have been previously shown to the player.

Further steps

Just by fixing the 30 most common unrecognized commands fixes one third of all unrecognized input. The cases where the player types more than one keyword at a time or tries to use a standard IF command can be handled with a better error message, something like "Please type keywords only, one at a time. Currently available keywords are ..."

Analyzing the transcripts is great for finding out which unimplemented commands need attention or what synonyms to add, but it's by no means a replacement for traditional testing: the transcripts don't tell you how good the story is, do the puzzles work, is the pacing correct, are there spelling errors in the text... The benefit of data collection from a live release is that the data represents the actual audience. The beta testers are probably on average more experienced players than people playing the game after its release.

It'd be great to have similar volume of data from a traditional parser IF game so that the results would be more generally applicable. Even a couple of transcripts from a lot of different games would probably show interesting results.

(Updates: follow-up after an update; the recording plugin for Parchment released)