The content is essentially the same. Instead of typing the keywords you click on hyperlinks that are highlighted in the text and shown in a separate list next to the story. A clickable, dynamic map of locations is displayed on the opposite side. There’s background music, but sound support is still a bit shaky in some browsers. Internet Explorer 7 or earlier will not work.

It uses the yet-unpublished version 1.2 of Vorple and demonstrates the use of the button interface in the map and in the keyword list, and tooltips that are displayed as brief instructions and as labels for the map. Under the hood it uses disposable links and other similar features. It’s also using the IF Recorder plugin, probably the first time for an Undum story.

• Play online
• Source code

Here are the major new features:

• Works with Undum. Now someone might wonder why anyone would want to record hypertext fiction stories, but even if there’s less need to check what kind of input readers give, other reasons still apply: you might want to get statistical data on which choices the readers make, or see if the readers give up at some certain point in the story, or even find out if some of the choices just don’t work as they should.
• Web interpreter template for Inform 7. There’s a ready-made Inform 7 template to use with the “Release along with an interpreter” option. This makes it easier to start using the recorder although you still have to set up the database and the server scripts.

The project’s new headquarters are at Github along with the instructions and downloads.

• Play the game here
• Annotated source code: zipped package or the JavaScript file and the HTML page (“View source” in your browser)

The plugin saves more than just plain text: it preserves text formatting and saves status lines as well, so the transcripts look (almost) the same as the actual game. (See this example of Bronze with status lines, colors and text formatting. Note that Parchment uses small caps in place of bold text.) The saved transcripts can be viewed with or without the status lines. In addition when the game waits for a single keypress, non-alphabet characters like enter, space and arrow keys are marked in the transcript.

There’s a live demo with a transcript viewer available. Games played in the demo installation can be seen in the viewer.

The plugin and accompanying tools can be downloaded from Google Code where you’ll also find installation instructions. Only Z-machine is supported at the moment, Glulx support will be added later.

Looking at the data from before and after the update, here’s what happened to the total amount of invalid commands:

In addition to new synonyms the update includes a new class of error message. The earlier version responded with “That is not a keyword the story recognizes” to all invalid input. The update shows the same message when the player gives only one word as input, but if the player had given more than one word the error message is ”Please type a single keyword to advance the story” followed by a list of available keywords.

This was inspired by the different ways people had given invalid input: some gave traditional IF commands, some typed more than one keyword at the same time. In addition to the new error message giving more accurate instructions on how to interact with the story, it also explicitly lists the input it accepts, leaving no room for confusion. I looked at how this affected “ragequitting” after giving invalid multiword input during the first turn:

(The sample set is so small that the results fit well within error margins, but they are still nice numbers to look at.)

In addition to reducing the total number of invalid input, the amount of error messages per game dropped as well. In the previous version about half of the players didn’t encounter any error messages while playing, but more than a quarter bumped into them more than once, 10 percent more than three times. In the latest release the no errors group jumped up to two thirds and those who saw more than one error message dropped to less than 10 percent.

It was expected that by making the parser more error tolerant and helpful the percentage of people who played the story to completion would go up, as well as time spent with the game, measured here with the average number of turns per game (not counting those who quit without typing a single command).

As the graphs show, the percentage of people who quit after one turn or right after the intro doesn’t show any significant change. There are people who don’t find this kind of medium their thing and there’s not much that can be done to make them keep playing. in the other end of the scale are the hardcore IF fans who play anything they can get their hands on. Most players fall somewhere in between and to keep them engaged requires both a good story and smooth gameplay.

Betatesting can do a lot, but analysis of real life transcripts can reveal surprising things that have a significant effect on how players enjoy the game. Who would’ve thought that invalid one-word and multiword input should have different error messages? What would a similar stack of transcripts from games with full parsers reveal?

The good news is that I’ve almost finished with making the transcript recording functionality an actual Parchment plugin. It’ll be released soon, first with Z-machine support and later with Glulx support, and even later with statistical tools to make all kinds of graphs like the ones presented here.

Last Wednesday however C.E.J. Pacian wrote a blog post about it which was picked up by the IndieGames.com blog. From there a couple of people tweeted about it and suddenly the play count jumped to about 600 plays per day. The main Parchment site reported similar amounts of traffic.

I’m running the online version of the story on a modified version of Parchment that saves the transcripts to the server. I now have 1557 transcripts consisting of 23 896 turns (and counting, but the traffic is slowing down considerably). To my knowledge this is more play data collected than for any other single IF game so far.

So here’s a random bunch of analysis from those transcripts. There are no spoilers other than keywords used throughout the game, but if you haven’t played yet the data would probably make more sense if you at least checked out a couple of turns just to see what the parser looks like. Online version and downloads are available.

Parser errors

Since the gameplay is just typing single keywords given by the story, the amount of understood input was substantially high (93%). Most of the errors were typos (53.3% of all invalid input).

Standard IF commands were disabled so they count as unrecognized commands. The “sign of confusion” category contains commands like HUH, ?, WHERE, or verb commands that don’t follow the standard IF convention (I GO OUT). A “poke” is the player testing the limits of the parser by trying to communicate with it (HELLO, BYE, OH NO) or trying to do things out of character, mostly involving trying to kill poor Kevin (DIE, KILL COUSIN), and even trying if the prompt recognizes Unix commands (LS, CD ..). “Gibberish” is a random string of letters or arbitrary words that don’t appear anywhere in the story text (WALRUS, AWESOME). Non-keywords are words that appear in the story text but aren’t keywords that you can choose (MOM, HOME).

Sticking with it to the end

About 11% of players quit without typing a single command, the same amount after the first command and a quarter kept playing until the game’s conclusion.

What happens when the story tells the player that the input wasn’t understood? In this case the game tells up front what kind of input it accepts. The parser can complain about the input only if the player disregards or misunderstands the instructions, makes a typo, or if the game has a bug that doesn’t parse valid input correctly. There is a bug in Starborn where commanding MAGELLAN in one room gives a frustrating “Which do you mean, Magellan or the Magellan?” disambiguation question, but other than that the keyword system is immune to implementation issues where input consistent with the game’s rules is disregarded because the author didn’t implement a correct response.

Player made errors like typos can affect the player’s experience just as much as the game’s shortcomings. If you aren’t able to make progress in the game, no matter the reason, you’re likely to become frustrated and quit. I took a look at what happened when the player gave invalid input at the first turn. I expected that those players would be more likely to quit immediately than those who gave a valid command, but it turned out that there was no significant difference in that regard. They were still more likely to stop playing later on before reaching the end.

The reason for this might be that people who are prone to making typos are more likely to become frustrated when the story keeps stalling. This theory is supported by the fact that people who didn’t complete the story gave almost twice as much invalid commands than people who completed the story (proportionate to the total amount of input; transcripts with less than 10 moves not counted).

The first turn

The story starts with short instructions on how to play.

After pressing space the screen is cleared and the game starts.

At this point the player has four keywords to choose from. It was expected that as the first command the first keyword was picked the most often and the last keyword the second most often and the transcripts show that the assumption was correct.

What wasn’t as expected was that some people (20 out of the total, that’s more than one percent) typed every available keyword at the same time (“MUSIC COUSIN WEIGHTLESSNESS HALLWAY”). Other typical mistakes were typos and assuming that the keyword in “…my COUSIN’s sleep” is “COUSIN’S” including the apostrophe and the “s”. Presumably out of habit the players would also try the usual IF commands (X ME, TURN MUSIC ON).

Category “other” includes, among others, choosing words from the text that weren’t keywords, swearing, and trying to communicate with the parser (HI, OK, LOL).

Leading the player

I assumed that if choosing a keyword leads to a description that contains exactly one new keyword, the player is likely to pick that keyword instead of any of the previous yet unexplored ones. For example, if the player chooses the keyword MUSIC in the intro, the description will be:

The transcripts reveal that the most common command was indeed MAGELLAN, although I expected the figure to be much higher.

By default the story shows available keywords in the story text in upper case, so about one sixth of the players typed them in upper case as well.

(“Other” includes empty input, numbers, and other input where there are no characters.)

As the game never instructs the player that they can type only one of the words in a two-word keyword, it wasn’t surprising that the players were generally inclined to type the entire name of the keyword.

Typos

How many ways can you write WEIGHTLESSNESS? At least 20 different ways:

Because you can never take into account every possible way a word could be typoed, it’d be great if the parser could just go through the available vocabulary and choose the closest match to the player’s input. You could then have the parser understand the most outrageous misspellings the players could come up with. This game is an exception of sorts because all available keywords (and therefore all acceptable commands) are at all times visible to the player so there’s no need to be wary of the parser choosing a word that the player hasn’t encountered yet, but even in a traditional IF game you could have the parser go through only the words that have been previously shown to the player.

Further steps

Just by fixing the 30 most common unrecognized commands fixes one third of all unrecognized input. The cases where the player types more than one keyword at a time or tries to use a standard IF command can be handled with a better error message, something like “Please type keywords only, one at a time. Currently available keywords are …”

Analyzing the transcripts is great for finding out which unimplemented commands need attention or what synonyms to add, but it’s by no means a replacement for traditional testing: the transcripts don’t tell you how good the story is, do the puzzles work, is the pacing correct, are there spelling errors in the text… The benefit of data collection from a live release is that the data represents the actual audience. The beta testers are probably on average more experienced players than people playing the game after its release.

It’d be great to have similar volume of data from a traditional parser IF game so that the results would be more generally applicable. Even a couple of transcripts from a lot of different games would probably show interesting results.

(Updates: follow-up after an update; the recording plugin for Parchment released)

There’s one integral, defining aspect to IF and it’s the world model. The author creates a virtual world and a story that takes place in it. So far this is only a simulation; to make it a game the author adds game mechanics and they are, more often than not, puzzles. But are puzzles the only option? Is there really nothing else we could do with a virtual world?

What about puzzleless IF, I hear you ask. There are a lot of those, right? Sure, but most of them are just puzzle games without puzzles, or with trivially easy puzzles. Removing puzzles and adding nothing in their place does not improve the situation. Without any game mechanics IF turns into hypertext fiction.

Galatea and its descendants are good examples of non-linear stories with strong non-puzzle game mechanics, in this case conversation. If I may take a look into the crystal ball, I suspect that in the future the puzzle aspects will diminish or at least they have changed considerably. Just like mazes and hunger puzzles have made room for better puzzles, the puzzles we write today will make room for even better design. More importantly puzzles will not be considered the only or even the primary option for IF game mechanics.

The key is to realize that puzzles are just one option for IF game mechanics and they are prevalent (I suspect) mostly because of tradition starting from the very first Adventure. “An adventure game is a crossword at war with a narrative,” the famous quote by Graham Nelson from 1995, has been the guideline for IF design. I’m afraid we’re stuck with conventions that have grown to define the medium and are now preventing us from seeing more options. It’s time to look farther and create something new instead of just improving what we have.

Wrong. Players would play for some while, then LOOK and go “Didn’t I pick that up already?” or INVENTORY and “* BUG: I never dropped those items”. They hadn’t noticed or didn’t remember anymore that the game had said “(first dropping X, Y and Z to make room)” when they had picked up something big earlier.

Lessons learned:

• Don’t put relevant information to places where the players don’t expect to find information. Players don’t expect command clarifications to contain anything other than notifications of implicit actions, and they expect TAKE ALL to just list items and “taken”, so they don’t pay them any more attention than that.
• Don’t expect the players to read room descriptions anymore after the first time. If a room’s description changes during the game and that change contains important information, make sure the players know they should re-read the description or convey the information to them also in some other way.
• Don’t expect the players to remember what they have read after a few turns.

It’d be interesting to hook a couple of players with different levels of playing experience into an eye tracker and study what parts they tend to skip or skim when playing.

Try it out yourself: Click here to start a game and then open this page to a new browser window. Enter a command in the game and see how the transcript in the live feed updates. (If you don’t see your game in the live feed page, reload it. It shows the three latest games being played at the time the feed page was loaded.)

And then there’s the beef: I’ve made a rudimentary statistics page that shows some interesting information using the collected data, including most used commands and average turn count and playtime.

The obvious use for this is of course collecting information on how people play the game and using that information to make improvements for the next release. You could also use this for closed beta-testing — give the web address to your testers and Parchment saves the transcripts automatically. The testers don’t have to remember to start the transcript or send it to the author.

Statistics from other games

First I want to stress that at the moment this is just a demo of a hack. It can and will behave strangely especially if the game bypasses the standard parser and takes input directly (menus etc.) That said, if you want to try other games than the sample game, you can use either http://parchment.game-testing.org/demo/parchment.full.html?story=xxx or http://parchment.game-testing.org/play/parchment.full.html?story=xxx (replace xxx with the full url of the game you want to play.) The difference is that the latter doesn’t have the live transcript feed and only has the statistics page (note a different address than what the demo site uses). You can’t single out individual players from the statistics so there’s no disclaimer notifying the player of the data collection (which might unnecessarily scare off some players.)

The statistics page collects information and displays the urls of all games played this way, so don’t use this for trying out something you don’t want the whole world to see.

How it works

I’ve added AJAX calls inside Parchment’s UI code at appropriate places where it sends the player’s command and the game’s response to the server. On the server a PHP script receives the data and saves it into the database. This is all done behind the curtain so for the player the gameplay experience is exactly the same as with unmodified Parchment.

The statistics page collects some extra information about Inform 7′s internal error message and library message codes. This requires some modifications to the game file itself. The Raising the Flag on Mount Yo Momma game used as the example has been modified to include this:

That code shows the error codes and the library message ids after every turn in the prompt, for example #LIBMSG:40#>. When the modified Parchment sees these strings it saves the information in them and removes them from the output so that the player doesn’t see them. Played on any other interpreter the game would of course display these, so the last line of code warns the player that they shouldn’t be using this game file if they can see the control codes.

Unfortunately the error codes and library message codes aren’t very reliable; some parser errors don’t seem to have an error code and some library messages don’t have an id. Most noticeably the “unknown verb” error doesn’t have either, so it doesn’t show in the statistics. I don’t actually know what the codes really do, I just noticed they’re there and used them; future development would include using custom codes for different situations in the game.

Future plans

• Test it more and fix bugs so that it could actually be reliably used.
• Clean up and comment the code so that it could be packaged for people to install on their own servers.
• Create a service where people could have saved transcripts and command statistics behind a password so that they could improve the game for later builds or use Parchment to save beta-testing transcripts.
• Add more information and charts to the statistics page (suggestions welcome.)
• Try to convince Parchment developers to include this functionality into the main build or provide hooks so that this could be added as a plugin. Right now I’m just modifying Parchment’s code directly, so whenever Parchment is updated the files have to be edited by hand again.

What would you like to see done with this?