Starborn statistics follow-up

Earlier I posted some statistics from transcripts collected from online plays of Starborn. Based on those statistics I released a new version at the end of January, mainly adding synonyms based on the most common typos players had made.

Looking at the data from before and after the update, here's what happened to the total amount of invalid commands:

Valid commands
Before update After update
Amount % of all commands Amount % of all commands
Valid commands 22208 93% 7545 98%
Invalid commands 1688 7% 146 2%
total 23896 7691

Pie graphs showing 7% of invalid commands before update and 2% after update.

In addition to new synonyms the update includes a new class of error message. The earlier version responded with "That is not a keyword the story recognizes" to all invalid input. The update shows the same message when the player gives only one word as input, but if the player had given more than one word the error message is ”Please type a single keyword to advance the story” followed by a list of available keywords.

This was inspired by the different ways people had given invalid input: some gave traditional IF commands, some typed more than one keyword at the same time. In addition to the new error message giving more accurate instructions on how to interact with the story, it also explicitly lists the input it accepts, leaving no room for confusion. I looked at how this affected "ragequitting" after giving invalid multiword input during the first turn:

Quitting after receiving an error message to input with more than two words during the first turn
"That is not a keyword the story recognizes." ”Please type a single keyword to advance the story. Available keywords: ...”
Quit immediately 10 9% 0 0%
Kept playing 96 91% 16 100%
total 106 16

Two pie charts showing how before the update 9% quit after getting an error message from two-word commands, after update no-one quit immediately.

(The sample set is so small that the results fit well within error margins, but they are still nice numbers to look at.)

In addition to reducing the total number of invalid input, the amount of error messages per game dropped as well. In the previous version about half of the players didn't encounter any error messages while playing, but more than a quarter bumped into them more than once, 10 percent more than three times. In the latest release the no errors group jumped up to two thirds and those who saw more than one error message dropped to less than 10 percent.

Number of invalid input per game
Before update After update
No errors during game 671 48% 250 69%
One error 339 24% 80 22%
Two errors 140 10% 24 7%
Three errors 93 7% 6 2%
More than three errors 144 10% 0 0%
total 1387 360

Two pie graphs, errors during game before and after update. Before update 48% of players had no errors during the story, after update 69%.

It was expected that by making the parser more error tolerant and helpful the percentage of people who played the story to completion would go up, as well as time spent with the game, measured here with the average number of turns per game (not counting those who quit without typing a single command).

Game completion
Before update After update
Loaded the story, didn’t play 170 11% 53 13%
Quit after first turn 177 11% 34 8%
Played more than one turn, not to completion 805 52% 183 44%
Played to completion 405 26% 143 35%
total 1557 413

Two pie graphs showing that 35% of players finished the game after update as opposed to 26% before the update.

Bar graphs showing an average of 17.2 turns per game before update and 21.4 turns per game after update

As the graphs show, the percentage of people who quit after one turn or right after the intro doesn't show any significant change. There are people who don't find this kind of medium their thing and there's not much that can be done to make them keep playing. in the other end of the scale are the hardcore IF fans who play anything they can get their hands on. Most players fall somewhere in between and to keep them engaged requires both a good story and smooth gameplay.

Betatesting can do a lot, but analysis of real life transcripts can reveal surprising things that have a significant effect on how players enjoy the game. Who would've thought that invalid one-word and multiword input should have different error messages? What would a similar stack of transcripts from games with full parsers reveal?

The good news is that I've almost finished with making the transcript recording functionality an actual Parchment plugin. It'll be released soon, first with Z-machine support and later with Glulx support, and even later with statistical tools to make all kinds of graphs like the ones presented here.

11 thoughts on “Starborn statistics follow-up

  1. I was thinking you might get an even higher completion rate if you display the list of valid keywords not only when typing two words but also on regular invalid keywords.

    I think it's also possible there's a friendlier phrasing than "That is not a keyword the story recognizes." It sounds kind of bureaucratic as is.

  2. Wie IF gespielt wird...

    Mit Starborn, einem der ersten Spiele 2011, hat Juhana Leinonen etwas Statistik betrieben und dafür über 1.500 Transkripte ausgewertet, die über die Online-Version des Spiels mitgeschnitten wurden. Es ist allerdings kein typisches Spiel, da es an Stell...

  3. Can this be ported to the HTML Playkit?
    I have an additional storyline idea for you as well, plus an inventory list that takes advantage of pictures in a treasure hunt type interface (can also use CYOA mini game).

    I haven't made this library nor have any intention to.
    It's just an idea for another chapter.

  4. I'm not quite sure what you mean. Are you interested in porting the story to TADS or would you like to have the transcript recording functionality in TADS?

    I have no plans for making a sequel or expanding the story at the moment, but I'm always interested in hearing about new user interface ideas.

  5. Well, my point was the SICK BAY has no description and no purpose, and HALLWAY is used multiple times.

    Spoilers for noob:

    Go to GYMNASIUM, get JACKET, take CARD from pocket, see it's a PASSPORT, go to HALLWAY, go to SICK BAY, get SUPPLIES for trip.

    I also thought his death at the end could be a near death experience for a second chapter, tying up loose ends and not letting the user feel like he lost when he couldn't really do anything else.

    Plus by the end, you feel sick by G-forces stronger then you, but if you had the supplies you could extend the storyline a little.

    Note: I might have played an older version of the game. release 2.

  6. I forgot about I had I had planned.
    It's an inventory system that works with Web links on lock icons.

    A closed lock means the item is not obtainable at this time.
    An open lock means you can get it.

    A scroll bar on the side of the interpreter tells you what's in your inventory.

    You can also loose this item depending on how many times you use it (loosing objects library already made in TADS).

  7. SICK BAY not working might be a bug in the older version. It certainly works in the most recent one.

  8. Why don't you get a register optional PM system so we can talk privately.

    I saw that library I was talking about when I was into AIF (yahoo groups), but it can be used for IF is well.

  9. I was thinking my ideas would be perfect for filfre, but it works with glutz, and I don't know if it's compatible with my OS.
    I don't even know what glutz is!
    Please enlighten me.
    The guy making it could use a few more ideas as well.

  10. Anonymous: This is both off-topic and not really my specialty so I'm going to close the comment thread now; please make a post at forums if you need assistance with your project.

Comments are closed.

Did you find this article useful? By subscribing to the blog's mailing list you will receive an email whenever a new blog post is published. The address won't be used for any other purposes.