Escapist Editorials

Escapist Editorials
Putting Gamers to Work: DARPA's Grand Design

Joshua Vanderwall | 17 Jan 2014 15:00
Escapist Editorials - RSS 2.0

How did you end up working with DARPA for StormBound?

My background is in entertainment games, mostly in the design field, on some AAA titles and big licenses. At our last studio, my partners and I had worked with DARPA on another video game project, and we knew that working with them offered the chance to tackle really hard, satisfying problems. I am sure our experience in 'traditional' video games helped get us onto the project.

We heard about a CSFV (Crowd-Sourced Formal Verification, the DARPA program name) Proposer's Day that took place in the SF Bay Area, where we are located. When my partner, Mark Day, described what the project was all about, honestly my first response was to go look up Formal Verification. I have a degree in CS, so I had a vague idea in mind, but trying to fit that together with 'video game' was making my head explode. I happened to be reading a book at the time with a similar concept, and I thought "I have GOT to check this out."

I quickly put together a one-slide presentation and got up in front of a room full of really scary-smart people to pitch them on working with our game studio. My goal was just to get through the day without saying something completely stupid.

After that meeting, we had several people reach out to us to explore partnering. We ended up working with Portland, OR-based Galois, Inc., which has been an absolute dream. They are leaders in the verification area, just incredibly bright folks. As a team, we really see eye-to-eye on the product, and there's been a great synergy between the companies, which I think has resulted in a project that is more than the sum of its parts.

Was the idea to crowdsource complex problems with a browser-based puzzle game yours, or did you get the task of making StormBound?

The idea to crowdsource the problem was all DARPA, and our answer to that challenge was StormBound. The project's parameters were set by DARPA's goals, and we did our best to provide a game that would meet those needs. It is been an incredible design challenge, to come up with a game that accomplishes 'real work', and we're interested to see where it goes.

If these kinds of problems can be crowdsourced, the obvious next question is, what else might we solve this way? Could neural networks be improved more quickly by re-using the 'train your Gola' idea? The concept was chosen very carefully and intentionally, and it could be extensible.
Frankly, I think we are seeing the birth of a whole new industry, or a huge new addition to the game industry. First came gaming as entertainment, then there was edutainment, then serious games that try to train or raise awareness or improve social consciousness and then gamification - the sugar-coating of boring tasks to make them more palatable. StormBound is on the leading edge of that vector. It is a completely new type of product - a game that serves a functional purpose. It is going to need a new name; I've been using "utilitainment."

Are the levels dynamic, or are the puzzles always the same on any given level?

Actually, yes, the levels are dynamic, and that's one of the most fascinating pieces of the system to me.

When you solve a level, it gets passed up to the server, which does incredibly hard, PhD-level math on it, and gives you back a result. Either your solution to the level was just what we needed to do some verification work, or it wasn't. If it wasn't, the server changes the level, and challenges the player back with "Ah ha - but what about this?" The player is notified that their solution wasn't quite right - in the game, that's called a Fuse Breach. The players then get to go back into the level and refine their answer.

In a way, your relationship with a level is almost more like in an RTS. You don't just solve it and move on, like in most puzzle games. Instead, your solution to the level is 'active' until the server gives you the thumbs up or down. For a while we actually toyed with a scoring system that took this into account - the more answers you had, and the longer they stayed 'alive' or valid, the more points you'd get. It was almost like mining in an RTS. In the end, though, we couldn't get the balancing and messaging right in the time we had, but it's something we might explore again.

When you look at it from the 10,000 foot view, that's a really mind-blowing thing - this is the first puzzle game with a "Game Master." The server is in a dialog with the player to literally change the game based on the player's actions. That is really an incredible concept, especially when you consider that the result of that dialog is to perform formal verification, to serve a real-world need that will benefit people. That just blows my mind!

Is there a certain point in game you have to get to before you're actually contributing to the testing process, or do low level puzzles offer some benefit as well?

Right now, yes, you need to finish the World Map tutorial before we give you any live data. One of our goals is to lower that bar so you can contribute from very early on in the process.

The problem is that it's a catch-22 - this isn't like a Sudoku or crossword, because we don't know the answer to a level before we give it to you. That's why we need you to play it! So what we do instead is look at some of the properties of the level, and give you only levels you have at least a chance of solving.

For example, if you haven't done the Etheric Basics tutorial, you must skip levels that have Akashic Streams. It would be impossible for you to solve them, because you can't even get into Etheric mode. Beyond that, though, we need you to at least be proficient with the three Basic Akashic patterns (Stack, Above, Tangle), and Compound patterns. Once you can do those three, and can combine them, you can solve any level with at least one Akashic Stream, so we start giving you real levels right away.


How long did development take, and what portion of that was dedicated to figuring out how to actually use player input to solve real problems?

We have been working on the game for about 18 months now, from initial concept. A lot of that time was building the bridge between gaming and verification. There were a lot of in-person meetings for us game folks to understand exactly what the verification team needed back from the game, and to help the verification team understand some basics of game design.

There were definitely a few false starts, which is par for the course on a project like this - that's what makes it a capital-R, DARPA-hard Research problem.

We finished building the first version of that bridge about three months into the project. We actually called it our Golden Spike milestone, like when they connected the first Transcontinental Railroad. It was the first time a player could load a level that came straight out of the verification server, solve it through gameplay and send the answer back. It is one of the real successes of the project that we got something up and running very quickly, as that gave us a long time to iterate. We have probably built and thrown away three or four game concepts along the way, that's how challenging it's been to find something that will work - and be at least somewhat entertaining.

In whatever detail you can provide, how exactly does solving a puzzle in StormBound assist in testing software?

If you want a fantastic, detailed description, I recommend Jef Bell (Galois)'s excellent blog post, here.

What motivated the story part of StormBound? Has the player reaction to the story or the gameplay been more pronounced?

We were trying to attract a particular kind of audience that enjoys hard puzzles, and made a decision early on to create something that would hold players' attention. We wanted to give people something cool to sink their teeth into, that they'd want to stick around and watch unfold. We are really trying to build a long-term relationship with the audience.

To create the story and the world, I reached out to a long-time friend, Christian Gossett, comic book artist, author and award-winning director. He has created entire universes before, so I knew he would help open the window into this new place we were creating. We are just getting started with the story - we actually have a long-term story arc document that we are hoping to use over the coming months and years to give players a much richer view into the StormBound world.

Player reaction has been an interesting mix - some players love that there are these cool new concepts to explore. Others have found that it makes learning the game a little harder for them, that some of the names for things were a bit too much. So we're going to work on making it a little easier to swallow, especially at the start, because we think the magepunk theme has a lot to offer, and will give players a real sense of place and community within a very cool world.

What kind of player base do you need to achieve in order to really make this project work?

It is still too early to tell - this is just our very first release, and we will be making improvements as we iterate on things. We are still learning the best way for players to play in terms of accomplishing project goals. I will say that so far we're actually way ahead of where we thought we would be, and it does not need to be a top-10 social game to be effective.

Where are you now with your player numbers, if you can share that information?

We can't share specifics, but I will say we got a fantastic turnout at launch, which I am very happy with, and we are getting way more interest than we had forecast. I would say we're on par with a very early-stage beta online game, which is really what we are. Right now we're in a mode of engaging with our players to hear what they do and don't like, and making the game better for them. We will be working on growing the audience over the coming months, by offering a better game than we have now, with the added benefit of helping solve a real problem.

Comments on