Is the World Ready for Deep Network AI Opponents?

Is the World Ready for Deep Network AI Opponents?

Could we use deep networks to revolutionize videogame AI? It's a tough sell.

Read Full Article

Yeah it certainly seems like a long, long way to go just to get decent AI. I think they can still be marginally improved by conventional means, but if people want more difficulty and immersion there's better ways to do that (personally wounds that self-heal after a few seconds tends to be more immersion-breaking to me than clunky AI).

I know Wargaming has made some strides in creating bots that emulate actual player behavior almost too well. When they started to populate World of Warplanes with bots due to low player numbers, some of the smarter bots were pretty impressive. I suppose one way to do it would be to mine player behavior in the game (particularly multiplayer versions) and try to get AI to emulate it, adapting to different situations.

Makes me dread the AI in MOBAs... The day I see a bot type "Report mid for feeding!", I'll know the robot apocalypse has started.
Yes, I know the joke doesn't really work since this kind of tech would mostly make sense with turn-based games (though maybe you could outsource some behaviors that affect the game on a strategic level, like preferring certain objectives above others depending on the situation?), still an interesting idea and I bet we'll see it done sooner or later, if only as one more entry on the ever expanding list of excuses for always online singleplayer games. Hell, if it actually works well it might make for a, dare I say it, legitimate reason!

Shamus Young:
for an audience that might not want, need, or even notice the feature...

That is the key here. Unless marketing does a very big deal out of the feature, and it works flawlessly, most people will not care about it, because a state machine with some fuzzy logic (a method with relatively little computational cost, and that we have used for decades in games AI now) gets the job done most of the time for most people.

With the disclaimer that I'm not an AI programmer either (though I took a few classes in college on the topic); and while I agree with the conclusion that neural network multiplayer game AI is not very feasible, I do think Shamus is wrong on a number of counts, here. The thing he identifies as the problem (needing a network connection) isn't actually a problem and the thing that's actually the problem isn't mentioned.

How Neural Networks Work: (grossly oversimplified and probably mostly wrong)

A neural network is fundamentally just a trained program that takes input and produces output. The term "network" here is probably confusing - it's a program that simulates a network of neurons, but it's not a "network" in the sense of involving multiple machines connected together - so I'm going to avoid it, and just call it a "program", because it essentially is one; it's just a program that was "written" by a different program, not a human.

You start with essentially a random, but fairly simple program with a lot of parameters that can be tweaked (a lot of knobs that can be adjusted to make the program run differently). You run the program with some input, and if it gets a good result, you leave the program alone, if you get a bad result, you fiddle with the knobs in a way that should make the result better. You do this a lot a lot of times, and eventually you get a program that gets good results often (hopefully).

But the program isn't any more complex than it started out; it just has different parameters. So you can easily ship this trained on a user's modest machine and run it, and it doesn't need any sort of network connection to run. So, assuming you could train a neural network to play Civ, then it'd be runnable on the users machine without needing to "phone home".

The Problem:

The problem, though, is the training. You can only train a program if you can algorithmically determine if the result of a given run of the program was good or bad. That's trivial for things like identifying hats in photographs. You collect a dataset of photographs that you know either have a hat or don't, and then give those to the program to see if it gets it right.

This is possible for some single-player games. The original question asker is probably thinking of something like MarI/O (which does a good job explaining neural networks) when they mention AI training to play video games. That AI is trained by using how far (and how fast) Mario gets to the right as the metric for evaluating how good the program did.

A multiplayer game is much harder. You can't just feed the program a position in Civ and check if it produces the "right"?set of moves, because there's no "correct" outcome, and it's a rather tricky subjective problem to determine whether a given move in a given position is helpful for winning or not. And games that aren't turn based are going to be even worse.

You could, however, train a neural network AI to imitate how humans play the game. That turns the problem from something subjective ("is this move good?") to something objective ("is this the move that the player actually made?"). But, to do that, you need do need a lot of historical data. So it's a chicken-egg problem: you need an AI to ship the game, but you need data from people playing the game to write an AI for it.

You could write a traditional AI, ship the game, collect data, train an AI against that data, then patch the trained AI into the game, but that's a ton of effort for not a lot of benefit: if Civ V's AI suddenly got better 6 months after launch, would that really bring in enough revenue to justify the expenditure?

Multiple Difficulty Levels:

You could, with enough data, overcome the difficulty thing that Shamus mentions. Instead of feeding all the data to train a single program, you train multiple programs by giving different parts of the data to each program. If you want to make a good AI train it against the moves that cause a player to win the game, to make a human-like bad AI, train it with the moves that cause the player to lose the game. (And some combination to make various levels in-between. Or use an ELO system and train it that way)

Other issues:

Another downside is that neural network AI are pretty fragile if the problem changes. If you initially train your network to identify bandannas as hats, and you change your mind and decide that they aren't hats, you probably have to retrain your network from scratch. A significant game patch (much less an expansion pack) could completely break your AI, unlike traditional AIs which are much easier to tweak.

And you would need the game to "phone home" to collect historical data; some players won't like the game gathering info (though I'd guess many of them just aren't telling you that they do it already) but unlike Shamus's scenario, the game won't stop working if it can't phone home, since the data collection isn't a core part of running the game.

You could also lead this back to the DRM argument if the company decides to drop support for the game. If the AI is unable to call home does it revert to a fumble footed moron that can't fight it's way out of a wet paper bag?

For strategy games like chess, go, Othello, Civilization, etc, there are two aspects to the traditional AI:

1. the depth of the move tree that can be explored.

2. the analysis of the resulting board.

Effectively, the game checks every move it can make and tries out different options and builds a tree of options. Once it maxes out the size of the tree, it has an algorithm that rates the value of the resulting board. The board with the highest value (or set of resultant boards for more interesting algorithms) is used to select the next move. Machine Learning algorithms would not actually have to be contacted that often. Perhaps once a week, a game could report in, provide a history, and get the current board-analysis algorithm. The machine learning could be used to improve board analysis itself. This means you don't need to check in that often, just often enough to get the regular updates and provide relevant data back. What this would mean is that the most played games would become the most challenging. The only real problem here is difficulty levels. It might be that you have traditional algorithms and then you have the "evolving difficulty" level.


Note that the issue that Shamus talks about specifically is deep networks that are too big to be used on a user's machine. Which is possible. You yourself note that the neural net is a set of parameters. It should be possible to have a set of parameters too big to reliably run on a user's machine, that can still be run cost-effectively on servers. computers are weird sometimes. But I'm betting that wouldn't be likely. It's usually not big enough for that.

As noted, still really hard to train the computer, at least via neural net. Doubt it would make enough sense to use that strategy though, at least for most games.

A big part of the issue with making the so called "Amazing A.I." is that often advertised is that the programmers often find that they ultimately have to make that A.I. much more stupid than they wanted to in order to get it to work in a fashion fitting a game. For instance, Bethesda tried to actually make effective Radial A.I. for the NPCs as they had promised they would when they made Oblivion, only to find out when they tested it that allowing the A.I. to properly react to the world and to each other in such a manner quickly resulted in the NPCs ended up quickly just running around slaughtering each other instead of actually acting believably and in a way that would facilitate gameplay. Trying to make an adaptive A.I. as Shamus proposes would run into the same sort of problem eventually, it would end up doing weird things that didn't lend itself to making a good challenge, such as deciding to run away from the player in such a way that would prevent them from being able to ever actually fight them and thus "winning", or just creating a massive defensive position and staying there to make the player come to them instead of actually trying to defeat the player directly.

As much as I would like the idea, the inherent problem with A.I. is that changes to it can have very unpredictable effects that might not work that well at actually making the game enjoyable, therefore they have to extensively test said A.I. in a wide variety of situations in order to make them work the way they should and thus take a long time to see any real change. That, and it's just easier to merely load the A.I. up with the same sort of parameters that have been shown to work before rather than do any real overhauling to make it better.

Whilst very few games nowadays even try to do AI well, we are not really regressing much IMHO.

The high water marks that are S.T.A.L.K.E.R. (overall single player character AI) and FEAR (combat AI) are not approached, true, but most recent games at least do not regress too much.

One might say they are as smart and as reactive as enemies from RTCW, except they dont bump into each-other as much. Which is still decent... I guess...

It certainly sounds like a niche affair. Most game AIs are made dumb enough so players can beat them or, in case of allies, surpass them; but not too dumb so it doesn't feel patronizing or badly programmed.

With the disclaimer that I'm not an AI programmer either (though I took a few classes in college on the topic); and while I agree with the conclusion that neural network multiplayer game AI is not very feasible, I do think Shamus is wrong on a number of counts, here. The thing he identifies as the problem (needing a network connection) isn't actually a problem and the thing that's actually the problem isn't mentioned.

You state quite clearly you don't know what you are talking about then ramble on about how Shamus is ... what exactly? More wrong than you and your admitted less than knowledgeable spewing? Write a blog, save us the bandwidth.

Consider the solution for the Xbox driving game that uses 'ghosts' of your friends to race against (Forsa?). IF that could be a basis (in thought only) then a solution might be possible.

I suspect we are confusing trying to find a hat with the quite different goal of stomping a player's dreams of conquest. The rules are smaller and the problem set is somewhat more defined than the training an AI to find cats on the web. A real AI programmer would be helpful here. I have had little interest in the field myself but I can recognize the misdirection 'cups vs hats' thoughts might be creating. That is really not the problem, nor is it even in the realm of the problem.

On the subject of current in game AI Rage had made a significant (now dated) advance in enemy AI. I have been playing FO4 and the changes there are enjoyable. The enemy uses cover much better than the previous two. There doesn't seem to be any squad based awareness, considering the one on one normal encounter that is not surprising.

Good talk Shamus. Not sure if I would want a game with that level of AI anyway but I am definitely against requiring a full time connection to play single player. So, even if it was good they wouldn't get my money.

The hate against always-on could perhaps be mitigated by having "regular grade" AI shipped with the game for the singleplayer mode, and offering the gloves-off full-powered beast of an AI as a multiplayer opponent - then if you need to you you can rate-limit how many people can play against it concurrently without gutting everyone else's game, and you avoid having to deliberately gimp it to make the singleplayer campaign feasible for weaker players.


Reply to Thread

Posting on this forum is disabled.