Reinforcement learning, the AI training technique that’s brought to fruition systems capable of defeating world poker champions and guiding self-driving cars, isn’t the simplest thing in the world to wrangle. That’s particularly true in the gaming domain, where cutting-edge approaches sometimes require bespoke tools that aren’t publicly available.
Fortunately, that’s changing. In a paper recently published on the preprint server Arxiv.org, researchers at Alphabet’s DeepMind describe a game-oriented reinforcement learning framework dubbed OpenSpiel. At its core, it’s a collection of environments and algorithms for research in general reinforcement learning and search and planning in games, with tools to analyze learning dynamics and other common evaluation metrics.
“The purpose of OpenSpiel is to promote general multiagent reinforcement learning across many different game types, in a similar way as general game-playing but with a heavy emphasis on learning and not in competition form,” wrote the researchers. “We hope that OpenSpiel could have a similar effect on general [reinforcement learning] in games as the Atari Learning Environment has had on single-agent [reinforcement learning].”
OpenSpiel supports over 20 single- and multi-agent game types, from cooperative and zero-sum games (in which each player’s losses and gains are exactly balanced by those of rivals) to general-sum and one-shot games (in which the stakes are high but carry no further repercussions) and sequential games. That’s in addition to strictly turn-taking games, auction games, matrix games, and simultaneous-move games, plus perfect games (where players are perfectly informed of all the events that have previously occurred when making a decision) and imperfect information games (where decisions are made simultaneously).
As the coauthors explain, OpenSpiel was built around two major design criteria: simplicity and minimalism. To this end, it uses reference implementations rather than fully optimized and high-performance code, narrowing its scope and breadth. Furthermore, it keeps dependencies to a minimum, minimizing install footprints, thus reducing the likelihood of compatibility issues down the line.
Games in OpenSpiel are written in C++ and wrapped in Python, with some custom reinforcement learning environments in Python, while algorithms are implemented in either programming language. (There’s also a subset of features available in Swift, intended to be used with Swift for Google’s TensorFlow machine learning framework.) DeepMind notes that the API is almost identical in the two languages, allowing the code to be easily translated if needed.
The DeepMind researchers note that OpenSpiel has only been tested on Linux (Debian 10 and Ubuntu 19.04), but they say that, because it uses freely available tools that are also available on MacOS and Windows, it should compile and run without issue on those platforms. As of today, it’s freely available on GitHub.
OpenSpiel targets games, but it’s not the first of its kind to hit GitHub. Last August, Google released an open source reinforcement framework based on TensorFlow, its machine learning library, ahead of the launch of Facebook’s Horizon reinforcement learning platform. Separately, Intel offers a collection of reinforcement learning tools in RL Coach, the latest version of which was released in July.