UNDER ACTIVE DEVELOPMENT
PyAge2 allows to interact with "Age of Empires II: The Conquerors" game from OpenAI Gym compatible Reinforcement Learning environment.
The environment runs the game process, injects the DLL that runs a server allowing external tools to interact with it programmatically. The system is capable of simulating games at incredibly high speed (20-30 minutes game could be finished in seconds).
WIP. Additional goal of the project is to have a tool to deploy learned agent policies directly into the game process, bypassing the need to have a dedicated server & client to run the agent. With the help of this, the agent could be release as a publically available game Mod.
IMPORTANT
The API used to manage the game process is not officially supported by game developers. As the system overwrites game process memory directly, it could lead to game crash. Usage of the environment on live games (against other human players) will lead to permanent ban both on Steam and Voobly. By running the environment, the user implicitly acknowledges the risk, and consents on using the system strickly for research purposes.
"Age of Empires II" provides a way to script AI bots using built-in lisp-like scripting langauge. Each bot, effectively, is a set of rules (conditions and actions):
(defrule
(can-train villager)
=>
(train villager)
)Built-in AI agents contains 10k+ rules. To get a high level understanding of how typical bot looks inside and what capabilities are available, check out Steam guide on "AI Scripting". In the game installation folder, you can find Ai directory with source code for bots available in the game.
PyAge2 allows you to build AI bot in Python using OpenGym-like API with a step function takes observations from the environment and returns actions that needs to be performed. For example
class VillagersOnlyAgent(BaseAgent):
"""I bet you not gonna win with this..."""
def step(self, obs):
if obs.observation['civilian_population'] < 130 \
and expert.can_train(obs.observation, ObjectType.VILLAGER):
return actions.train(ObjectType.VILLAGER)Check out pyage2.agents.ScriptedAgent as an example of a primitive but fully-functional agent.
The system allows you to match different AI bots/agents in a single game.
Compared to other learning environment based on games (e.g. Atari, VizDoom, PySC2, MicroRTS), "Age of Empires II" presents interesting challenges for RL/ML research community.
RTS (Real Time Strategy) games are known for their complexity with respect to action planning. As it turned out, micromanagement (per-unit control) was easier to tackle, when macromanagement remains largely unsolved problem. "Age of Empires II" is a game where success is mainly determined by macromanagement (economy, build order, research, units production). Basic micromanagement tasks are scripted by the game engine, which means you don't need to worry about which villager should build a house and which unit should go first into the attack first.
Another challenge with macromanagement level control is delayed reward signal. For example, setting workers allocation for resources in the beginning on the game will have large impact only on the late stages of the game.
The game allows up to 8 independent agents on a single map with flexible team formations (e.g. each agent on it's own team, or 4x4, or 2x2x2x2, etc). Agents do not have access to each other states. The coordination could be done only using in-game Chat.
"Age of Empires II" comes with loads of different civilizations, modes, maps, and custom scenarious. Default option would be to learn a new agent for each formation, which leads to combinatorical explosion. Ideally, a good agent should "understand" core game mechanics and strategies, while being flexible to utilize different tech trees, available units, flexible map conditions, etc.
An agent could be deployed into the environment using scripting language in a form of a decision tree (more formally, set of actions and corresponding activation conditions). This approach is not common in RL comminuty but it has tremendous practical advantage: explainability. This makes it possible for game designers and game engineers to understand and interact with internals, which might lead to much better story for game teams when incorporating learned AIs to improve game experience for players (e.g. by introducing "harder" levels of AI by incorporating RL-based improvements, or automatically adjusting agents to perform custom scenarious instead of scripting them each time manually).
Please use this bibtex if you want to cite this repository in your publications:
@misc{pyage2,
author = {Oleksii Kachaiev},
title = {{PyAge2 - "Age of Empires II" Learning Environment}},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/kachayev/pyage2}},
}
Important. Running PyAge2 requires 32-bit Python3.7+, you can download 32-bit installer from python.org.
The faster and the easiest way to install PyAge2 with pip:
$ pip install pyage2Alternatively,
$ pip install --upgrade https://github.com/kachayev/pyage2/archive/main.zipor from a local clone:
$ git clone https://github.com/kachayev/pyage2
$ pip install --upgrade pyage2/PyAge2 depends on the "Age of Empires II: The Conquerors" game and (as of now) does not support DE versions. Which means, it won't work on Steam's versions of the game. Though, there's a tool that can help you to convert your AOE2HD to original AOC engine format: AoE2Tool (please, follow the instructions from the repo on how to perform update). The environment was tested on most common game versions: UserPatch 1.5, UserPatch 1.6, and WololoKingdoms expansion.
PyAge2 expects default installation pass to be C:\Users\<USER>\AppData\Roaming\Microsoft Games\Age of Empires ii\Age2_x1\age2_x1.5.exe. In case you modified install path, please specify PYAGE2PATH environment variable.
PyAge2 heavily relies on Win32 API, so far it's unclear if it's possible to run same hacks on Linux on MacOS. But both on my TODO list.
The environment works with all built-in maps, scenarious, and AIs. And with pretty much all custom modes, e.g. here (follow the instruction for a specific mod).
To get the list of all available AI bots:
$ python -m pyage2.bin.bots_listTo get the list of all available maps:
$ python -m pyage2.bin.map_listTo get the list of all available civilizations:
$ python -m pyage2.bin.civilization_listLet's run a simple game between AI bots:
$ python -m pyage2.bin.play --agent1 Barbarian --agent2 Illuminati --run-full-speedor between AI bot and scripted agent:
$ python -m pyage2.bin.play --agent1 pyage2.agents.ScriptedAgent --agent2 Illuminati --run-full-speedYou can specify up to 8 agents on a map flexibily combining them into teams. The enviroment differenciate agents based on . in the agent name (names with dots recognized as a Python module and class name that will be used to instantiate an agent). Special name "HUMAN" is reserved to identify human player. There's only a single human player allowed in the game.
Additional game configuration options include map type, map size, starting age, starting resources, victory type, and more. Use --help to get information about all flags.
A replay lets you review what happened during the game. To run specific replay with a game client, use
$ python -m pyage2.bin.replay <path-to-replay>openage is an open-source reimplementation of AOC engine that uses game assets but provides custom engine to run the game. It runs on all platforms, and exposes Python API for scripting. It should be possible to make the environment works on this engine, though I didn't have a chance to try to do so (as of now). Help wanted!
The environments uses the following DLLs:
- Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
- Fork the repository on Github & fork master to
feature-*branch to start making your changes. - Write a test which shows that the bug was fixed or that the feature works as expected.