LLM Agents Play Pokemon Showdown

Researchers Oliver Kitching and Oliver Grabner have created a text-based Pokémon battle simulator using Large Language Models (LLMs), specifically a transformer-based architecture, to simulate battles between Pokémon. The model was trained on a dataset of 1.3 million battles from the Pokémon Showdown! website and achieved a 50% win rate against a human player. The model's performance was evaluated using metrics such as accuracy, precision, and recall, and it outperformed a baseline model that was trained on a smaller dataset. The researchers found that the LLM's ability to generate moves and predict opponent moves was key to its success, and that its performance improved significantly when it was given more training data. The study suggests that LLMs have the potential to be used for a wide range of applications beyond text generation, including strategy games and decision-making systems.