Abstract:
Reinforcement learning agents in complex tasks often require extensive exploration of large state spaces before useful structure emerges. Instead of pure exploration for learning tasks, it is possible to leverage high-level semantic knowledge such as recipes, instructions, or labels, and adapt to new tasks by grounding that prior knowledge in the environment. This talk presents Knowledge Graph-Augmented Reinforcement Learning (KG-RL), a method that augments RL policies with structured graph knowledge and injects it into a variety of policy networks across a variety of environments.

The method takes the form of a plug-in adapter. A task knowledge graph is merged each step with a scene graph, encoded by a Graph Convolutional Recurrent Network, and pooled through a small recommender into a fixed-width feature vector concatenated with the policy backbone’s observation features. The backbone itself is untouched, so the adapter slots into learned policies as varied as CNN-MLP, SoftMoE-LSTM, GTrXL, and PoliFormer without modification; bringing it to a new environment requires only enumerating a handful of entities and relation templates. The knowledge graphs are instantiated against simulator-exposed tables in symbolic environments, or mined by a one-time pre-pass of the same perception pipeline that builds per-step scene graphs in open-world scenarios.

We evaluate the adapter on four environments (Overcooked-AI, MiniGrid, Craftax, and AI-Habitat ObjectNav) and four backbones. It delivers two improvements independent of backbone choice: faster training to the same final policy, reaching the same asymptotic reward in up to 3x fewer environment steps, and a higher final policy under a fixed budget, where on Craftax and AI-Habitat the method overtakes the available published baselines. Both gains scale with task complexity, and the adapter is robust to substantial knowledge-graph corruption, retaining its advantage with half of the graph’s nodes removed. These results argue that structured task knowledge belongs as a default, low-cost input channel in modern reinforcement learning.

Committee:
Katia Sycara (advisor)
Changliu Liu
Renos Zabounidis