Title: Enforcing Neuro-Symbolic Structure in Deep Reinforcement Learning

Abstract: Monolithic deep reinforcement learning trains a single network to learn vision, physics, planning, and control from reward alone. The result is poor sample efficiency, brittle generalization, and uninterpretable decisions. This thesis shows how to build domain knowledge into policy architecture and enforce these architectural priors during training.

We develop this claim at three levels of abstractions. Concept-level abstractions route predictions through human-interpretable predicates such as `door_present’ and `key_in_inventory’, enabling runtime inspection and intervention. Action-level constraints enforce state-dependent action validity, preventing unmasked training from suppressing rarely valid behaviors through shared representations. Compositional abstractions represent long-horizon tasks as reusable symbolic skills that a planner can sequence while RL grounds each skill in low-level control.

Building on these foundations, this proposal focuses on two future directions. Concept-conditioned latent action models impose semantic structure on variational motor representations, allowing high-level controllers to sample behaviors by name. A planning-guided option critic learns dynamic skill scheduling under precondition constraints, replacing static plan traversal with on-policy option selection.

Together, these contributions show that domain knowledge in the policy architecture reduces sample complexity, enables cross-task skill composition, and makes internal decisions available for inspection and override.

Thesis Committee:
Katia Sycara, CMU (Chair)
Sebastian Scherer, CMU
Yonatan Bisk, CMU
Kevin Ellis, Cornell University