Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control

Sample-efficient policy learning from pixels remains a key challenge in reinforcement learning (RL). Dynamics-based representation learning methods improve sample efficiency by learning representations through auxiliary prediction either in latent space (self-prediction) or observation space (observation prediction), but both approaches struggle under limited data. The authors argue that relying on either objective alone is insufficient: observation prediction grounds representations in observation-level dynamics but does not directly regularize temporal predictability of latent representations, while latent self-prediction may not align with observation-level dynamics.

To address this, they propose Observation-Grounded Self-Predictive Representations (OG-SPR), a model-free visual RL algorithm for continuous control. OG-SPR learns representations that are both temporally predictive in latent space and grounded in observation-level dynamics, incorporating two auxiliary objectives: multi-step latent self-prediction and next-observation prediction. A key insight is that directly imposing latent self-prediction on the shared representation can over-constrain it, so OG-SPR introduces lightweight adapters for latent self-prediction, allowing the shared representation to benefit from temporal predictive signals without being forced to satisfy the self-prediction objective directly.

Experiments on 28 visual control tasks from the DeepMind Control Suite show that OG-SPR improves aggregate performance over state-of-the-art self-predictive and observation-predictive RL methods, with particularly strong gains in challenging domains such as dog and humanoid. The results suggest that combining both predictive objectives with appropriate architectural constraints can significantly enhance sample efficiency for visual RL.