2026

1. 

   A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning

   Anish Abhijit Diwan, Jan Peters, and Oleg Arenz

   In Advances in Neural Information Processing Systems (NeurIPS), 2026

   Abs arXiv Bib Website

   Inverse Reinforcement Learning (IRL) is classically solved using Lagrangian optimisation, by jointly optimising a primal-dual problem with gradient descent to obtain the optimal reward function parameters and corresponding optimal policy. While this algorithmic view of IRL has been highly influential, gradient-based reward updates only paint a partial picture, largely ignoring the geometry of the problem in occupancy space. In this paper, we present an occupancy-space view of reward updates in IRL. We leverage the insight that both the optimal solution and its geometry in occupancy space are known, and show that the geometry along the e-geodesic (straight line in occupancy space) toward the optimal occupancy directly affects the gradient of the IRL dual problem in reward parameter space. This perspective naturally suggests preconditioning the dual gradient with the average Fisher information along the occupancy trajectory. We show that this preconditioned gradient is just a convex interpolation of the old reward and a new target reward, yielding efficient natural-gradient-style updates with little additional computational overhead. We then analyse these occupancy-space reward updates from a Mirror Descent (MD) perspective, and show that policy search over the resulting rewards induces approximate entropic MD on the policy. Finally, we conclude with convergence results of such geometry-aware IRL algorithms.

   @inproceedings{diwan2026geometric,
     title = {A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning},
     author = {Diwan, Anish Abhijit and Peters, Jan and Arenz, Oleg},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
     year = {2026},
   }

2. 

   Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

   Anish Abhijit Diwan, Davide Tateo, Christopher. E Mower, Haitham Bou-Ammar, Jan Peters, and Oleg Arenz

   In International Conference on Machine Learning (ICML), 2026

   Abs arXiv Bib Website

   Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL–one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.

   @inproceedings{diwan2026trust,
     title = {Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates},
     author = {Diwan, Anish Abhijit and Tateo, Davide and Mower, Christopher. E and Bou-Ammar, Haitham and Peters, Jan and Arenz, Oleg},
     booktitle = {International Conference on Machine Learning (ICML)},
     year = {2026},
   }

2025

1. 

   Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation

   Anish Abhijit Diwan, Julen Urain, Jens Kober, and Jan Peters

   In International Conference on Learning Representations (ICLR), 2025

   Abs arXiv Bib Website

   This paper introduces a new imitation learning framework based on energy-based generative models capable of learning complex, physics-dependent, robot motion policies through state-only expert motion trajectories. Our algorithm, called Noise-conditioned Energy-based Annealed Rewards (NEAR), constructs several perturbed versions of the expert’s motion data distribution and learns smooth, and well-defined representations of the data distribution’s energy function using denoising score matching. We propose to use these learnt energy functions as reward functions to learn imitation policies via reinforcement learning. We also present a strategy to gradually switch between the learnt energy functions, ensuring that the learnt rewards are always well-defined in the manifold of policy-generated samples. We evaluate our algorithm on complex humanoid tasks such as locomotion and martial arts and compare it with state-only adversarial imitation learning algorithms like Adversarial Motion Priors (AMP). Our framework sidesteps the optimisation challenges of adversarial imitation learning techniques and produces results comparable to AMP in several quantitative metrics across multiple imitation settings.

   @inproceedings{diwan2025noise,
     title = {Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation},
     author = {Diwan, Anish Abhijit and Urain, Julen and Kober, Jens and Peters, Jan},
     booktitle = {International Conference on Learning Representations (ICLR)},
     year = {2025},
   }