/Safety-Aware Algorithms for Adversarial Contextual Bandit

Safety-Aware Algorithms for Adversarial Contextual Bandit

Wen Sun, Debadeepta Dey and Ashish Kapoor
Conference Paper, Thirty-fourth International Conference on Machine Learning (ICML 2017), August, 2017

Download Publication (PDF)

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


In this work we study the safe sequential decision making problem under the setting of adversarial contextual bandits with sequential risk constraints. At each round, nature prepares a context, a cost for each arm, and additionally a risk for each arm. The learner leverages the context to pull an arm and receives the corresponding cost and risk associated with the pulled arm. In addition to minimizing the cumulative cost, for safety purposes, the learner needs to make safe decisions such that the average of the cumulative risk from all pulled arms should not be larger than a pre-defined threshold. To address this problem, we first study online convex programming in the full information setting where in each round the learner receives an adversarial convex loss and a convex constraint. We develop a meta algorithm leveraging online mirror descent for the full information setting and then extend it to contextual bandit with sequential risk constraints setting using expert advice. Our algorithms can achieve near-optimal regret in terms of minimizing the total cost, while successfully maintaining a sub-linear growth of accumulative risk constraint violation. We support our theoretical results by demonstrating our algorithm on a simple simulated robotics reactive control task.

BibTeX Reference
author = {Wen Sun and Debadeepta Dey and Ashish Kapoor},
title = {Safety-Aware Algorithms for Adversarial Contextual Bandit},
booktitle = {Thirty-fourth International Conference on Machine Learning (ICML 2017)},
year = {2017},
month = {August},
keywords = {Contextual Bandit},