In many reinforcement learning applications executing a poor policy may be costly or even dangerous. Thus, it is desirable to determine confidence interval lower bounds on the performance of any given policy without executing said policy. Current methods for high confidence off-policy evaluation require a substantial amount of data to achieve a tight lower bound, while existing model-based methods only address the problem in discrete state spaces. We propose two bootstrapping approaches combined with learned MDP transition models in order to efficiently estimate lower confidence bounds on policy performance with limited data in both continuous and discrete state spaces. Since direct use of a model may introduce bias, we derive a theoretical upper bound on model bias when we estimate the model transitions with i.i.d. sampled trajectories. This bound can be used to guide selection between the two methods. Finally, we empirically validate the data-efficiency of our proposed methods across three domains and analyze the settings where one method is preferable to the other.
from cs.AI updates on arXiv.org http://ift.tt/28MFYDc
via IFTTT
No comments:
Post a Comment