Prierarchy: Implicit Hierarchies

Note that, if the prior is the uniform distribution over actions, then this is exactly equivalent to traditional policy gradient algorithms. My claim, which I won’t demonstrate today, is that you can achieve a similar effect by pre-training a recurrent policy to mimic the frame-skip action distribution, and then applying a policy gradient algorithm to this policy with an appropriately modified discount factor (e.g. 0.99 might become 0.990.25) and a KL regularizer against the frame-skip prior. The main benefit of HRL is really just that it compresses sequences of low-level actions, resulting in better exploration and less noise in the high-level policy gradient.

Source: blog.aqnichol.com