rlworkgroup / garage / 3057

64%

push

travis-ci-com

Change replay buffer initialization (#356)

This commit changes replay buffer initialization in a more proper way.
Formerly, replay buffer initialized the shape of each key in the
transition by getting the flat_dim of env_spec.*_space. This works for
Box and Dict spaces. But it has two flaws:
1. It only considers when observation is Dict space. But doesn't deal
with the situation when action or other key has Dict space.
2. If action_space is Discrete(3), the ReplayBuffer._buffer["action"]
will have a shape of (size, time_horizon, 3). However, the actual value
of action is of shape (1). So that causes a serious error.

Based on the above disadvantages, a better idea would be initialize the
buffer shape by sampling. Like when adding a real transition to the
buffer. Or use env.sample(). I used the former one. Because when trying
the latter option, the Box.sample() causes an invalid range error. To
solve this we may need to creat an Box space instance and then use
sample(). It is not proper to do this in a construcing process in replay
buffer. Thus, I initialize the shape of each key in buffer when adding a
real transition. This has an advantage is that we may add info into the
buffer for GoalEnv.compute_reward().

Also, the update plotter line in ddpg were in wrong position. Fixed it.

14 of 14 new or added lines in 2 files covered. (100.0%)

9757 of 14995 relevant lines covered (65.07%)

0.65 hits per line