Clarification on terminology: 'Memory' vs 'Replay Buffer'

[Jendker]

Hello,

Thank you for providing such a well-documented and easy-to-follow RL framework!

I have a quick question regarding terminology: is there a specific reason why the replay buffers are referred to as "memories" in the code? From my understanding, the term "replay buffer" is more commonly used in the literature. I’d appreciate any clarification on this.

[Toni-SM]

Hi @Jendker

skrl implemnents a generic memory definition that are not tied to the agent implementation and can be used for any role, such as rollout or replay buffers. The skrl's memory is an empty shell, and the agents are in charge of defining the tensors according to their needs.

Depending on the agent, the memory can be used as:

• Replay buffer: large in size memories typically used by off-policy algorithms to storage environment transitions (state, action, reward, next state, done) with various sample strategies: random (implemented), prioritized, etc.
• Rollout buffer: small in size memories typically used by on-policy algorithms to store batches collected by the current policy and that are discarded after use. Apart from the environment transition, other policy-specific data (such as policy log_prob) is collected.

[Jendker]

Thank you for your explanation! I appreciate that you took the time for it!

[Toni-SM]

Also, you can take a look at the way PPO (on-policy) and DDPG (off-policy) uses the same memory class for each particular case:

• PPO:
  • tensor creation:

    skrl/skrl/agents/torch/ppo/ppo.py

    Lines 185 to 197 in 2bd3b7a

    	# create tensors in memory
    	if self.memory is not None:
    	self.memory.create_tensor(name="states", size=self.observation_space, dtype=torch.float32)
    	self.memory.create_tensor(name="actions", size=self.action_space, dtype=torch.float32)
    	self.memory.create_tensor(name="rewards", size=1, dtype=torch.float32)
    	self.memory.create_tensor(name="terminated", size=1, dtype=torch.bool)
    	self.memory.create_tensor(name="log_prob", size=1, dtype=torch.float32)
    	self.memory.create_tensor(name="values", size=1, dtype=torch.float32)
    	self.memory.create_tensor(name="returns", size=1, dtype=torch.float32)
    	self.memory.create_tensor(name="advantages", size=1, dtype=torch.float32)
    	# tensors sampled during training
    	self._tensors_names = ["states", "actions", "log_prob", "values", "returns", "advantages"]

  • sampling:

    skrl/skrl/agents/torch/ppo/ppo.py

    Line 378 in 2bd3b7a

    sampled_batches = self.memory.sample_all(names=self._tensors_names, mini_batches=self._mini_batches)

• DDPG
  • tensor creation:

    skrl/skrl/agents/torch/ddpg/ddpg.py

    Lines 179 to 187 in 2bd3b7a

    	# create tensors in memory
    	if self.memory is not None:
    	self.memory.create_tensor(name="states", size=self.observation_space, dtype=torch.float32)
    	self.memory.create_tensor(name="next_states", size=self.observation_space, dtype=torch.float32)
    	self.memory.create_tensor(name="actions", size=self.action_space, dtype=torch.float32)
    	self.memory.create_tensor(name="rewards", size=1, dtype=torch.float32)
    	self.memory.create_tensor(name="terminated", size=1, dtype=torch.bool)
    	self._tensors_names = ["states", "actions", "rewards", "next_states", "terminated"]

  • sampling:

    skrl/skrl/agents/torch/ddpg/ddpg.py

    Lines 332 to 333 in 2bd3b7a

    	sampled_states, sampled_actions, sampled_rewards, sampled_next_states, sampled_dones = \
    	self.memory.sample(names=self._tensors_names, batch_size=self._batch_size)[0]