Sequential vs Parallel trainer

[berttggg]

May I Know what is the fundamental difference between Sequential and Parallel trainer?

According to the documentation, the sequential trainer process agents sequentially while the parallel one process agents in a parallel way, meaning training time for the parallel trainer is much lesser than the sequential trainer?

If yes, why all the provided examples are using the sequential trainer?

Thank you.

[Toni-SM]

Hi @berttggg

The next charts show some stats about the use of the sequential and parallel trainer for the following case:

DDPG, TD3 and SAC training, with different trainers and memory modalities, using the same and hyperparameters and the number of parallel environments for Omniverse Isaac Gym Ant. Standalone and simultaneous sequential and parallel (without shared memory) training was done on 64 environments per agents. Simultaneous sequential training (with shared memory) was also done on 64 environments for all agents (where the scope was divided equally among the available agents).

Summary: Training several algorithms (or the same algorithm several times) is faster by using simultaneous training rather than standalone training.

For few agents, there is no gain on training time reduction. However, some time ago I trained about 10000 PPO agents for some experiments. At that time, the parallel training (of 24 agents at a time, what the GPU memory allowed me) was faster than the sequential one. Unfortunately, I don't keep statistics about it