Commits on Sep 22, 2021

1. Add a Herring-like benchmark …

   Summary:
   This diff adds two benchmarks that perform an allreduce operation using internally a technique inspired from Amazon's Herring paper. The two versions, one for TCP and one for InfiniBand interconnect, operate similarly: they spawn a set of clients (one per device, with multiple devices per machine, and multiple machines), then each allreduce call first performs a NCCL reduce or reduce_scatter step within each machine, then a shared send to a set of servers (one per machine, or one per device), then these servers perform the aggregation, and send back the result to the clients, which then do a final NCCL broadcast or all_gather step. Multiple allreduce calls, one per "bucket", are launched in parallel during each "epoch". There's also a baseline script that does these allreduces simply using NCCL.
   
   Since the benchmarks depend on PyTorch, they are a bit "awkward" to build and install from within the TensorPipe repo. So, instead of integrating them inside our CMake system, I opted to build them as a separate PyTorch extension, hence giving them a setup.py file. Thus the installation steps are to first install TensorPipe on its own, then NCCL, then PyTorch, and then the benchmarks.
   
   Differential Revision: D30220559
   
   fbshipit-source-id: 43ab5bc440271942698313e6b06b39104a0e44de

   

   lw authored and facebook-github-bot committed Sep 22, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for e059033
   • Browse repository at this point

   Copy the full SHA

   e059033 View commit details

   Browse the repository at this point in the history