CudaBasic/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevice fails on CUDA 11.1

[beauby]

No description provided.

[pruthvistony]

Along with above test case I see below test case fail too - On Cuda 11.1 + CuDNN 8,
CudaIpc/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevices

[beauby]

Interesting, would you mind pasting the output here?

[pruthvistony]

[----------] 3 tests from CudaBasic/CudaMultiGPUChannelTestSuite
[ RUN ] CudaBasic/CudaMultiGPUChannelTestSuite.SendAcrossDevices/0
[ OK ] CudaBasic/CudaMultiGPUChannelTestSuite.SendAcrossDevices/0 (10046 ms)
[ RUN ] CudaBasic/CudaMultiGPUChannelTestSuite.SendReverseAcrossDevices/0
[ OK ] CudaBasic/CudaMultiGPUChannelTestSuite.SendReverseAcrossDevices/0 (9914 ms)
[ RUN ] CudaBasic/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevices/0
/root/tensorpipe/tensorpipe/test/channel/channel_test_cuda_multi_gpu.cc:304: Failure
Value of: initializedCudaContexts({1})
Actual: false (a CUDA context should have been initialized on device #1 but that didn't happen)
Expected: true
/root/tensorpipe/tensorpipe/test/peer_group.h:211: Failure
Expected equality of these values:
0
exitStatus
Which is: 1
[ FAILED ] CudaBasic/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevices/0, where GetParam() = 0x55e3421fa068 (9753 ms)
[----------] 3 tests from CudaBasic/CudaMultiGPUChannelTestSuite (29713 ms total)

[----------] 3 tests from CudaIpc/CudaMultiGPUChannelTestSuite
[ RUN ] CudaIpc/CudaMultiGPUChannelTestSuite.SendAcrossDevices/0
[ OK ] CudaIpc/CudaMultiGPUChannelTestSuite.SendAcrossDevices/0 (10177 ms)
[ RUN ] CudaIpc/CudaMultiGPUChannelTestSuite.SendReverseAcrossDevices/0
[ OK ] CudaIpc/CudaMultiGPUChannelTestSuite.SendReverseAcrossDevices/0 (9965 ms)
[ RUN ] CudaIpc/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevices/0
/root/tensorpipe/tensorpipe/test/channel/channel_test_cuda_multi_gpu.cc:304: Failure
Value of: initializedCudaContexts({1})
Actual: false (a CUDA context should have been initialized on device #1 but that didn't happen)
Expected: true
/root/tensorpipe/tensorpipe/test/peer_group.h:211: Failure
Expected equality of these values:
0
exitStatus
Which is: 1
[ FAILED ] CudaIpc/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevices/0, where GetParam() = 0x55e3421fa078 (9963 ms)
[----------] 3 tests from CudaIpc/CudaMultiGPUChannelTestSuite (30106 ms total)

[----------] 1 test from CudaIpc/CudaIpcChannelTestSuite
[ RUN ] CudaIpc/CudaIpcChannelTestSuite.CannotCommunicateInSameProcess/0
[ OK ] CudaIpc/CudaIpcChannelTestSuite.CannotCommunicateInSameProcess/0 (9149 ms)
[----------] 1 test from CudaIpc/CudaIpcChannelTestSuite (9149 ms total)

[----------] Global test environment tear-down
[==========] 92 tests from 26 test suites ran. (452172 ms total)
[ PASSED ] 90 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] CudaBasic/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevices/0, where GetParam() = 0x55e3421fa068
[ FAILED ] CudaIpc/CudaMultiGPUChannelTestSuite.SendAcrossNonDefaultDevices/0, where GetParam() = 0x55e3421fa078

2 FAILED TESTS
YOU HAVE 2 DISABLED TESTS