Failed to compile 'raw' transport

[syspro4]

Hi,

I am trying to compile latest eRPC and getting some compilation errors. Can you please help?

Thanks!

Following are the setup details:

[root@dev_mc eRPC]# cat /etc/release | grep release
cat: /etc/lsb-release.d: Is a directory
Oracle Linux Server release 8.4
Red Hat Enterprise Linux release 8.4 (Ootpa)
Oracle Linux Server release 8.4
[root@dev_mc eRPC]#

[root@dev_mc eRPC]# ofed_info -s
MLNX_OFED_LINUX-5.4-1.0.3.0:
[root@dev_mc eRPC]#
[root@dev_mc eRPC]# lspci | grep Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
04:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
[root@dev_mc eRPC]#

[root@dev_mc eRPC]# cmake . -DPERF=ON -DTRANSPORT=raw; make -j
.
.
.

[ 48%] Building CXX object CMakeFiles/erpc.dir/src/util/numautils.cc.o
[ 49%] Building CXX object CMakeFiles/erpc.dir/src/transport_impl/fake/fake_transport.cc.o
[ 50%] Building CXX object CMakeFiles/erpc.dir/src/util/tls_registry.cc.o
[ 51%] Linking C shared library libhdr_histogram.so
[ 51%] Built target hdr_histogram
[ 52%] Linking C static library libhdr_histogram_static.a
[ 52%] Built target hdr_histogram_static
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::tx_burst(const erpc::Transport::tx_burst_item_t*, size_t)’:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:12:40: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘msg_buffer’; did you mean ‘msg_buffer_’?
const MsgBuffer* msg_buffer = item.msg_buffer;
^~~~~~~~~~
msg_buffer_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:27:14: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pkt_idx_’?
if (item.pkt_idx == 0) {
^~~~~~~
pkt_idx_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:32:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer_’?
sgl[0].lkey = msg_buffer->buffer.lkey;
^~~~~~
buffer_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:43:46: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pkt_idx_’?
pkthdr = msg_buffer->get_pkthdr_n(item.pkt_idx);
^~~~~~~
pkt_idx_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:46:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer_’?
sgl[0].lkey = msg_buffer->buffer.lkey;
^~~~~~
buffer_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:48:28: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pkt_idx_’?
size_t offset = item.pkt_idx * kMaxDataPerPkt;
^~~~~~~
pkt_idx_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:49:61: error: ‘const class erpc::MsgBuffer’ has no member named ‘buf’; did you mean ‘buf_’?
sgl[1].addr = reinterpret_cast<uint64_t>(&msg_buffer->buf[offset]);
^~~
buf_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:50:62: error: ‘const class erpc::MsgBuffer’ has no member named ‘data_size’; did you mean ‘data_size_’?
sgl[1].length = (std::min)(kMaxDataPerPkt, msg_buffer->data_size - offset);
^~~~~~~~~
data_size_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:51:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer_’?
sgl[1].lkey = msg_buffer->buffer.lkey;
^~~~~~
buffer_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:60:21: error: ‘struct erpc::pkthdr_t’ has no member named ‘headroom’; did you mean ‘headroom_’?
memcpy(&pkthdr->headroom[0], item.routing_info, hdr_copy_sz);
^~~~~~~~
headroom_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:60:39: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘routing_info’; did you mean ‘routing_info_’?
memcpy(&pkthdr->headroom[0], item.routing_info, hdr_copy_sz);
^~~~~~~~~~~~
routing_info_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:62:26: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘drop’; did you mean ‘drop_’?
if (kTesting && item.drop) {
^~~~
drop_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:64:60: error: ‘struct erpc::pkthdr_t’ has no member named ‘headroom’; did you mean ‘headroom_’?
auto* eth_hdr = reinterpret_cast<eth_hdr_t*>(pkthdr->headroom);
^~~~~~~~
headroom_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:69:48: error: ‘struct erpc::pkthdr_t’ has no member named ‘headroom’; did you mean ‘headroom_’?
reinterpret_cast<ipv4_hdr_t*>(&pkthdr->headroom[sizeof(eth_hdr_t)]);
^~~~~~~~
headroom_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::tx_flush()’:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:106:19: error: ‘huge_alloc’ was not declared in this scope
Buffer buffer = huge_alloc->alloc(pkt_size); // Get a registered buffer
^~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:106:19: note: suggested alternative: ‘huge_alloc_’
Buffer buffer = huge_alloc->alloc(pkt_size); // Get a registered buffer
^~~~~~~~~~
huge_alloc_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:109:17: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf_’?
memset(buffer.buf, 0, pkt_size);
^~~
buf_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:110:53: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf_’?
auto* pkthdr = reinterpret_cast<pkthdr_t*>(buffer.buf);
^~~
buf_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:139:24: error: ‘class erpc::Buffer’ has no member named ‘lkey’; did you mean ‘lkey_’?
sgl[0].lkey = buffer.lkey;
^~~~
lkey_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:159:3: error: ‘testing’ was not declared in this scope
testing.tx_flush_count++;
^~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:159:3: note: suggested alternative: ‘testing_’
testing.tx_flush_count++;
^~~~~~~
testing_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘size_t erpc::RawTransport::rx_burst()’:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:171:52: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf_’?
reinterpret_cast<pkthdr_t*>(&ring_extent.buf[recv_head * kRecvSize]);
^~~
buf_
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::post_recvs(size_t)’:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:200:24: error: invalid use of incomplete type ‘struct erpc::ibv_exp_wq_family’
int ret = wq_family->recv_burst(wq, &mp_recv_sge[mp_sge_idx], 1);
^~
In file included from /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:3:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.h:202:10: note: forward declaration of ‘struct erpc::ibv_exp_wq_family’
struct ibv_exp_wq_family *wq_family;
^~~~~~~~~~~~~~~~~
At global scope:
cc1plus: error: unrecognized command line option ‘-Wno-keyword-macro’ [-Werror]
cc1plus: error: unrecognized command line option ‘-Wno-nested-anon-types’ [-Werror]
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/erpc.dir/build.make:454: CMakeFiles/erpc.dir/src/transport_impl/raw/raw_transport_datapath.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In destructor ‘erpc::RawTransport::~RawTransport()’:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:58:40: error: aggregate ‘erpc::RawTransport::~RawTransport()::ibv_exp_release_intf_params rel_intf_params’ has incomplete type and cannot be defined
struct ibv_exp_release_intf_params rel_intf_params;
^~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:61:9: error: ‘ibv_exp_release_intf’ was not declared in this scope
ibv_exp_release_intf(resolve.ib_ctx, wq_family, &rel_intf_params) == 0,
^~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:61:9: note: suggested alternative: ‘ibv_exp_release_intf_params’
ibv_exp_release_intf(resolve.ib_ctx, wq_family, &rel_intf_params) == 0,
^~~~~~~~~~~~~~~~~~~~
ibv_exp_release_intf_params
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:64:17: error: ‘ibv_exp_destroy_flow’ was not declared in this scope
exit_assert(ibv_exp_destroy_flow(recv_flow) == 0,
^~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:64:17: note: suggested alternative: ‘ibv_destroy_flow’
exit_assert(ibv_exp_destroy_flow(recv_flow) == 0,
^~~~~~~~~~~~~~~~~~~~
ibv_destroy_flow
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:70:17: error: ‘ibv_exp_destroy_rwq_ind_table’ was not declared in this scope
exit_assert(ibv_exp_destroy_rwq_ind_table(ind_tbl) == 0,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:70:17: note: suggested alternative: ‘ibv_destroy_rwq_ind_table’
exit_assert(ibv_exp_destroy_rwq_ind_table(ind_tbl) == 0,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ibv_destroy_rwq_ind_table
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:73:17: error: ‘ibv_exp_destroy_wq’ was not declared in this scope
exit_assert(ibv_exp_destroy_wq(wq) == 0, "Failed to destroy WQ");
^~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:73:17: note: suggested alternative: ‘ibv_destroy_wq’
exit_assert(ibv_exp_destroy_wq(wq) == 0, "Failed to destroy WQ");
^~~~~~~~~~~~~~~~~~
ibv_destroy_wq
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:75:17: error: ‘ibv_exp_destroy_flow’ was not declared in this scope
exit_assert(ibv_exp_destroy_flow(recv_flow) == 0,
^~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:75:17: note: suggested alternative: ‘ibv_destroy_flow’
exit_assert(ibv_exp_destroy_flow(recv_flow) == 0,
^~~~~~~~~~~~~~~~~~~~
ibv_destroy_flow
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In member function ‘void erpc::RawTransport::init_basic_qp()’:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:125:31: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_cq_init_attr cq_init_attr’ has incomplete type and cannot be defined
struct ibv_exp_cq_init_attr cq_init_attr;
^~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:127:13: error: ‘ibv_exp_create_cq’ was not declared in this scope
send_cq = ibv_exp_create_cq(resolve.ib_ctx, kSQDepth, nullptr, nullptr, 0,
^~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:127:13: note: suggested alternative: ‘ibv_create_cq’
send_cq = ibv_exp_create_cq(resolve.ib_ctx, kSQDepth, nullptr, nullptr, 0,
^~~~~~~~~~~~~~~~~
ibv_create_cq
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:138:31: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_qp_init_attr qp_init_attr’ has incomplete type and cannot be defined
struct ibv_exp_qp_init_attr qp_init_attr;
^~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:140:28: error: ‘IBV_EXP_QP_INIT_ATTR_PD’ was not declared in this scope
qp_init_attr.comp_mask = IBV_EXP_QP_INIT_ATTR_PD;
^~~~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:140:28: note: suggested alternative: ‘IBV_QP_INIT_ATTR_PD’
qp_init_attr.comp_mask = IBV_EXP_QP_INIT_ATTR_PD;
^~~~~~~~~~~~~~~~~~~~~~~
IBV_QP_INIT_ATTR_PD
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:152:8: error: ‘ibv_exp_create_qp’ was not declared in this scope
qp = ibv_exp_create_qp(resolve.ib_ctx, &qp_init_attr);
^~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:152:8: note: suggested alternative: ‘ibv_create_qp’
qp = ibv_exp_create_qp(resolve.ib_ctx, &qp_init_attr);
^~~~~~~~~~~~~~~~~
ibv_create_qp
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:155:26: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_qp_attr qp_attr’ has incomplete type and cannot be defined
struct ibv_exp_qp_attr qp_attr;
^~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:159:13: error: ‘ibv_exp_modify_qp’ was not declared in this scope
rt_assert(ibv_exp_modify_qp(qp, &qp_attr, IBV_QP_STATE | IBV_QP_PORT) == 0);
^~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:159:13: note: suggested alternative: ‘ibv_modify_qp’
rt_assert(ibv_exp_modify_qp(qp, &qp_attr, IBV_QP_STATE | IBV_QP_PORT) == 0);
^~~~~~~~~~~~~~~~~
ibv_modify_qp
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In member function ‘void erpc::RawTransport::init_mp_recv_qp()’:
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:175:31: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_cq_init_attr cq_init_attr’ has incomplete type and cannot be defined
struct ibv_exp_cq_init_attr cq_init_attr;
^~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:177:13: error: ‘ibv_exp_create_cq’ was not declared in this scope
recv_cq = ibv_exp_create_cq(resolve.ib_ctx, kRecvCQDepth / 2, nullptr,
^~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:177:13: note: suggested alternative: ‘ibv_create_cq’
recv_cq = ibv_exp_create_cq(resolve.ib_ctx, kRecvCQDepth / 2, nullptr,
^~~~~~~~~~~~~~~~~
ibv_create_cq
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:182:26: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_cq_attr cq_attr’ has incomplete type and cannot be defined
struct ibv_exp_cq_attr cq_attr;
^~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:184:23: error: ‘IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS’ was not declared in this scope
cq_attr.comp_mask = IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:184:23: note: suggested alternative: ‘IBV_QP_INIT_ATTR_CREATE_FLAGS’
cq_attr.comp_mask = IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
IBV_QP_INIT_ATTR_CREATE_FLAGS
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:185:26: error: ‘IBV_EXP_CQ_IGNORE_OVERRUN’ was not declared in this scope
cq_attr.cq_cap_flags = IBV_EXP_CQ_IGNORE_OVERRUN;
^~~~~~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:185:26: note: suggested alternative: ‘IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN’
cq_attr.cq_cap_flags = IBV_EXP_CQ_IGNORE_OVERRUN;
^~~~~~~~~~~~~~~~~~~~~~~~~
IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:50: error: ‘IBV_EXP_CQ_CAP_FLAGS’ was not declared in this scope
rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0);
^~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:50: note: suggested alternative: ‘IBV_QP_ACCESS_FLAGS’
rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0);
^~~~~~~~~~~~~~~~~~~~
IBV_QP_ACCESS_FLAGS
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:13: error: ‘ibv_exp_modify_cq’ was not declared in this scope
rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0);
^~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:13: note: suggested alternative: ‘ibv_modify_cq’
rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0);
^~~~~~~~~~~~~~~~~
ibv_modify_cq
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:188:31: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_wq_init_attr wq_init_attr’ has incomplete type and cannot be defined
struct ibv_exp_wq_init_attr wq_init_attr;
^~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:191:26: error: ‘IBV_EXP_WQT_RQ’ was not declared in this scope
wq_init_attr.wq_type = IBV_EXP_WQT_RQ;
^~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:191:26: note: suggested alternative: ‘IBV_WQT_RQ’
wq_init_attr.wq_type = IBV_EXP_WQT_RQ;
^~~~~~~~~~~~~~
IBV_WQT_RQ
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:197:29: error: ‘IBV_EXP_CREATE_WQ_MP_RQ’ was not declared in this scope
wq_init_attr.comp_mask |= IBV_EXP_CREATE_WQ_MP_RQ;
^~~~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:197:29: note: suggested alternative: ‘IBV_QP_RATE_LIMIT’
wq_init_attr.comp_mask |= IBV_EXP_CREATE_WQ_MP_RQ;
^~~~~~~~~~~~~~~~~~~~~~~
IBV_QP_RATE_LIMIT
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:198:34: error: ‘IBV_EXP_MP_RQ_NO_SHIFT’ was not declared in this scope
wq_init_attr.mp_rq.use_shift = IBV_EXP_MP_RQ_NO_SHIFT;
^~~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:198:34: note: suggested alternative: ‘IBV_WC_IP_CSUM_OK_SHIFT’
wq_init_attr.mp_rq.use_shift = IBV_EXP_MP_RQ_NO_SHIFT;
^~~~~~~~~~~~~~~~~~~~~~
IBV_WC_IP_CSUM_OK_SHIFT
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:201:8: error: ‘ibv_exp_create_wq’ was not declared in this scope
wq = ibv_exp_create_wq(resolve.ib_ctx, &wq_init_attr);
^~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:201:8: note: suggested alternative: ‘ibv_create_wq’
wq = ibv_exp_create_wq(resolve.ib_ctx, &wq_init_attr);
^~~~~~~~~~~~~~~~~
ibv_create_wq
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:205:26: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_wq_attr wq_attr’ has incomplete type and cannot be defined
struct ibv_exp_wq_attr wq_attr;
^~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:207:23: error: ‘IBV_EXP_WQ_ATTR_STATE’ was not declared in this scope
wq_attr.attr_mask = IBV_EXP_WQ_ATTR_STATE;
^~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:207:23: note: suggested alternative: ‘IBV_WQ_ATTR_STATE’
wq_attr.attr_mask = IBV_EXP_WQ_ATTR_STATE;
^~~~~~~~~~~~~~~~~~~~~
IBV_WQ_ATTR_STATE
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:208:22: error: ‘IBV_EXP_WQS_RDY’ was not declared in this scope
wq_attr.wq_state = IBV_EXP_WQS_RDY;
^~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:208:22: note: suggested alternative: ‘IBV_WQS_RDY’
wq_attr.wq_state = IBV_EXP_WQS_RDY;
^~~~~~~~~~~~~~~
IBV_WQS_RDY
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:209:13: error: ‘ibv_exp_modify_wq’ was not declared in this scope
rt_assert(ibv_exp_modify_wq(wq, &wq_attr) == 0, "Failed to ready WQ");
^~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:209:13: note: suggested alternative: ‘ibv_modify_wq’
rt_assert(ibv_exp_modify_wq(wq, &wq_attr) == 0, "Failed to ready WQ");
^~~~~~~~~~~~~~~~~
ibv_modify_wq
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:8: error: use of enum ‘ibv_exp_query_intf_status’ without previous declaration
enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK;
^~~~~~~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:48: error: ‘IBV_EXP_INTF_STAT_OK’ was not declared in this scope
enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK;
^~~~~~~~~~~~~~~~~~~~
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:48: note: suggested alternative: ‘IBV_QP_CUR_STATE’
enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK;
^~~~~~~~~~~~~~~~~~~~
IBV_QP_CUR_STATE
~

[anujkaliaiitd]

Hi @syspro4. Thanks for your interest.

The current version of eRPC does not support the Raw transport anymore. I will mark it as broken in the CMakeLists.txt to avoid future confusion. If you wish to use the Raw transport, please use this version (https://github.com/erpc-io/eRPC/releases/tag/v0.1). You'll need to install an old version of Mellanox OFED (4.4 or older).

I'd recommend using eRPC's DPDK transport instead with DPDK 19.11 LTS. That will require installing Mellanox userspace drivers from rdma_core instead of Mellanox OFED.

[syspro4]

Thanks for the reply.

Regarding support for latest OFED drivers how difficult it is to fix the issue? Can it be possible to change the Experimental IBVERBs and use the standard IBVERBs? If the changes are straight forward then I can fix it with some help.

[anujkaliaiitd]

It's not difficult if we ignore the kDumb flag in raw_transport.h. That flag enables optimizations like multi-packet RX queue and "overrunning CQ", which aren't really necessary. As you said, most of the work here involves replacing experimental verbs with standard ibverbs.

I will be happy to test and merge a PR that migrates eRPC's raw transport to standard ibverbs, while ignoring the kDumb flag.

[syspro4]

Great! Thanks for the help!
I will give it a try and let you know.

[syspro4]

Hi,
Please find attached patch which has a fix for raw transport issue.
With this I am able to run hello_server & hello_client without any issue.

Following are the Mellanox versions I used:
Server: MLNX_OFED_LINUX-4.7-3.2.9.0
Client: MLNX_OFED_LINUX-5.0-2.1.8.0

Note: In the patch I have kept kDumb code commented so that in the future we can fix it if needed. I can remove the kDumb code if it is not needed.

TODO:
I am trying to understand the benchmark test and will run them later.

Questions:

1. Will eRPC work on standard NICs (without DPDK)?
2. I want to run RPC test which gives these numbers. "(Performance for small 32-byte RPCs: ~10M RPCs/sec with one CPU core, 60--80M RPCs/sec with one NIC.)"
   Can you please let me know how to run this test?

Please let me know if any changes needed.

Thanks,
Vaibhav

eRPC_raw_transport_patch.tar.gz

[anujkaliaiitd]

Thanks so much! I've tested that it seems to work in the latency benchmark with rdma_core. May I suggest the following:

• Please create a pull request so we can track this. If possible, please run clang-format on the changed files.
• The changes in verbs_common.h (to count only active devices) are not needed. eRPC takes a phy_port argument, which for verbs-based transports corresponds to the index of the device as listed by ibv_devinfo, which includes inactive devices.

If you are unable to, I will make the required changes soon and merge.

Answers to your questions:

• eRPC currently doesn't work on standard NICs without DPDK. I currently have a skeleton for a transport backend (fake_transport.h and fake_transport.cc) that can be used to implement a UDP sockets--based (ideally with AF_XDP) transport. I can test and merge if someone contributes this.
• The server_rate benchmark is best suited to get high RPCs/sec. apps/server_rate/config allows configuring the number of server and client threads, and you can run multiple client processes to saturate the server.

[syspro4]

Thank you for testing the patch.
I will sure address your comments and repost the patch.

I tried the server_rate app & I fixed some compilation issue and then ran ./scripts/do.sh from serve & client but I see following messages and I am not sure what to make out of the messages:

Note: I enabled LOG_LEVEL=info

On server side I only see following messages continuously. I do not see any progress
"thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00"

On client side I see following message & nothing is happening after that:
"Process 1, thread 15: Connected. Starting work."

Detailed logs:
====================== server side ====================================
server_dev :: ~/eRPC » cat scripts/autorun_process_file
192.168.2.222 31850 0
192.168.2.223 31850 0
server_dev :: ~/eRPC »

server_dev :: ~/eRPC » ./scripts/do.sh 0 0
Installing modded drivers
do.sh: Launching process 0 on NUMA node 0
46:105830 INFOR: eRPC Nexus: Launching 0 background threads.
50:099920 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX]
50:100049 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 0]. Issue: None. Sending response.
50:100065 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 0].
50:100441 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX]
50:100500 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 0]. Issue: None. Sending response.
50:100514 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 1].
50:100858 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]
50:100925 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 1]. Issue: None. Sending response.
50:100941 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 0].
50:101425 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX]
50:101486 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 0]. Issue: None. Sending response.
50:101510 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 2].
50:101917 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]
50:101976 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 1]. Issue: None. Sending response.
50:101990 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 1].
50:102902 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]
50:102945 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX]
50:102971 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 1]. Issue: None. Sending response.
50:102988 INFOR: 50:102995 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 2].
Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 2]. Issue: None. Sending response.
50:103029 INFOR: Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 0].
50:103376 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX]
50:103419 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX]
50:103433 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 2]. Issue: None. Sending response.
50:103463 INFOR: 50:103452 INFOR: Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 1].
eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX]
50:103521 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 2]. Issue: None. Sending response.
50:103537 INFOR: 50:103540 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 3]. Issue: None. Sending response.
Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 2].
50:103558 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 0].
50:104667 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX]
50:104710 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX]
50:104726 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 3]. Issue: None. Sending response.
50:104754 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 1].
50:104811 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 3]. Issue: None. Sending response.
.
.
.
50:134880 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 13, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 14].
eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX]
Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 15].
50:134909 INFOR: 50:134927 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 15, S: 2]. Issue: None. Sending response.
50:134945 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX]
Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 15].
50:134980 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 15, S: 3]. Issue: None. Sending response.
50:134996 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 15].
thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00
thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00

====================== client side ====================================
client_dev :: ~/eRPC » cat scripts/autorun_process_file
192.168.2.222 31850 0
192.168.2.223 31850 0
client_dev :: ~/eRPC »

client_dev :: ~/eRPC » ./scripts/do.sh 1 0
Installing modded drivers
do.sh: Launching process 1 on NUMA node 0
1 192.168.2.222 31850 02 192.168.2.223 31850 01 192.168.2.222 31850 02 192.168.2.223 31850 050:067736 INFOR: eRPC Nexus: Launching 0 background threads.
50:067751 INFOR: eRPC Nexus: Launching session management thread on core 23.
50:067887 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 192.168.2.223.
50:099929 INFOR: 50:100390 INFOR: 50:100396 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 0]
Rpc 7: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX].
50:100426 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 1]
50:100754 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 0]
Rpc 0: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX].
50:100977 INFOR: RawTransport created for Rpc ID 15. Device mlx4_0/eth5, port 2. IPv4 192.168.2.223, MAC f4:52:14:74:d9:e1. Datapath UDP port 31897.
50:101279 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 2]
50:101417 INFOR: Rpc 1: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX].
50:101933 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 1]
50:102400 INFOR: 50:102402 INFOR: Rpc 0: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX].
Rpc 7: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX].
50:102455 INFOR: ERPC: Vaibhav num_devices 1
50:102752 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 2]
50:102798 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 0]
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 11]
50:144159 INFOR: 50:144167 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 11] for session 0. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 11]
50:144257 INFOR: 50:144259 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 11] for session 1. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 10]
50:144278 INFOR: 50:144280 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 10] for session 2. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 10]
50:144296 INFOR: 50:144297 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 10] for session 3. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 6, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 11]
50:144313 INFOR: 50:144319 INFOR: Rpc 6: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 11] for session 2. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 6, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 11]
50:155156 INFOR: 50:155162 INFOR: Rpc 6: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 11] for session 3. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 12]
50:155177 INFOR: 50:155186 INFOR: Rpc 12: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 12] for session 0. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 12]
50:166160 INFOR: 50:166165 INFOR: Rpc 12: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 12] for session 1. Issue: None. Session connected.
eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 12]

50:210277 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 15] for session 0. Issue: None. Session connected.
50:210288 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 15] for session 1. Issue: None. Session connected.
50:210297 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 15] for session 2. Issue: None. Session connected.
50:210307 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 15] for session 3. Issue: None. Session connected.
50:210307 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 15] for session 3. Issue: None. Session connected.
Process 1, thread 7: Connected. Starting work.
Process 1, thread 0: Connected. Starting work.
Process 1, thread 1: Connected. Starting work.
thread_id: median_us 5th_us 99th_us 999th_us Mops
Process 1, thread 5: Connected. Starting work.
Process 1, thread 14: Connected. Starting work.
Process 1, thread 4: Connected. Starting work.
Process 1, thread 3: Connected. Starting work.
Process 1, thread 8: Connected. Starting work.
Process 1, thread 10: Connected. Starting work.
Process 1, thread 2: Connected. Starting work.
Process 1, thread 6: Connected. Starting work.
Process 1, thread 11: Connected. Starting work.
Process 1, thread 12: Connected. Starting work.
Process 1, thread 9: Connected. Starting work.
Process 1, thread 13: Connected. Starting work.
Process 1, thread 15: Connected. Starting work.

---

[anujkaliaiitd]

I pushed the required compilation fixes to server_rate.cc. It works on my cluster with the raw transport.

In your case, I suspect that the datapath packets are getting dropped somewhere. You could try re-compiling eRPC with detailed packet-level logging (ex: cmake . -DLOG_LEVEL=trace). Then when you run the server_rate client and server, they will create files in /tmp with detailed per-packet logs.

[syspro4]

You are correct. In the log I see packet drop error. I am getting it for hello_server/client test too.

Log: /tmp/erpc_trace_31850-rpc_0:
83:467073 TRACE: Rpc 0, lsn 0 (server_dev): TX [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Slot [num_tx 0, num_rx 0].
83:467120 TRACE: eRPC RawTransport: Sending packet (idx = 0, drop = 0). SGE #1 72B, SGE #2 = 0B. pkthdr = [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Frame header = [ETH: dst f4:52:14:5e:60:e0, src f4:52:14:74:d9:e1, eth_type 2048], [IPv4: ihl 5, version 4, ecn 1, tot_len 58, id 0, frag_off 0, ttl 128, protocol 17, check 0, src IP 192.168.2.223, dst IP 192.168.2.222], [UDP: src_port 31882, dst_port 31882, len 38, check 0].
83:472158 REORD: Rpc 0, lsn 0 (server_dev): Pkt loss suspected for req 8 ([num_tx 1, num_rx 0]). Action: Retransmitting requests.
83:472166 TRACE: Rpc 0, lsn 0 (server_dev): TX [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Slot [num_tx 0, num_rx 0].
83:472175 TRACE: eRPC RawTransport: Sending packet (idx = 0, drop = 0). SGE #1 72B, SGE #2 = 0B. pkthdr = [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Frame header = [ETH: dst f4:52:14:5e:60:e0, src f4:52:14:74:d9:e1, eth_type 2048], [IPv4: ihl 5, version 4, ecn 1, tot_len 58, id 0, frag_off 0, ttl 128, protocol 17, check 0, src IP 192.168.2.223, dst IP 192.168.2.222], [UDP: src_port 31882, dst_port 31882, len 38, check 0].

I did rping test between server & client and it is working fine.

server_dev :: ~/eRPC 1 » rping -s -C 10 -v <<<
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrst
server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstu
server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuv
server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvw
server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwx
server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxy
server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz
server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyzA
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
server_dev :: ~/eRPC »

client_dev :: ~/eRPC » /usr/bin/rping -c -a 192.168.2.222 -C 10 -v
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyzA
client DISCONNECT EVENT...
server_dev :: ~/eRPC »

Please help.

Thanks

[syspro4]

Hi,
I tried with different set of server/client but I see same "Pkt loss suspected for req 8" error in the trace log file. All these servers can do rping without any issue. Can you please help?

Thanks!

[anujkaliaiitd]

Couple of suggestions:

• Could you try building with eRPC's DPDK transport and see if the packets go through? There are detailed instructions in the README to use DPDK with eRPC (in the section on Azure).

• One issue could be that the switch or NIC drops packets because the Raw transport sets CRC = 0. The DPDK transport does the right thing and sets valid CRC.

• The ultimate way to debug this is a a bit complicated: You'll need to construct the exact Ethernet frame that eRPC is sending in a simpler test application (the /tmp logs should help in doing this), and modify the packet fields until it goes through. That should shed some light on the packet drops.

[anujkaliaiitd]

Ah, I just remembered that you were able to run the hello_world test, so CRC is likely not an issue. Still, it might be worth trying with the DPDK transport. Other suggestions:

• Are the IP addresses in /tmp logs as expected?
• Could you try reducing the number of client and server threads in server_rate/config to one thread per process?

[syspro4]

Thanks for the suggestions.

Actually earlier hello_world also did not work. I retested it and I see same pkt loss message.

When you tested my patch on your setup did you changed anything in your code base?

[syspro4]

I tried erpc dpdk but that is giving following errors.

server_dev :: ~/eRPC » ./build/hello_server
6:616156 INFOR: eRPC Nexus: Launching 0 background threads.
6:616202 INFOR: eRPC Nexus: Launching session management thread on core 23.
6:617241 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 192.168.2.222.
6:617452 INFOR: DPDK transport for Rpc 0 initializing DPDK EAL.
EAL: Detected 32 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.2 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.3 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 15b3:1003 net_mlx4
PMD: net_mlx4: PCI information matches, using device "mlx4_0" (VF: false)
PMD: net_mlx4: 2 port(s) detected
PMD: net_mlx4: port 1 MAC address is f4:52:14:5e:60:e0
PMD: net_mlx4: port 2 MAC address is f4:52:14:5e:60:e1
6:947277 WARNG: Running as primary DPDK process. eRPC DPDK daemon is not running.
6:947316 INFOR: DPDK transport for Rpc 0 got QP 0
6:947354 INFOR: dev_info.rx_desc_lim.nb_max 0 kNumRxRingEntries 4096 <========================== Added this debug print
terminate called after throwing an instance of 'std::runtime_error'
what(): Device RX ring too small
[2] 10374 abort ./build/hello_server

server_dev :: ~/eRPC »

I tried running dpdk-helloworld test program to check if dpdk is configured and working correctly and it is working fine.

server_dev :: ~/dpdk-stable-19.11.5/build » examples/dpdk-helloworld -l 0-3 -n 4
EAL: Detected 32 lcore(s)
/**
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.2 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.3 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 15b3:1003 net_mlx4
hello from core 1
hello from core 2
hello from core 3
hello from core 0
server_dev :: ~/dpdk-stable-19.11.5/build »

[anujkaliaiitd]

It's strange that the port reports zero as the max number of RX descriptors. Could you try:

• Launch DPDK testpmd in interactive mode: testpmd -- -i
• info port 0

This should show the number of RX ring descriptors detected by testpmd.

Also: I see your system has two CX3 ports. Are both connected to the switch? If not, please ensure that eRPC is using the correct port.

[syspro4]

Thanks for the reply.
I see following info:

testpmd> show port info 0

********************* Infos for port 0 *********************
MAC address: F4:52:14:5E:60:E0
Device name: mlx4_0 port 1
Driver name: net_mlx4
Connect to socket: 1
memory allocation on the socket: 1
Link status: up
Link speed: 40000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Supported RSS offload flow types:
ipv4
ipv4-frag
ipv4-tcp
ipv4-udp
ipv4-other
ipv6
ipv6-frag
ipv6-tcp
ipv6-udp
ipv6-other
user defined 15
user defined 16
user defined 17
Minimum size of RX buffer: 32
Maximum configurable length of RX packet: 65536
Maximum configurable size of LRO aggregated packet: 0
Current number of RX queues: 1
Max possible RX queues: 1024
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 1024
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 65535
Max segment number per MTU/TSO: 65535
testpmd>

testpmd> show config rxtx
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=2
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
port 1: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
testpmd>

I don't see the ring descriptors in "info port" command but in "show config rxtx" I see 0 for descriptor value.

Regarding the CX3 ports, yes I have a card with 2 ports and the program is picking up correct port. If we manually select the port then how can I do that?

Thanks!