Segment fault depending on PCAP file size

[pchaseh]

I'm attempting to use pktgen to replay a PCAP file, and I ran into the following crash which left me scratching my head:

sudo -E ./usr/local/bin/pktgen -l 1-2 --proc-type auto --log-level 7 --file-prefix pg -d librte_net_mlx5.so -a 86:00.0 -- -v -T -P -m [2:2].0 -s 0:/home/chase-hiltz/pktgen-dpdk/syn.pcap -f themes/black-yellow.theme

*** Copyright(c) <2010-2024>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by <<<

EAL: Detected CPU lcores: 48
EAL: Detected NUMA nodes: 2
EAL: Auto-detected process type: PRIMARY
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized


Lua 5.4.4  Copyright (C) 1994-2022 Lua.org, PUC-Rio
  Create: 'RX-L2/P0/S1     ' - Memory used (MBUFs 16,384 x size  2,176) =   34,817 KB @ 0x1100af5e80
  Create: 'TX-L2/P0/S1     ' - Memory used (MBUFs 16,384 x size  2,176) =   34,817 KB @ 0x11032fdf00
  Create: 'SP-L2/P0/S1     ' - Memory used (MBUFs  1,024 x size  2,176) =    2,177 KB @ 0x1100bf9700
Err: lcore 2 already in use
                                                      Total memory used =   71,811 KB
>>> Packet Max Burst 128/128, RX Desc 1024, TX Desc 2048, mbufs/port 24576, mbuf cache 128
Initialize Port 0 ...
                                                         Port memory used =  71811 KB
** Device Info (86:00.0, if_index:135, flags 00000075) **
   min_rx_bufsize :   32  max_rx_pktlen     :65536  hash_key_size :   40
   max_rx_queues  : 1024  max_tx_queues     : 1024  max_vfs       :    0
   max_mac_addrs  :  128  max_hash_mac_addrs:    0  max_vmdq_pools:    0
   vmdq_queue_base:    0  vmdq_queue_num    :    0  vmdq_pool_base:    0
   nb_rx_queues   :    1  nb_tx_queues      :    1  speed_capa    : 00005d20

   flow_type_rss_offloads:f00000000803afbc  reta_size             :  512
   rx_offload_capa       :VLAN_STRIP IPV4_CKSUM UDP_CKSUM TCP_CKSUM VLAN_FILTER SCATTER TIMESTAMP KEEP_CRC RSS_HASH BUFFER_SPLIT
   tx_offload_capa       :VLAN_INSERT IPV4_CKSUM UDP_CKSUM TCP_CKSUM TCP_TSO OUTER_IPV4_CKSUM VXLAN_TNL_TSO GRE_TNL_TSO GENEVE_TNL_TSO MULTI_SEGS MBUF_FAST_FREE UDP_TNL_TSO IP_TNL_TSO
   rx_queue_offload_capa :000000000019600f  tx_queue_offload_capa :0000000000000000
   dev_capa              :0000000000000010

  RX Conf:
     pthresh        :    0 hthresh          :    0 wthresh        :    0
     Free Thresh    :    0 Drop Enable      :    0 Deferred Start :    0
     offloads       :0000000000000000
  TX Conf:
     pthresh        :    0 hthresh          :    0 wthresh        :    0
     Free Thresh    :    0 RS Thresh        :    0 Deferred Start :    0
     offloads       :0000000000000000
  Rx: descriptor Limits
     nb_max         :65535  nb_min          :    0  nb_align      :    1
     nb_seg_max     :65535  nb_mtu_seg_max  :65535
  Tx: descriptor Limits
     nb_max         :65535  nb_min          :    0  nb_align      :    1
     nb_seg_max     :   40  nb_mtu_seg_max  :   40
  Rx: Port Config
     burst_size     :   64  ring_size       :  256  nb_queues     :   16
  Tx: Port Config
     burst_size     :   64  ring_size       :  256  nb_queues     :   16
  Switch Info: 86:00.0
     domain_id      :    0  port_id         :65535



====== Pktgen got a Segment Fault

Obtained 15 stack frames.
./usr/local/bin/pktgen(+0x292ac) [0x560d3a7002ac]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffbce1c9520]
/usr/local/lib/x86_64-linux-gnu/librte_common_mlx5.so.25(+0x15e15) [0x7ffbcd887e15]
/usr/local/lib/x86_64-linux-gnu/librte_mempool.so.25(rte_mempool_mem_iter+0x37) [0x7ffbce686397]
/usr/local/lib/x86_64-linux-gnu/librte_common_mlx5.so.25(mlx5_mr_mempool_register+0x375) [0x7ffbcd888f95]
/usr/local/lib/x86_64-linux-gnu/librte_common_mlx5.so.25(+0x1475e) [0x7ffbcd88675e]
/usr/local/lib/x86_64-linux-gnu/librte_mempool.so.25(rte_mempool_walk+0x38) [0x7ffbce688428]
/usr/local/lib/x86_64-linux-gnu/librte_common_mlx5.so.25(mlx5_dev_mempool_subscribe+0xcf) [0x7ffbcd88743f]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x15a3b) [0x7ffbcd8b8a3b]
/usr/local/lib/x86_64-linux-gnu/librte_ethdev.so.25(rte_eth_dev_start+0xd4) [0x7ffbce6cb684]
./usr/local/bin/pktgen(+0x2a620) [0x560d3a701620]
./usr/local/bin/pktgen(+0xe157) [0x560d3a6e5157]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ffbce1b0d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7ffbce1b0e40]
./usr/local/bin/pktgen(+0xea55) [0x560d3a6e5a55]

I discovered that this problem occurs depending on the size of the PCAP file being loaded, more specifically I'm using a script along the lines of the following to generate one:

from scapy.all import *

pkts = []
for i in range(10000):
    pkts.append(Ether(src="xx:xx:xx:xx:xx:xx", dst="yy:yy:yy:yy:yy:yy") / Dot1Q(vlan=1) / IP(src="1.2.3.4", dst="4.3.2.1") / TCP(sport=i, dport=i, seq=123, options=[("MSS", 1440)]))

wrpcap('./syn.pcap', pkts)

I've found that pktgen crashes whenever the PCAP file contains 14 or more of these packets. At 13 packets, the total file size is 1038 bytes, whereas with 14 (where I observe the problem) it occupies 1116 bytes. Unfortunately I haven't had much time to dig into where the issue may lay

[KeithWiles]

It should not crash and I will have to look at the problem. I have been able to load much large files then 14 frames at one time :-(

[KeithWiles]

Please look at the new Pktgen release 24.10.0 and use the latest DPDK version as DPDK changed again, which caused compile problems.