Hi,
I'm trying to measure my ib bandwidth by using ib_send_bw. Im using the Connect X-3 with 56GbE. I connect two machine point to point with infiniband.
[mimos@wnode4 ~]$ ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:f452:1403:0033:b4c1
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:f452:1403:0033:b4c2
base lid: 0x1
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 56 Gb/sec (4X FDR)
link_layer: InfiniBand
[mimos@wnode4 ~]$ ib_send_bw -a -R -i 2 if2
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 2048[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
local address: LID 0x01 QPN 0x008a PSN 0xe66ab2
remote address: LID 0x02 QPN 0x008a PSN 0x6d5c92
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
2 1000 10.72 6.87 3.599459
4 1000 22.02 21.96 5.756531
8 1000 44.12 44.02 5.770259
16 1000 88.24 88.04 5.769635
32 1000 175.85 175.60 5.753926
64 1000 351.69 350.98 5.750456
128 1000 674.28 643.71 5.273302
256 1000 1341.63 1252.38 5.129763
512 1000 2324.42 2131.24 4.364776
1024 1000 2469.36 2442.37 2.500986
2048 1000 2678.66 2672.90 1.368524
4096 1000 2790.35 2790.23 0.714299
8192 1000 2878.93 2871.59 0.367564
16384 1000 2908.87 2905.95 0.185981
32768 1000 2934.02 2932.92 0.093853
65536 1000 2948.41 2948.18 0.047171
131072 1000 2953.17 2952.56 0.023620
262144 1000 2955.65 2955.38 0.011822
524288 1000 2956.79 2956.77 0.005914
1048576 1000 2957.37 2957.31 0.002957
2097152 1000 2957.75 2957.73 0.001479
4194304 1000 2957.68 2957.67 0.000739
8388608 1000 2957.71 2957.66 0.000370
---------------------------------------------------------------------------------------
QUESTION:
- why are the ib_send_bw showing my peak BW around half the speed as it suppose to be?
- Any available benchmark i could refer to?
Please advice, thanks.
Amirul.