Hello,
With iperf 3 I see around 36-40 Gpbs :
"numactl --cpunodebind=1 --membind=1 iperf -c 1.1.1.11-ib -P 8"
With IPoIB I believe affinity management is key. try using the set_irq_affinity script or the MLNX affinity tuner located in /etc/infiniband/openib.conf
I guess that with ConnectX-3s QDRs you should see around the twenties, can you copy paste your ib_write_bw tests ?
Cheers