We have a small network for testing. Its layout is shown below. The orange-colored IP addresses are the current IPoIB setup. They are done with /etc/sysconfig/network-scripts/ifcfg0-ib0 using IP addresses in an existing subnet 192.168.11.0/24. The switch is a Mellanox SN2410. As indicated in the picture, the gateway IP address for this subnet is 192.168.11.3, assigned to a 10GbE interface of the bastion host.
With this IPoIB setup,
- I have run into problems with them using e.g. ipv_rc_pingpong - please see below for test session:
- We can't get more than 10Gbps with iperf3 or iperf over such IPoIB IPv4 addressse, despite the fact that we have a SB7700 IB switch!
A. Server:
local address: LID 0x0000, QPN 0x0000e0, PSN 0x79172e, GID ::
Failed to modify QP to RTR
Couldn't connect to remote QP
I noticed the GID as :: which is not right to me.
B. Client:
local address: LID 0x0000, QPN 0x0000e0, PSN 0x4ba0b3, GID ::
client read: Unknown error 524
Couldn't read remote address
As above, the GID is shown as ::.
ping to each IP address works however.
ibping also works:
A' Server:
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.33.5040
Hardware version: 1
Node GUID: 0x7cfe900300b98f30
System image GUID: 0x7cfe900300b98f33
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 5
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x7cfe900300b98f31
Link layer: InfiniBand
[root@sc2u0n0 ~]# ibping -S -C mlx4_0 -P 1
B' Client:
--- sc2u0n0.(none) (Lid 5) ibping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 573 ms
rtt min/avg/max = 0.017/0.057/0.099 ms
All nodes run:
OFED: MLNX_OFED_LINUX-3.3-1.0.4.0-rhel7.2-x86_64
OS: CentOS 7.2: Linux 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
As such, I suspect that we can't use the 192.168.11.0/24 for IPoIB. But to use another subnet, the question is where and how do I setup the gateway?
Setting up IPoIB seems to be simple, especially the Mellanox Academy has this short video: "IPoIB Performance Measurement". Towards the end, the speaker for the video showed the use of ifconfig to setup temporary IPv4 addresses for the two IB equipped servers. But IMHO that's not enough! According to my understanding, an Ethernet broadcast domain like what IPoIB uses is an "emulated" one. Thus, somewhere there must be a gateway. I believe this should be done in the IB switch. But I have Googled a lot without finding any info as to how to do this part. I read carefully my SB7700 IB switch manuals, no clues there either.
I am still getting up to speed with setting up InfiniBand, although I am very comfortable in configuring switches and HCAs already. If there is something that I missed, please let me know.
Thanks!