I recently bought 2 dual-port FDR ConnectX-3 HCAs and have put each HCA in a separate machine. I have them hooked up to each other via an Infiniband cable. There is no switch. OpenSM is installed on both machines, where both SMs have LIDs of 0x01. I installed the MLNX_OFED 2.0 tarball from the drivers section and followed the User Manual instructions for installation. The operating system for both machines is Ubuntu 12.04 (64-bit). I've also configured the ib0 and ib1 network interfaces and IP.
Running the 'hca_self_test.ofed' and 'ibdiagnet' commands seems to show that it is working fine. 'ibstat' shows that both ports are working. I can run ibping and ibv_rc_pingpong on the first port of each device to itself. I've also done an ibtrace and it seems to work okay. 'ibnodes' and 'ibhosts' seems to work as expected.
I wanted to test the throughput with ibv_rc_pingpong to the other machine by passing in the ip of ib0 interface of the other node. However it tells me that it "Couldn't connect to <other ip>:18515".
I've started OpenSM on both nodes. Is there anything special that needs to be done in terms of setting up the SM configurations for this particular switchless config?
Help is appreciated. Here is some accessory information from ibstat on both machines.
ibstat on node 1:
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0x0002c9030017c080
System image GUID: 0x0002c9030017c083
Port 1:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x0251486a
Port GUID: 0x0002c9030017c081
Link layer: InfiniBand
Port 2:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 2
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x0002c9030017c082
Link layer: InfiniBand
ibstat on node 2:
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0x0002c903009f5740
System image GUID: 0x0002c903009f5743
Port 1:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x0251486a
Port GUID: 0x0002c903009f5741
Link layer: InfiniBand
Port 2:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 2
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x0002c903009f5742
Link layer: InfiniBand