Infiniband newbie here, I'm trying to get a small 2-node HPC cluster going w/ a shared large data space on a separate server. Here's the hardware info:
Infiniband cards: HP593412-001 (Mellanox 26428) ConnectX-2 QDR VPI; firmware = 2.9.1
Nodes: Windows10, using WinOF_4.80_win81_x64
Server: Debian 8.4 (kernel 3.16.0-4-amd64), using OFED 3.2-2.0.0.0-debian8.2
I've tried to do as much homework/reading as I've had free time for (i.e. not hours and hours) and played around with the install but networking isn't really my forte and I didn't really get anywhere beyond "it kind of works" w/ something functioning (basic file transfer) and somethings that don't (any ib_write/read utilities).
So let's just assume this is all from a standard vanilla install, fresh from installing OFED drivers in addition to running opensmd from the linux server, but not much other configuration. Let's also just worry about connection from one node to the server. I setup both computers on their own subnet (192.170.1.X) and they can ping eachother, as well as ibping the server's port GUID from the windows node. But ib_write_bw fails w/ "Couldn't read remote address Unknown error (0)" on the node, and "Couldn't parse line <9f:08001...> Unable to read to socket/rdam_cm Failed to exchange data between server and clients" on the server.
I *can* transfer files to the SMB share over the link, but it's kind of normal or sub-normal GigE performance, and iperf agrees w/ somewhere between 0.5-1Gbps. Using ifconfig, MTU of ib0 on server is listed as 2044, so I set the "Jumbo Packet" property on the Advanced tab to the same on the node computer.
Any advice on what steps I should take next? I'm assuming I need to get the ib_write/read utilies functioning before I can really tackle any performance optimization.