Thanks, the flint -mac hints were spot on. I can now see the two eth* devices in ifconfig, but when doing:
machine1$ ifconfig eth0 192.192.168.1.1
machine2$ ifconfig eth0 192.192.168.1.2
this comes up in dmesg:
mlx4_en 0000:06:00.0: Activating port:1
mlx4_en: eth0: Using 16 TX rings
mlx4_en: eth0: Using 16 RX rings
mlx4_en: eth0: Initializing port
ADDRCONF(NETDEV_UP): eth0: link is not ready
And ping doesn't work between the machines.
When setting port types to ib and using the ib0 devices, ping works, netperf tests work, etc (I started an opensm service on one machine). I've tried "ifconfig eth0 down" then back up on the machines with no success. I checked with ibdev2netdev that I was using the right eth device (ib0 <==> eth0).
I did "yum groupinstall Infiniband\ Support" and then configured things in /etc/rdma but got the same result ("link is not ready").
Is there anything I missed ? The MLNX_OFED manual doesn't list anything besides ifconfig under "4.1.9 A detailed example".