One more thing to pay attention to Jon,
if you take two machines and connect then one to each other using both ports of each HCA AND have a switch in between this is all consider as "one IB fabric/network"
if you take two servers and connect them one to each other using both ports of each HCA (port 0 to port 0 and port 1 to port 1) *without* a switch in between, you got yourself TWO separate networks. those two networks will live in parallel but will not be as one like in the previous example.
i think that the source of your problems is right there. to prove my point, disconnect one of the two cables or put a switch in between.
Obviously, make sure each interface (ib0, ib1) has different IP and subnet (this is IP and routing. has nothing 2do with IB). or alternatively if you need both ports, you can use bonding with one address.
works??