Thanks wesb!
I upgraded the IB drivers to latest Mellanox 2.1, set the system to "connected" mode and MTU size to 65520. Now, I'm getting performance of ~13.5Gb/s.
I'm getting much less performance than what you noted. Not sure if it is a QDR vs FDR performance difference. I've QDR setup with X5650, 24GB memory, 8x PCI slot.