Thanks for the link, I didn't notice MLNX_OFED 2.0 was already out. I went through the tuning steps from the MLNX_OFED-1.5.3 release, and got the numbers above (11.4Gbps for plain IP forwarding, 6Gbps for forwarding + a netfilter kernel hook that ACCEPTs every packet).
I noticed a new mlnx_affinity script was introduced in MLNX_OFED 2, if that does something different than what I can do via /proc/irq/x/smp_affinity then I will give it a try.
Note that I specified that I can't use mlnx_en since I have an IB-only switch. My question remains: are hardware queues available when running in IPoIB mode ? The 2.0 release notes mention "Flow Steering for Ethernet and InfiniBand" was introduced. Should I take this to mean 'yes' ?
Thanks
Bogdan