Hi,
starting with your switches, I am not sure what SX5012 is. Maybe you were referring to SX6012?
Connecting things with the HCA you were thinking of using, keep in mind that the InfiniBand technology is currently at FDR rates (56Gb/sec). the HCA you specified is cable of this speed so you probably need an FDR switch to complete the set.
to your specific questions:
1. Are there any other components required to do the above?
Nop. basic network would need NICs, cables and switches. everything else is SW
2. Can the NICs be used to connect servers peer to peer without the switch?
Yes. Back2Back works.
3. Is there a limit on how many NIC can be placed in a server(besides physical)
No. as many PCI-e slots you can have. Keep in mind that the PCI slot you need is a high-speed one. typically PCI-e Gen3 x8 or x16
4. If the cables are not included can point me to the correct cables
Cables - Mellanox Technologies
5. Can the SX5012 Switches be linked together via on board ports?
No on-board ports. for connecting between two or more switches you can use any switch port.
6. Is the management software included with the switch or does this need to be purchased separately?
Mellanox carries Managed switches which include all the network management necessary tools (included in the price of the switch) and some other switches are none-managed (have no on-board CPU) and therefore not cables of manage the network but can be managed using an external Subnet Manager.
I hope i could help. anyway, I recommend you touch base with a reseller or Mellanox sales rep for the final touch ups.
Good luck..