Hi inbusiness,
The way of VLAN configuration for IPoIB on vSphere 4.x/5.x didn't change.
I suspect that you may have the wrong configuration on 5.x.
Can you please provide more information.
- Subnet Manager PKEY configuration file
- vmnic configuration (esxcfg-vswitch -l on ESX, etc..)