Guys,
I've hit a dead-end on something that _should_ work fine as far as I can see. A pci passthrough vf from a Connectx-3 card to a Windows 7 Xenserver guest, with the latest Mellanox drivers installed, refuses to start the the VPI drifver with "code 10".
The background...
The card is only being used as an Ethernet adapter. When I configure it as a single device (no SR-IOV) I can add it to a bridge and use the Citrix provided paravirt drivers, in a Windows 7 HVM guest, to get network connectivity. Trouble is performance is appalling. It maxes out at about 30MB/sec despite the underlying interface being perfectly capable of 10Gb. This seems to be a common problem with the Xen drivers in my experience, so I started investigating SR-IOV and passing through a VF 10Gb device to get some actual performance.
The setup...
HP SL250 with a ConnectX-3 VPI installed (Flex LOM).
Machine is now running Xenserver 6.2 SP1 (kernel 3 2.6.32.43-0.4.1.xs1.8.0.847.170785xen)
Mellanox provided OFED 2.1 drivers installed for XS6.2SP1, working fine. Non SR-IOV interfaces can be configured for 10Gb ethernet and are functional.
SR-IOV and VT-D enabled in SL250 BIOS
intel_iommu=on set in xenserver boot options for Dom0
root=LABEL=root-abxxdeyw ro xencons=hvc console=hvc0 console=tty0 intel_iommu=on
PCI device visible with SR-IOV
# lspci -s 07:00.0 -v
07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Hewlett-Packard Company Device 1777
Flags: fast devsel, IRQ 40
Memory at d9f00000 (64-bit, non-prefetchable) [size=1M]
Memory at bc000000 (64-bit, prefetchable) [size=32M]
Capabilities: [40] Power Management version 3
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable- Count=128 Masked-
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [148] Device Serial Number 24-be-05-ff-ff-a3-0b-f0
Capabilities: [108] Single Root I/O Virtualization (SR-IOV)
Capabilities: [154] Advanced Error Reporting
Capabilities: [18c] #19
Kernel modules: mlx4_core
Updated card to latest firmware available
| Versions: | Current | Available | |
| FW | 2.30.8000 | N/A | |
| CLP | 8025 | N/A | |
| PXE | 3.4.0151 | N/A |
mlx4_core loaded with SR-IOV options, produces the two vfs I'm looking for
# modprobe mlx4_core num_vfs=2 probe_vf=2 port_type_array=2,2
# dmesg | grep mlx
[ 966.628831] mlx4_core: Mellanox ConnectX core driver v1.1 (Feb 24 2014)
[ 966.628837] mlx4_core: Initializing 0000:07:00.0
[ 966.628913] mlx4_core 0000:07:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
[ 966.628969] mlx4_core 0000:07:00.0: setting latency timer to 64
[ 966.629141] mlx4_core 0000:07:00.0: Enabling SR-IOV with 2 VFs
[ 966.755960] mlx4_core 0000:07:00.0: Running in master mode
[ 971.413000] mlx4_core 0000:07:00.0: get owner: 7ff0
[ 971.414222] mlx4_core 0000:07:00.0: irq 1236 (284) for MSI/MSI-X
[ 971.414223] mlx4_core 0000:07:00.0: get owner: 7ff0
[ 971.415376] mlx4_core 0000:07:00.0: irq 1235 (283) for MSI/MSI-X
[ 971.415377] mlx4_core 0000:07:00.0: get owner: 7ff0
[ 971.416530] mlx4_core 0000:07:00.0: irq 1234 (282) for MSI/MSI-X
[ 971.416532] mlx4_core 0000:07:00.0: get owner: 7ff0
[ 971.417686] mlx4_core 0000:07:00.0: irq 1233 (281) for MSI/MSI-X
[ 971.439340] mlx4_core: Initializing 0000:07:00.1
[ 971.439356] mlx4_core 0000:07:00.1: enabling device (0000 -> 0002)
[ 971.439989] mlx4_core 0000:07:00.1: setting latency timer to 64
[ 971.440301] mlx4_core 0000:07:00.1: Detected virtual function - running in slave mode
[ 971.440341] mlx4_core 0000:07:00.1: Sending reset
[ 971.440421] mlx4_core 0000:07:00.0: Received reset from slave:1
[ 971.440652] mlx4_core 0000:07:00.1: Sending vhcr0
[ 971.441714] mlx4_core 0000:07:00.1: HCA minimum page size:512
[ 971.442439] mlx4_core 0000:07:00.1: Timestamping is not supported in slave mode.
[ 971.443749] mlx4_core 0000:07:00.1: get owner: 7ff0
[ 971.444871] mlx4_core 0000:07:00.1: irq 1232 (280) for MSI/MSI-X
[ 971.444875] mlx4_core 0000:07:00.1: get owner: 7ff0
[ 971.445838] mlx4_core 0000:07:00.1: irq 1231 (279) for MSI/MSI-X
[ 971.445842] mlx4_core 0000:07:00.1: get owner: 7ff0
[ 971.446830] mlx4_core 0000:07:00.1: irq 1230 (278) for MSI/MSI-X
[ 971.446832] mlx4_core 0000:07:00.1: get owner: 7ff0
[ 971.447810] mlx4_core 0000:07:00.1: irq 1229 (277) for MSI/MSI-X
[ 971.486498] mlx4_core: Initializing 0000:07:00.2
[ 971.486515] mlx4_core 0000:07:00.2: enabling device (0000 -> 0002)
[ 971.487161] mlx4_core 0000:07:00.2: setting latency timer to 64
[ 971.487483] mlx4_core 0000:07:00.2: Detected virtual function - running in slave mode
[ 971.487534] mlx4_core 0000:07:00.2: Sending reset
[ 971.487573] mlx4_core 0000:07:00.0: Received reset from slave:2
[ 971.487786] mlx4_core 0000:07:00.2: Sending vhcr0
[ 971.488971] mlx4_core 0000:07:00.2: HCA minimum page size:512
[ 971.489771] mlx4_core 0000:07:00.2: Timestamping is not supported in slave mode.
[ 971.491091] mlx4_core 0000:07:00.2: get owner: 7ff0
[ 971.492210] mlx4_core 0000:07:00.2: irq 1228 (276) for MSI/MSI-X
[ 971.492213] mlx4_core 0000:07:00.2: get owner: 7ff0
[ 971.493179] mlx4_core 0000:07:00.2: irq 1227 (275) for MSI/MSI-X
[ 971.493182] mlx4_core 0000:07:00.2: get owner: 7ff0
[ 971.494148] mlx4_core 0000:07:00.2: irq 1226 (274) for MSI/MSI-X
[ 971.494151] mlx4_core 0000:07:00.2: get owner: 7ff0
[ 971.495111] mlx4_core 0000:07:00.2: irq 1225 (273) for MSI/MSI-X
[ 971.543473] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1-1.0.6 (Feb 24 2014)
[ 971.543722] mlx4_en 0000:07:00.0: Activating port:1
[ 971.556498] mlx4_en: eth2: Using 8 TX rings
[ 971.556501] mlx4_en: eth2: Using 4 RX rings
[ 971.556675] mlx4_en: eth2: Initializing port
[ 971.556860] mlx4_en 0000:07:00.0: Activating port:2
[ 971.566569] mlx4_en: eth3: Using 8 TX rings
[ 971.566572] mlx4_en: eth3: Using 4 RX rings
[ 971.566792] mlx4_en: eth3: Initializing port
[ 971.567469] mlx4_en 0000:07:00.1: Activating port:1
[ 971.567496] mlx4_en: 0000:07:00.1: Port 1: Assigned random MAC address 9a:76:dd:d2:d2:68
[ 971.586369] mlx4_en: eth4: Using 8 TX rings
[ 971.586372] mlx4_en: eth4: Using 4 RX rings
[ 971.586622] mlx4_en: eth4: Initializing port
[ 971.586736] mlx4_en 0000:07:00.1: Activating port:2
[ 971.586762] mlx4_en: 0000:07:00.1: Port 2: Assigned random MAC address 02:f4:a3:4d:56:66
[ 971.600273] mlx4_en: eth2: Link Up
[ 971.602715] mlx4_en: eth5: Using 8 TX rings
[ 971.602718] mlx4_en: eth5: Using 4 RX rings
[ 971.602948] mlx4_en: eth5: Initializing port
[ 971.604033] mlx4_en 0000:07:00.2: Activating port:1
[ 971.604080] mlx4_en: 0000:07:00.2: Port 1: Assigned random MAC address 02:7e:7b:9a:a3:2d
[ 971.623501] mlx4_en: eth6: Using 8 TX rings
[ 971.623505] mlx4_en: eth6: Using 4 RX rings
[ 971.623756] mlx4_en: eth6: Initializing port
[ 971.623875] mlx4_en 0000:07:00.2: Activating port:2
[ 971.623914] mlx4_en: 0000:07:00.2: Port 2: Assigned random MAC address 1a:b4:9d:d5:b0:94
[ 971.640530] mlx4_en: eth4: Using 8 TX rings
[ 971.640533] mlx4_en: eth4: Using 4 RX rings
[ 971.640802] mlx4_en: eth4: Initializing port
# lspci -s 07:00
07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
07:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
07:00.2 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
# lspci -s 07:00.1 -v
07:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
Subsystem: Hewlett-Packard Company Device 61b0
Flags: bus master, fast devsel, latency 0
[virtual] Memory at 9c000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
Capabilities: [40] #00 [0000]
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core
Create Xen host with Mellanox VF passed through as pci device
# xe vm-param-set uuid=1fc56292-f78f-ea62-ea70-03fcc6060dbc other-config:pci=0/0000:07:00.1
Build windows 7 guest, fine. Device is visible as an unknown network interface.
Install Mellanox Windows VPI drivers. Installs fine. Device recognised as Mellanox VPI Ethernet device.
The problem...
However... The windows 7 driver never starts. Give the reason "Code 10". Which doesn't tell me anything.
I've also tried a Windows 2008r2 guest with the same setup and drivers. Same result.
I've also tried a HVM Centos6.4 VM with the same device passed through. (Centos 6.4 doesn't support paravirt for PCI passthrough of SR-IOV devices I don't beleive). That gave me some more diagnostic info. The mlx4_core module generates:
mlx4_core: Initializing 0000:00:05.0
mlx4_core 0000:00:05.0: Detected virtual function - running in slave mode
mlx4_core 0000:00:05.0: Sending reset
mlx4_core 0000:00:05.0: Sending vhcr0
mlx4_core 0000:00:05.0: HCA minimum page size:512
mlx4_core 0000:00:05.0: irq 48 for MSI/MSI-X
mlx4_core 0000:00:05.0: irq 49 for MSI/MSI-X
mlx4_core 0000:00:05.0: failed execution of VHCR_POST commandopcode 0x31
mlx4_core 0000:00:05.0: NOP command failed to generate MSI-X interrupt IRQ 49).
mlx4_core 0000:00:05.0: Trying again without MSI-X.
mlx4_core: probe of 0000:00:05.0 failed with error -16
Which looks exactly like this unanswered question on the Xen project Mailing List.
I presume the Windows 7/2k8 driver isn't starting for the same reasons..
Has anyone got VF PCI passthrough working on Xen?
If the mellanox card has the SR-IOV functionality, are there any platforms on which that's actually been demonstrated?
I'm this close to having half-decent network performance available to these virutal machine, but falling at the final hurdle. Any help appreciated.
Leon