There seems to be a driver that might be able to work with 8.1 client :
http://www.mellanox.com/downloads/WinOF/MLNX_VPI_WinOF-4_95_All_win2012R2_x64.exe
I would go through the archives and install an older version in case that doesn't work. Remember that if you don't have a managed switch you need 1 (or both) ports to run an instance of OpenSM that will manage the connections.
http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_User_Manual_v4.95.pdf
did you find any answer?
Hi,
if you're using IP forwarding you might want to disable LRO on the interfaces, seems like the kernel is pointing to a bad packet flow because of it.
ethtool -K <ethx> lro off
Please try and let us know how it went
As far as I'm aware of there's no official package for Debian 6.0.10, so your best bet is to see if the package for 6.0.9 driver is able to compile using DKMS which is supported in these distributions.
Hi,
I am working on following configuration:
Mellanox Adapter: ConnectX-3 Pro FDR InfiniBand + 40 GigE (MX354A-FCCT)
Mellanox Switch: SX1012
Mellanox Fibre Optic Cable: MC220731V-010(Upto 56Gb/s)
Mellanox OFED driver version: 2.4-1.0.4
I want to trace the on wire contents. I have 10G on-wire analyzer. But my current setup is 40G. So I want to downgrade the adapter and switch to 10G.
Is there a way to downgrade above Mellanox Adapter and Switch to 10G??
Thanks,
Komal.
Hi Komal,
You can change the switch port speed to 10G with:
switch(config)# interface ethernet 1/x speed 10000 force
Hi Eddie,
Thanks for the reply.
Any change required on adapter side as well?
And can I use same cable(upto 56Gbps)?
Hi Komal,
no change is needed from the adapter side.
As for the cable, AFAIK it should work with 10GbE as well
Hi Trevor,
are you still experiencing issues with IPoIB?
if yes - can you please provide the opensm configuration file you are using (if exists) and the opensm command line you used in order to start opensm.
Please also provide the output of:
saquery -g
saquery -m
yes, I am still having this problem. Although at this point I believe my IPoIB config is correct and it seems to be a firmware issue.
my opensm folder in /etc is empty so it looks like I do not have an opensm config file. If you could tell me if my IPoIB config is correct that would be very helpful.
Thanks,
Trevor
Found a similar comment on Google, but the recommended ethtool -K <ethx> gro off.
That did not work, so went through and turned off all offloading in the driver, used ethtool with the following
tso, gro, gso, and lro
With all of these I can use the 10G interface in Virtualbox, have not tried yet to determine exactly which one is the culprit or if I need them all.
> FC = Flow control, assuming it is enabled on the NIC (ethtool -a ethX) it is needed also to be enabled on every switch port on your Arista.
Got it. It is enabled on the NIC
$ ethtool -a p6p1
Pause parameters for p6p1:
Autonegotiate: off
RX: on
TX: on
But not enabled on the switch:
# show interfaces ethernet 14 flowcontrol
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
--------- -------- -------- -------- -------- ------------- -------------
Et14 off off off off 0 0
Do the two devices need to agree on flow control before the NIC will try to pause? ethtool -S shows all the "rx_pause" counters are set to 0 (zero).
> Assuming you're using OFED 2.4 - Can you copy paste dmesg output when the low lever driver loads (mlx4_core) ?
So I have a variety of driver versions that seem to have this issue. I plan to get everything to 2.4 shortly. Here is the dmesg output from one server with 2.4:
$ dmesg | grep mlx
[ 17.650462] mlx4_core: Mellanox ConnectX core driver v2.4-1.0.4 (Apr 24 2015)
[ 17.737351] mlx4_core: Initializing 0000:02:00.0
[ 23.820087] mlx4_core 0000:02:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s
[ 23.908150] mlx4_core 0000:02:00.0: PCIe link width is x4, device supports x4
[ 26.069942] mlx4_core 0000:02:00.0: irq 86 for MSI/MSI-X
[ 26.069947] mlx4_core 0000:02:00.0: irq 87 for MSI/MSI-X
[ 26.069951] mlx4_core 0000:02:00.0: irq 88 for MSI/MSI-X
[ 26.069955] mlx4_core 0000:02:00.0: irq 89 for MSI/MSI-X
[ 26.069958] mlx4_core 0000:02:00.0: irq 90 for MSI/MSI-X
[ 26.069962] mlx4_core 0000:02:00.0: irq 91 for MSI/MSI-X
[ 26.069966] mlx4_core 0000:02:00.0: irq 92 for MSI/MSI-X
[ 26.069970] mlx4_core 0000:02:00.0: irq 93 for MSI/MSI-X
[ 26.069974] mlx4_core 0000:02:00.0: irq 94 for MSI/MSI-X
[ 26.069978] mlx4_core 0000:02:00.0: irq 95 for MSI/MSI-X
[ 26.069982] mlx4_core 0000:02:00.0: irq 96 for MSI/MSI-X
[ 26.069985] mlx4_core 0000:02:00.0: irq 97 for MSI/MSI-X
[ 26.069989] mlx4_core 0000:02:00.0: irq 98 for MSI/MSI-X
[ 26.114201] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.4-1.0.4 (Apr 24 2015)
[ 26.205113] mlx4_en 0000:02:00.0: registered PHC clock
[ 26.296333] mlx4_en 0000:02:00.0: Activating port:1
[ 26.402739] mlx4_en: eth0: Using 96 TX rings
[ 26.494296] mlx4_en: eth0: Using 8 RX rings
[ 26.586020] mlx4_en: eth0: Initializing port
[ 26.679114] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.4-1.0.4 (Apr 24 2015)
[ 26.775863] mlx4_core 0000:02:00.0: mlx4_ib_add: allocated counter index 1 for port 1
[ 28.825562] mlx4_en: p6p1: frag:0 - size:1522 prefix:0 stride:1536
[ 29.545220] mlx4_core 0000:02:00.0: mlx4_ib: Port 1 logical link is up
[ 29.545232] mlx4_en: p6p1: Link Up
Also:
$ ethtool --show-priv-flags p6p1
Private flags for p6p1:
pm_qos_request_low_latency : off
mlx4_rss_xor_hash_function : off
mlx4_flow_steering_ethernet_l2: on
mlx4_flow_steering_ipv4 : on
mlx4_flow_steering_tcp : on
mlx4_flow_steering_udp : on
qcn_disable_32_14_4_e : off
blueflame : on
> and you can also try optimizing the flow steering mode - just create a mlnx.conf file under /etc/modprobe.d/ and put the following line :
>
> options mlx4_core log_num_mgm_entry_size=-7
Seems like for most use cases, this would be a good thing to do. Any reason it isn't the default? I'll give it a try. Thanks again for your help. -- Bud
I have a Mellanox ConnextX2 ethernet adapter and I having some issues receiving packets on it. I am sending simple layer 2 vlan tagged packets. I can see the packets were sent on the wire to the ethernet card, but nothing gets received on the port. Where should I start debugging ? I looked at all the software tools, but no clue so far. Here are the details of my card
06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
Subsystem: Mellanox Technologies Device 0018
08:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
Subsystem: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]
21:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
Subsystem: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]
Any help would be appreciated
Thanks
Srini
Erez, In looking at the VMA github repo
Commits · Mellanox/libvma · GitHub
it seems like there has been some work with ack timers and retransmits between VMA 6.8.3 (which ships with OFED 2.4) and VMA 6.8.4. Should I consider installing a newer version of vma on top of my OFED?
-- Bud
I have a strange issue. I've set ib mode on FreeBSD driver 2.1.6 (same behavior on 2.1.5), and set ip for ib0. The issue that i'm facing is that after a reboot, i need to do ifconfig ib0 down && ifconfig ib0 up to actually have link on ib0. Also i must do this after "mlx4_core0: mlx4_ib: Port 1 logical link is up".
See here:
#dmesg ... mlx4_core0: <mlx4_core> mem 0xfb200000-0xfb2fffff,0xf9000000-0xf97fffff irq 40 at device 0.0 on pci4 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen1.2: <vendor 0x8087> at usbus1 uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1 ugen0.2: <vendor 0x8087> at usbus0 uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0 uhub3: 6 ports with 6 removable, self powered uhub2: 8 ports with 8 removable, self powered ugen0.3: <Winbond Electronics Corp> at usbus0 ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0 kbd0 at ukbd0 mlxen0: Ethernet address: e4:1d:2d:0d:bf:60 mlx4_en: mlx4_core0: Port 1: Using 12 TX rings mlx4_en: mlx4_core0: Port 1: Using 8 RX rings mlx4_en: mlxen0: Using 12 TX rings mlx4_en: mlxen0: Using 8 RX rings mlx4_en: mlxen0: Initializing port mlx4_core0: mlx4_ib_add: allocated counter index 1 for port 1 SMP: AP CPU #1 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC" frequency 2000046460 Hz quality 1000 Trying to mount root from zfs:zroot/ROOT/default []... ib0: Attached to mlx4_0 port 1 ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 igb0: link state changed to UP lagg0: link state changed to UP vlan200: link state changed to UP igb1: link state changed to UP igb2: link state changed to UP igb3: link state changed to UP mlx4_core0: mlx4_ib: Port 1 logical link is up root@fat:~ # ifconfig ib0 ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 4092 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.d.bf.61 inet 10.105.0.230 netmask 0xffffff00 broadcast 10.105.0.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@fat:~ # ping 10.105.0.225 PING 10.105.0.225 (10.105.0.225): 56 data bytes ping: sendto: No route to host ping: sendto: No route to host ^C --- 10.105.0.225 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss root@fat:~ # ifconfig ib0 down && ifconfig ib0 up root@fat:~ # ping 10.105.0.225 PING 10.105.0.225 (10.105.0.225): 56 data bytes 64 bytes from 10.105.0.225: icmp_seq=0 ttl=64 time=0.208 ms ^C --- 10.105.0.225 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.208/0.208/0.208/0.000 ms root@fat:~ # ifconfig ib0 ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 4092 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.d.bf.61 inet 10.105.0.230 netmask 0xffffff00 broadcast 10.105.0.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
What i have in config:
/boot/loader.conf mlx4_core="YES" ibcore_load="YES" mlxen_load="YES" mlx4ib_load="YES" ipoib_load="YES" /etc/sysctl.conf sys.device.mlx4_core0.mlx4_port1=ib /etc/rc.conf ifconfig_ib0="inet 10.105.0.230 netmask 255.255.255.0 up mtu 4092"
Using a ConnectX-3 with an 10Gb Ethernet port I installed Debian 8 (jessie). Booting is fine until I install the Mellanox OFED stack (3.0-1.0.1), at which point I no longer get a DHCP lease at boot. Doing and 'ifdown' followed by an 'ifup' on the interface fixes the issue.
Just an FYI, MLNX_OFED_LINUX-3.0-1.0.1-ubuntu14.04-x86_64 seems to work fine with Ubuntu 14.04.2. -- Bud
I have windows 2012R2 OS on Dell R710.When i connect mellanox nics and cables through a Dell switch with 40G ports, It shows network cable unplugged.
Hello all,
I am trying to install flexboot on two HT26428's. The most recent version of flexboot I can install for my FW version is 3.4.124, but there seem to be 3 different versions (VPI, ethernet, and infiniband). I am using them as infiniband ports, but I was just wondering if I need just the IB package or the IB package and the VPI package?
Thanks,
Trevor
If you're running things in Infiniband mode, is there a subnet manager running on the network?
This kind of sounds like what happens when you run without a subnet manager. If you're not sure, try running the OpenSM service and see if the network cables suddenly start showing up as connected.
