He is saying Dell switch, I don't think that it is InfiniBand.
Few questions:
Did you install WinOF driver?
Could you make sure that the port is configured with Ethernet (if this is the case)?
Try to replace cable or switch port to make sure that there is no problem with the HW.
Thanks,
Ophir.
I have been trying to set up a new cluster using an 18 port MSX6015 switch and 9 systems with connectx-3 cards/built in IB:
08:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
and 8X
02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
All cables are new FDR cables.
They are running a newish (3.17.3) kernel, although I have 4.0.3 ready to go when I can reboot them.
They are running debian with the current OFED, and even opensm 3.3.18
All firmware has been brought up to the most recent posted one.
I have set
# Force PortInfo:LinkSpeedExtEnabled on ports
# If 0, don't modify PortInfo:LinkSpeedExtEnabled on port
# Otherwise, use value for PortInfo:LinkSpeedExtEnabled on port
# Values are (MgtWG RefID #4722)
# 1: 14.0625 Gbps
# 2: 25.78125 Gbps
# 3: 14.0625 Gbps or 25.78125 Gbps
# 30: Disable extended link speeds
# Default 31: set to PortInfo:LinkSpeedExtSupported
#force_link_speed_ext 31
force_link_speed_ext 1
(have also tried 31)
# FDR10 on ports on devices that support FDR10
# Values are:
# 0: don't use fdr10 (no MLNX ExtendedPortInfo MADs)
# Default 1: enable fdr10 when supported
# 2: disable fdr10 when supported
fdr10 1
in the opensm.conf
I still get a connection of 40 and iblinkinfo saye:
4 10[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 3 1[ ] "MT25408 ConnectX Mellanox Technologies" ( Could be FDR10)
or
CA: MT25408 ConnectX Mellanox Technologies:
0x002590fffff7b3c5 33 1[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 4 13[ ] "SwitchX - Mellanox Technologies" ( Could be FDR10)
for all live ports.
Does anyone have any ideas of what i could be doing wrong?
Thanks.
Ok I have an update -
I found another posting on how to modify the firmware to be able to run back to back 56Gig IB. I did this with all the cards and the switch. I set tboth fdr10 and 56gig on in the init files and rebuilt firmware. After reboots of everything one of our models is running 25 minutes faster than before (was 3. hrs). However, ibportstate and ibstat still report tha everything is running at 40. Also iblinkinfo reports that it all "could be 56Gig"
Any ideas? Is it working and the reporting is just bad?
Message was edited by: David Warren 6/25/2015
First, I would make sure you are using supported adapters, cables, and SFPs for both the Mellanox cards and the Dell switch. There are known issues when mixing certain types of SFPs and cables, and with certain kinds of Twinax.
If all the hardware is supported, please make sure the WinOF driver is installed and that the Mellanox port is configured to Ethernet (for VPI adapters).
I would then try directly connecting the NICs to eachother to bypass the Dell switch. Try running some IO using direct connect to verify your cards and cables are working. If this works --> either a compatibility issue with SFPs/cables and Dell switch, or the switch is configured incorrectly.
Build warnings/errors in sles12 happened in %install.
Are these problems known? Already a patch available?
Mlnx-ofa_kernel 3.0 was built fine in sles11sp3.
1)
extracting debug info from /usr/src/packages/BUILDROOT/mlnx-ofa_kernel-3.0-OFED.3.0.0.3.7.1.g3c1d583sgi712r1.sles12.x86_64/usr/src/ofa_kernel/default/compat/mlx_compat.ko
*** WARNING: identical binaries are copied, not linked:
/usr/src/ofa_kernel/default/compat/mlx_compat.ko
and /lib/modules/3.12.39-47-default/updates/compat/mlx_compat.ko
2)
calling /usr/lib/rpm/brp-suse.d/brp-55-boot-scripts
E: File `openibd' without LSB header found in /usr/src/packages/BUILDROOT/mlnx-ofa_kernel-3.0-OFED.3.0.0.3.7.1.g3c1d583sgi712r1.sles12.x86_64/etc/init.d/
ERROR: found one or more broken init or boot scripts, please fix them.
For more information about LSB headers please read the manual
page of of insserv by executing the command `man 8 insserv'.
If you don't understand this, mailto=werner@suse.de
3)
RPM build errors:
bogus date in %changelog: Wed Sep 8 2008 Vladimir Sokolovsky <vlad@mellanox.co.il>
Hi
IBMPowerKVM server, which has a Connectx-3 Pro card [FW: 2.33.5106] , I tried loading mlx4_core as below:
modprobe mlx4_core port_type_array=2,2 num_vfs=8,0,0 probe_vf=8,0,0 log_num_mgm_entry_size=-1
Which should ideally have created 8 VFs probed on 1st port. But I am seeing below under ip link o/p:
ip link show enP5p1s0
62: enP5p1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether f4:52:14:3b:22:10 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 5 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 6 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 7 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
ip link show enP5p1s0d1
63: enP5p1s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether f4:52:14:3b:22:11 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 5 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 6 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 7 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
Where enP5p1s0d1 and enP5p1s0 are the interface names of dual port mellanox connectx-3 Pro.
I have read instructions @ HowTo Configure SR-IOV VFs on Different ConnectX-3 Ports
I am seeing VF for both ports and none of them seems to be dummy.. as I understand from HowTo Configure SR-IOV VFs on Different ConnectX-3 Ports , dummy vf ports will be assigned with MAC and vlan to 0xff(s).
I could open a bug from official channel if you think its a genuine issue.
I did get resolution. The ports of connect3-x pro cards were not connected to the switch/cross connected. By connecting the ports to external switch I was able to get the intra-host communication among guests to work. Although I dont see a prerequisite which cites this requirement, the exercise which I performed confirmed it.
Hi all,
I am trying to configure VXLAN by following the below link.
HowTo Configure VXLAN for ConnectX-3 Pro (Linux Bridge)
I am done with the host configuratiom. I had a question regarding veth0. How to bring up VM with veth0 attached to it?
Please if any one can help me it will be great...
Hi all! Никто не сталкивался с сабжевым вопросом? Проблема в том: втыкаю MC2209124-005 в Cisco Catalyst 2960х (стекованная)
1 52 WS-C2960X-48FPD-L 15.0(2)EX5 C2960X-UNIVERSALK9-M
2 52 WS-C2960X-48FPD-L 15.0(2)EX5 C2960X-UNIVERSALK9-M
и не работает, хотя вендер в cisco виден, видны регистры.
Втыкаю в тоже место Cisco SFP-H10GB-CU3M (не полный аналог, короче на 2 метра, но такой у меня под рукой) - и все работает.
Втыкаю тот же MC2209124-005 в Cisco Catalyst 3850 (пока не стековая) 1 56 WS-C3850-48T 03.06.00E cat3k_caa-universalk9
и все работает.
Понятно что дело в IOS-е, однако, у меня 2 вопроса:
1. Что делать:
1.1 заменять на кабель cisco?
1.2. вводить какие-то команды полностью недокументированные в cisco?
1.3. перешивать кабель mellanox под cisco?
2. Это не из-за того, что коммутаторы находятся в стеке? Поясняю: вторую Cisco Catalyst 3850 я пока думаю покупать для стека, вдруг эффект повторится?
Hi Michael,
Do you have issues with a Mellanox cables between 2 Cisco switches?
I think you should contact Cisco for this issue.
Also I don't see the PN MC2209124-005 at http://www.mellanox.com/page/cables?mtag=cable_overview
Are you sure the PN is correct?
Please reply asap.
Thanks,
Rupesh
Hi! Sorry, my English is bad. I'm connect Cisco switch's (2960x and 3850 - see my f'st message) and Mellanox netcard (Mellanox ConnectX®-3 Pro EN network interface card, 10GbE, dual port SFP+, PCIe3.0 x8 8GT/s, tall bracket, RoHS R6 - PN MCX312A-XCBT).
So, also sorry! It's my mistake: PN MS3309124-005
On Russian:
Здравствуйте! Покорнейше прошу меня извинить за мое плохое владение английским языком. Я соединял коммутаторы Cisco (модели 2960Х и 3850 - смотрите мое первое сообщение) и сетевую карту Mellanox (Mellanox ConnectX®-3 Pro EN network interface card, 10GbE, dual port SFP+, PCIe3.0 x8 8GT/s, tall bracket, RoHS R6 - PN MCX312A-XCBT).
Извините также за мою ошибку: PN MS3309124-005
I see vxlan offload parameter for the module mlx4_core but I am unable to see the number of packets offloaded for vxlan. the tput is also very less
~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:01:00.00 tg3 Up 1000Mbps Full f8:bc:12:4c:29:70 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1 0000:01:00.01 tg3 Up 1000Mbps Full f8:bc:12:4c:29:71 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1000402 0000:82:00.00 mlx4_en Down 0Mbps Half 00:02:c9:b5:f0:91 1500 Mellanox Technologies MT27520 Family
vmnic2 0000:02:00.00 tg3 Down 0Mbps Half f8:bc:12:4c:29:72 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3 0000:02:00.01 tg3 Down 0Mbps Half f8:bc:12:4c:29:73 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4 0000:82:00.00 mlx4_en Up 40000Mbps Full 00:02:c9:b5:f0:90 1600 Mellanox Technologies MT27520 Family
~ # esxcfg-module -g mlx4_core
mlx4_core enabled = 1 options = 'enable_vxlan_offloads=1'
~ # esxcli network nic get -n vmnic4
Advertised Auto Negotiation: true
Advertised Link Modes: 1000baseT/Full, 10000baseT/Full, 40000baseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 20
Driver Info:
Bus Info: 0000:82:00.0
Driver: mlx4_en
Firmware Version: 2.33.5100
Version: 1.9.9.4 (Sep-02-2014)
Link Detected: false
Link Status: Down
Name: vmnic4
PHYAddress: 0
Pause Autonegotiate: true
Pause RX: true
Pause TX: true
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver: internal
Wakeon: None
I am trying to get Mellanox QSFP cables (MC220731V / Active Fiber 40Gb/s, see attached picture) to work between a variety of vendor switches. However, I cannot get it to work on our Cisco Nexus 6004, but I can get the cable to work on Cisco Nexus 3172s and Arista switches just fine.
I have enabled support for 3rd party transceivers by enabling them with "service unsupported-transciever" and have other 3rd party cables that work just fine on it like a TrippLite Twinax cable.
When I plug the cable in to the 6004 I get the following message:
6004# 2015 Jun 26 13:27:12 DCO-6004-1 %ETHPORT-3-IF_NON_CISCO_TRANSCEIVER: Non-Cisco transceiver on interface Ethernet6/1 is detected
Here are the additional details about the cable:
6004# sh interface transceiver details
Ethernet6/1
transceiver is present
type is QSFP Unknown Type-(unknown)
name is Mellanox
part number is MC220731V-003
revision is A1
serial number is DC501401368
nominal bitrate is 14100 MBit/sec
Link length supported for copper is 3 m
cisco id is –
cisco extended id number is 0
DOM is not supported
We are setting up a test environment where we will be running equipment off of Cisco and Mellanox switches. We would prefer to use and run only one set of cables between everything. I am aware that Cisco does not support 3rd party cables/transceivers, but neither does Mellanox. So we have to choose somebody and Mellanox is about 1/3 cheaper with quotes on their cables.
Any help or advice would be greatly appreciated.
Did you try ProLabs cables?
http://www.prolabs.com/data/downloads/5275.pdf
They are based on Mellanox Cables but open for Cisco.
steveburke can help here.
Ophir.
Thanks for the suggestion, I have not heard of ProLabs, but have reached
out to them to see if they have fiber AOC cables compatible with Cisco and
Mellanox.
From what others have suggested, it does not seem like the problem can be
fixed with a configuration change on the Cisco switch, but rather just
finding a cable that will work with it. Seems odd to me that the Mellanox
cable works on a Cisco Nexus 3172 and not the 6004, but I guess that's how
it goes sometimes with optics.
Greetings,
I'd like to ask, will this work?
http://www.ebay.com/itm/671798-001-HP-10GB-MELLANOX-CON...
+
http://h20565.www2.hp.com/hpsc/swd/public/detail?swItem...
Two of these adapters and Asus Maximus VII Formula with Windows 8.1 and VII Impact Windows Server 2012 R2?
I'd like to use my server as better & safer NAS. It's running raid 1 + 0 [360 MB/s both ways] and I'm limited by Intel 1 gbps nic.
Please let me know,
kind regards
uplink
I have SX1012 in a test setup. Eth1/1 is connected to Connect-X3 56GbE NIC on Intel Xeon GPU server. I am using a split cable on eth1/2 and it is configure to have 4x10GbE, to which a custom FPGA board send data to the switch. However the LED corresponding to Eth1/2 blinks red, and so is the LED labelled "!" - is this a problem? I do see the packets send by the FPGA board on my server though.
Thanks!
Any one knows this, tcpdump is not able to dump the broadcast pkts.
Multicast, Unicast, Unknown Unicast are good to be dumped by tcpdump.
Plus, ethtool -t ethx, self-test failed.
Card infor:
driver: mlx4_en
version: 2.4-1.0.0.1 (Jun 29 2015)
firmware-version: 2.34.5000
bus-info: 0000:06:00.0
Don't know why the self-diagnose is failed.
The test result is FAIL
The test extra info:
Interrupt Test -5
Link Test -12
Speed Test -12
Register Test 0
Loopback Test 0
Sending 100 Pkts
>>> sendp(pkt,iface='eth3',count=1000)
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Sent 1000 packets.
rx_broadcast_packets: 1000
tx_broadcast_packets: 0
vport_rx_broadcast_packets: 0
vport_rx_broadcast_bytes: 0
vport_tx_broadcast_packets: 0
vport_tx_broadcast_bytes: 0
rx_broadcast_packets is 1000, but tcpdump is not able to dump the pkts.
Server info:
Linux 2.6.32-279.el6.x86_64 x86_64
Centos6.3
The problem is that SLES 12 requires all init scripts under /etc/init.d to be LSB (Linux Standard Basic)
compliant, and openibd does not.
Hi Ramesh,
The Ethernet port blinking Red (it's Orange Actually) - means that there are physical erros on the link.
The LED labeled "!" which is the "Bad Port Led" indicates that a symbol error is detected on one of the ports.
http://www.mellanox.com/related-docs/user_manuals/1U_HW_UM_SX10XX_SX1X00.pdf