Hi all,
I'm trying to set up Mellanox-Neutron-Havana-Redhat on OpenStack RDO using SR-IOV. I have sucessfully set up the Nova Compute hosts with openstack-neutron-mellanox, eswitchd and mlnxvif driver. Nova compute hosts are equipped with Mellanox ConnectX-3 hca with SR-IOV enabled firmware (CentOS 6.5, Mellanox OFED 2.1).
[root@n08 ~]# lspci |grep Mellanox
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
01:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.2 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.4 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.5 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.6 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.7 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:01.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
Nova compute works like a charm (VMs are able to ping each other and RDMA, tested with ib_{write,read}_*, is working) but I'm having problems configuring the Network Node. The Network node is equipped with a Mellanox ConnectX hca.
[root@n02 ~]# lspci |grep Mellanox
85:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
Following the instructions at Mellanox-Neutron-Havana-Redhat I have configured it in Paravirtualized mode to make use of DHCP with Linux Bridge Neutron plugin (patched as described in the guide). The eIPoIB interface (eth2) is used by Linux Bridge as the physical interface
[linux_bridge]
physical_interface_mappings = default:eth2
Following the configuration guide I have replaced the original IPoIB daemon (/sbin/ipoibd) with that supplied in the Openstack Havana repo (http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd) but it's not working. The python script exits with following error:
[root@n02 ~]# /sbin/ipoibd log_level=debug
2014-01-24 15:14:29,042 [root] DEBUG: cmdline /sbin/ipoibd log_level=debug
2014-01-24 15:14:29,045 [root] DEBUG: cmd [/bin/logger -p daemon.info -t ipoibd "ipoibd daemon (pid 4943) started -- version 1.2.2302"], rc [0], out_len [0], retry [0]
2014-01-24 15:14:29,045 [root] INFO: ipoibd daemon (pid 4943) started -- version 1.2.2302
2014-01-24 15:14:29,045 [root] DEBUG: all_str all
2014-01-24 15:14:29,046 [root] DEBUG: interval 2
2014-01-24 15:14:29,046 [root] DEBUG: max_iter 0
2014-01-24 15:14:29,046 [root] DEBUG: gc_enable yes
2014-01-24 15:14:29,046 [root] DEBUG: gc_enable yes
2014-01-24 15:14:29,046 [root] DEBUG: log_level 10
2014-01-24 15:14:29,049 [root] DEBUG: cmd [/bin/cat /etc/issue], rc [0], out_len [47], retry [0]
2014-01-24 15:14:29,050 [root] DEBUG: system CentOS
2014-01-24 15:14:29,050 [root] DEBUG: NO MANAGMENT !!!
2014-01-24 15:14:29,050 [root] DEBUG: UNKNOWN UNKNOWN
2014-01-24 15:14:29,050 [root] ERROR: Couldn't find managment tools (e.g. virsh/xenstore)
It is looking for libvirt or Xen client packages that are, obviously, not installed on the Network node.
If I manually configure the eth2 interface (following these instructions: eIPoIB Manual Configuration) everything seems to work fine and VMs are able to get an address from the DHCP on the Network node.
This is my configuration of eIPoIB driver:
[root@n02 ~]# tail /sys/class/net/eth2/eth/{slaves,vifs}
==> /sys/class/net/eth2/eth/slaves <==
ib0.8001.1
ib0.8001.2
==> /sys/class/net/eth2/eth/vifs <==
SLAVE=ib0.8001.1 MAC=fa:16:3e:04:4c:f7 VLAN=1
SLAVE=ib0.8001.2 MAC=fa:16:3e:52:32:68 VLAN=1
where the mac addresses are those of the Neutron DHCP and ROUTER
[root@n02 ~]# ip netns exec qdhcp-570dc494-a876-401c-a80d-6696e2140a5f ip a | grep -A2 'ns-'
12: ns-bf3135d2-4b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:04:4c:f7 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.3/24 brd 10.0.0.255 scope global ns-bf3135d2-4b
inet6 fe80::f816:3eff:fe04:4cf7/64 scope link
valid_lft forever preferred_lft forever
[root@n02 ~]# ip netns exec qrouter-6e95aabe-b64a-4139-b12a-1413444071c1 ip a | grep -A 2 'q[rg]'
16: qr-03eecdb6-1b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:52:32:68 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-03eecdb6-1b
inet6 fe80::f816:3eff:fe52:3268/64 scope link
valid_lft forever preferred_lft forever
18: qg-96f34856-9d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:51:aa:65 brd ff:ff:ff:ff:ff:ff
inet XXX.XXX.XXX.XXX/XX brd XXX.XXX.XXX.XXX scope global qg-96f34856-9d
inet6 fe80::f816:3eff:fe51:aa65/64 scope link
valid_lft forever preferred_lft forever
The eIPoIB interfaces should be automatically configured by the ipoibd server. Is the script in the repository correct? Should I have to download it from another repo?
Thank you very much.
Ale
Message was edited by: Alessandro Federico Adding more infos about eth_ipoib config.