Quantcast
Channel: Mellanox Interconnect Community: Message List
Viewing all articles
Browse latest Browse all 6278

Re: How to setup IPoIB correctly?

$
0
0

I have been kept digging, and here is what I have observed:

 

I just did a

 

[root@sc2u0n0 ~]# lsinitrd /boot/initramfs-3.10.0-327.28.2.el7.x86_64.img|less

[...]

drwxr-xr-x  5 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core

-rw-r--r--  1 root    root        21149 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_addr.ko

-rw-r--r--  1 root    root        82669 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_cm.ko

-rw-r--r--  1 root    root      159989 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_core.ko

-rw-r--r--  1 root    root        77565 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_mad.ko

-rw-r--r--  1 root    root        51765 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_sa.ko

-rw-r--r--  1 root    root        33549 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_ucm.ko

-rw-r--r--  1 root    root        35829 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_umad.ko

-rw-r--r--  1 root    root        87141 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/ib_uverbs.ko

-rw-r--r--  1 root    root        68989 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/iw_cm.ko

-rw-r--r--  1 root    root        75765 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/rdma_cm.ko

-rw-r--r--  1 root    root        37637 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/core/rdma_ucm.ko

drwxr-xr-x  12 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb3

-rw-r--r--  1 root    root      233861 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb4

-rw-r--r--  1 root    root      286621 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ipath

-rw-r--r--  1 root    root      439149 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ipath/ib_ipath.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx4

-rw-r--r--  1 root    root      250733 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx5

-rw-r--r--  1 root    root      192053 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mthca

-rw-r--r--  1 root    root      221773 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/nes

-rw-r--r--  1 root    root      274037 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/nes/iw_nes.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ocrdma

-rw-r--r--  1 root    root      131165 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/qib

-rw-r--r--  1 root    root      600661 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/qib/ib_qib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/usnic

-rw-r--r--  1 root    root      135229 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko

drwxr-xr-x  7 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/ipoib

-rw-r--r--  1 root    root      161509 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/iser

-rw-r--r--  1 root    root        85917 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/iser/ib_iser.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/isert

-rw-r--r--  1 root    root        91245 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/isert/ib_isert.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srp

-rw-r--r--  1 root    root        85757 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srp/ib_srp.ko

drwxr-xr-x  2 root    root            0 Aug  7 15:31 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srpt

-rw-r--r--  1 root    root        92813 Aug  3 04:52 usr/lib/modules/3.10.0-327.28.2.el7.x86_64/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko

[...]

 

Yes. I am new to IB, but I am very sure the above is from the distro.  The mlnxofedinstall perl script doesn't run dracut -f, as proved by the following:

 

 

[root@sc2u0n0 mlnx_ofed]# grep dracut mlnxofedinstall

[root@sc2u0n0 mlnx_ofed]#

 

Thus, even the Mellanox drivers/kernel modules are loaded with its openibd running, I suspect that both the inbox and Mellanox ones are loaded.  The outcome under such a circumstance is not something I have much experience - since I am new to IB. But my experience with other drivers tells me that it's not going to be good.  If this is indeed the case, then there is a bug in the mlnxofedinstall perl script (missing a critical step!) that should be fixed ASAP.

 

I have started thinking whether I should use the uninstall.sh that comes with MLNX_OFED to uninstall it, and then just use the inbox IB modules/drivers.


Viewing all articles
Browse latest Browse all 6278

Trending Articles