It is embarrassing, there was a problem with the network configuration, so I tested Ethernet not Infiniband
The problem stemed from the fact that before I tried so many things and then we did a clean install of the OS and the newest IB drivers and then it happened we did not remember all steps...