During my latest Nutanix cluster installations, I ran into a problem that drove me crazy.
In two cases, on different hardware, I found myself in a rather unusual situation that prevented me from going ahead with the cluster configuration.
Basically, once Foundation was complete, the servers’ NICs were detected with different speeds from one server to the next, even though they were all physically identical and connected to the same switches (with identical adapters and cables).
I should point out that in both cases the NICs were cards with 10/25 Gbps ports, but connected to switches with 10Gbps ports only.
The anomalous behavior was the following:
In Prism the NICs were shown with different speeds, some at 25G and others at 10G:
In practice, Nutanix detected some of the NICs at 10G and others at 25G, even though they were all connected to 10G switch ports. On top of that, the split between 10G and 25G ports wasn’t the same across the different servers, which prevented me from properly adding the NICs to the defined bridges. In fact, before it lets you add NICs to a bridge, Nutanix checks that they are all connected at the same speed.
As a first attempt we tried forcing the port speed to 10G on the switch side, disabling auto-negotiation. However, this didn’t change anything on the Nutanix side.
So I did some digging in the Nutanix documentation and found this command, which forces the speed on the AHV side:
ethtool -s ethX speed 10000 duplex full autoneg off
The command must be run in AHV, forcing the speed to 10000 for every NIC involved (to be safe, I forced it on all the 25/10 NICs).
This way, after a few minutes, Prism correctly detected all the NICs at 10G, allowing me to manage them properly in the various bridges.