My boss wanted me to set up a link aggregation, otherwise known as NIC Teaming, Ethernet Trunking, and a bunch of other terms, between our VMware ESX 3.5 Standard installation and a 3COM 4200G switch. This gives you the benefit of load sharing bandwidth, as well as failover in case a link goes down. It took me awhile yesterday to figure it out, since I’ve never done much with it, so I figured I’d document it a bit.
You’ll need to change a few settings that appear to work for me in VMware ESX. Add a second virtual NIC under the networking settings. If you already have one virtual switch, you should be good with just adding a second NIC, and making a few setting changes.
VMware_NIC
After you add another NIC, click on the properties button above it, highlight the vSwitch, and choose edit at the bottom. Under the NIC Teaming tab, change the Load Balancing selection to Route based on ip hash. Ensure Network Failover Detection is set to Link Status only and also make sure that Notify Switches and Failback are both set to yes. Click OK, and then close the vSwitch properties window. If everything looks like the following image, that should be all you need to do with VMware.
VMware_Options
Next, login to your switch. In my example, I’m using a 3COM 4200G switch and 3COM happens to call it Link Aggregation. If you’re using Cisco, or some other brand of switch, consult their documentation on how to set this up. The first thing you have to do is set the ports you are going to aggregate to be of a trunk link type, and enable flow control. On the command line, it should be something like this:
# telnet switch
<4200G>system-view
[4200G] interface GigabitEthernet 1/0/6
[4200G-GigabitEthernet1/0/6] port link-type trunk
[4200G-GigabitEthernet1/0/6] flow-control
Now, do the same for the other port(s) you want in the aggregation. For me, I wanted port 6 and 12 of the switch aggregated.
[4200G] interface GigabitEthernet 1/0/12
[4200G-GigabitEthernet1/0/12] port link-type trunk
[4200G-GigabitEthernet1/0/12] flow-control
[4200G-GigabitEthernet1/0/12] save
If you go to the web interface, you can find the same settings here and change them in one shot: Port -> Administration -> Setup
Highlight the port numbers you want to change, and choose the appropriate drop-down box settings above. Make sure you click Apply down below, and then save the configuration from the menu on the left-hand side.
The last step is to configure the actual link aggregation group. On the command line, do the following for each port you want added to the group.
[4200G] link-aggregation group 1 mode manual
[4200G] interface GigabitEthernet 1/0/6
[4200G-GigabitEthernet1/0/6] port link-aggregation group 1
[4200G-GigabitEthernet1/0/6] interface GigabitEthernet 1/0/12
[4200G-GigabitEthernet1/0/12] port link-aggregation group 1
[4200G-GigabitEthernet1/0/12] save
That’s it. Do do the same on the web interface, go to Port -> Link Aggregation -> Create. Then, choose Manual and highlight the ports that you want in your aggregation. Choose Apply and then save the configuration.
My understanding is that you use LACP between switches, or between devices that have LACP protocol enabled. This allows two devices some automagical detections and what not. From what I read, VMware’s NIC Teaming is not LACP, so don’t be confused. This is why I used the manual LACP disabled mode for the 3COM Link Aggregation. If you know better than I do, please leave a comment and correct me.
Once you have this all done, you should be able to go ahead and pull one of your links down, and your systems will continue to work. In my setup, my VMware ESX server was plugged into the network on port 12 of the 4200G switch. So, I added port 6 and then bundled 6 and 12 together. For a test, I unplugged the original connection port 12, and not so much of a burp happened. The network on VMware ESX failed over and used port 6. Also, I can tell that they are sharing the two NICs now, because if you log into the switch you can issue a status command. Both transmit and receive lights are blinking on both NICs as well.
[4200G]display link-aggregation summary
Aggregation Group Type:D -- Dynamic, S -- Static , M -- Manual
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Actor ID: 0x8000, 001a-c160-6e40
AL AL Partner ID Select Unselect Share Master
ID Type Ports Ports Type Port
--------------------------------------------------------------------------------
1 M none 2 0 Shar GigabitEthernet1/0/6
A more detailed view can be gathered from this command:
[4200G] display link-aggregation summary
Remember, you can add as many of these as you want, so long as you have NICs available. And also remember you can use this method to tie switches together as well, in case you don’t have the special fiber connections between the 3COM switches.
