There are a number of things to note when connecting dual-homed servers to paired-ToRs.
- The switchports on the two ToRs have to be configured the same way, when connecting a dual-homed server to the two ToRs.
- The server ports have to be linux-bonded in a particular mode.
- If configuring hosts in ONOS, multiple locations have to be provided.
Configuring Switch Ports
Configuring switch ports is exactly the same as in ONOS 1.10.4 release.
A couple of things to note:
First, dual-homed servers should have the identical configuration on each switch port they connect to on the ToR pairs.
The example below shows that the "vlans" and "ips" configured are the same on both switchports of:205/12 and of:206/29. They are both configured to be access ports in vlan 20, the subnet 10.0.2.0/24 is assigned to these ports, and the gateway-IP is 10.0.2.254/32.
"ports" : {
"of:0000000000000205/12" : {
"interfaces" : [{
"name" : "h3-intf-1",
"ips" : [ "10.0.2.254/24"],
"vlan-untagged": 20
}]
},
"of:0000000000000206/29" : {
"interfaces" : [{
"name" : "h3-intf-2",
"ips" : [ "10.0.2.254/24"],
"vlan-untagged": 20
}]
}
}
It is worth noting the meaning behind the configuration above from a routing perspective. Simply put, by configuring the same subnets on these switchports, the fabric now believes that the entire subnet 10.0.2.0/24 is reachable by BOTH ToR switches of:205 and of:206.
Configuring different vlans, or different subnets, or mismatches like "vlan-untagged" in one switchport and "vlan-tagged" in the corresponding switchport facing the dual-homed server, will result in incorrect behavior.
Second, we need to configure the pair-link ports on both ToR switches to be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets. This is an extra piece of configuration, the need for which will be removed in future releases.
In the example above, a dual-homed server connects to the ToR pair on port 12 on of:205 and port 29 on of:206. Assume that the pair-link between the two ToRs is connected to port 5 of both of:205 and of:206. The config for these switch ports is shown below:
...
"of:0000000000000205/5" : {
"interfaces" : [{
"name" : "205-pair-port",
"ips" : [ "10.0.2.254/24"],
"vlan-tagged": [20]
}]
},
"of:0000000000000206/5" : {
"interfaces" : [{
"name" : "206-pair-port",
"ips" : [ "10.0.2.254/24"],
"vlan-tagged": [20]
}]
}
...
Note that even though the ports of:205/12 and of:206/29 facing the dual-homed server are configured as "vlan-untagged", the same vlan MUST be configured as "vlan-tagged" on the pair-ports. If additional subnets and vlans are configured facing other dual-homed servers, they need to be similarly added to the "ips" and "vlan-tagged" arrays in the pair-port config.
Configuring Servers (Hosts)
Assuming the interfaces we are going to use for bonding are eth1 and eth2.
Stop interfaces
sudo ifdown eth1 sudo ifdown eth2
Modify /etc/network/interfaces
auto bond0 iface bond0 inet dhcp bond-mode balance-xor bond-xmit_hash_policy layer2+3 bond-slaves none auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0
Restart interfaces
sudo ifup bond0 sudo ifup eth1 sudo ifup eth2
Useful command to check bonding status
# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: load balancing (xor) Transmit Hash Policy: layer2+3 (2) MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:1c:42:5b:07:6a Slave queue ID: 0 Slave Interface: eth2 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 00:1c:42:1c:a1:7c Slave queue ID: 0
Configuring Hosts in ONOS
Avoid configuring dual-homed hosts in ONOS 1.11.1. While the configuration itself will work, failiure recovery will not – see Known Issues. Single-homed host configuration is fine.
Configuring dual-homed hosts is similar to single-homed hosts, except that two locations have to be provided. As a result, in ONOS 1.11.1, the "location" field has changed to "locations" and is now an array.
As before, you need the "netcfghostprovider" app to successfully configure hosts.
Host configuration via netcfg is shown below.
"hosts" : {
"00:00:00:00:00:01/-1" : {
"basic": {
"ips": ["10.0.2.22"],
"locations": ["of:0000000000000005/12", "of:0000000000000006/29"]
}
}
}
line 2: Host is identified by mac-address/vlan tuple. In this case vlan -1 means there's no vlan tag.
line 4: IP address of the host. There can be multiple IP addresses.
line 5: The switchports to which this host attaches is identified by device-id/port-number tuple. Single homed hosts would have only one location, and dual-homed hosts should have two locations.
Configuring Static Routes with Dual-Homed Next Hops
We also support static routes with dual-homed next hop in ONOS 1.11.1. The way to configure it is exactly the same as regular single-homed next hop, as described in this page: Static Routes (ONOS 1.10).
onos> route-add <prefix> <next-hop>
ONOS will automatically recognize when the next-hop IP resolves to a dual-homed host and program both switches (the host connects to) accordingly.
The failure recovery mechanism for dual-homed hosts also applies to static routes that point to the host as their next hop.