There are a few known issues with the support for dual-homed servers in ONOS 1.11.1.

Delayed discovery of both connectPoints of a Dual Homed host

In Trellis, host discovery is accomplished by trapping/punting 3 kinds of traffic from the dataplane to the controller - ARP, NDP, and DHCP.

With dual-homed hosts, it is possible that these kinds of traffic are only sent from a single slave-interface on the bond (depending on the hash). In this case, only one of the interfaces of the bond will be detected by ONOS as a connect-point.

For example, in the example shown below, of:206/29 could be detected first, but of:205/12 could be detected much later, depending on when the host hashes  ARP, NDP, or DHCP traffic on to the eth0 slave of the bonded interface bond0.

With discovery of only a single connect-point, traffic flows through the switches as shown above. Outgoing traffic from the server has no issues, and is hashed on both ports and bridged or routed through both switches.

However incoming traffic is only sent through the discovered connect-point. Which means that since of:205/12 has not been discovered, of:205 will use the pair-link to send traffic via of:206/29 to the dual-homed server.

Action Item: This is clearly inefficient and we will implement more active discovery of both connect-points in the next release. 

Workaround in this release: Manually force the discovery of the 2nd connect point from the server using arping through the undiscovered slave of the bond.

In this example, a single arping (-c 1) is sent from the undiscovered interface (-i eth0), with source IP (-S 10.0.5.55) that actually belongs to the bonded interface (bond0), and the destination IP (10.0.5.254) is the gateway IP configured on both of:205/12 and of:206/29.

Once the 2nd connect-point is discovered incoming traffic in of:205 will be re-routed to use of:205/12 to reach the dual-homed server.

Server slave-down not discovered

For a single-homed host, when the host goes down, or the host-interface goes down, the corresponding switch-port to which the host is connected also goes down, leading to a PORT-DOWN notification sent to the controller. The controller typically does not take any re-routing action because there is no other way to reach a singly-homed host anyway.

In the dual-homed case, when a switch port goes down, the controller is notified, and by design, it takes action to re-route incoming traffic via the pair-link over to the active connect-point to the sever. This is correct behavior.

However, we have noticed that when one of the slaves of the bond port goes down (or is brought down) the corresponding switch port does not detect the link-down. As a result, no notification is sent to the controller, and so, no corrective action is undertaken.

For example, if eth0 is taken down (say via 'sudo ifdown eth0'), the interface of:205/12 remains up, and no notification is sent to the controller. As a result, incoming traffic to of:205 meant for the dual homed host will be black-holed.

Action Item: Discuss with Broadcom and EdgeCore why of:205/12 remains up

Workaround in this release: None

Outgoing Traffic dropped in a particular sequence of events 

This issue requires a more detailed description of the sequence of events.

Lets assume that the dual-homed server is trying to reach a destination, and the traffic is hashed out of eth1 in bond. As a result the outgoing traffic is going out of of:206. The incoming traffic in the reverse direction may come back to of:205 or of:206 depending on the hash at the destination switch. In the example below, the incoming traffic is shown to come back to of:206.

Now lets assume that we lose the connect point of:206/29 for some reason. In this case, the server will detect that eth1 link is down and switch the outgoing traffic to eth0 -ie. the hash will only happen over the 1 active slave in the bonded interface.

The return traffic may still come back to of:206, as the destination continues to hash to both switches of:205 and of:206 when reaching the server's subnet. In this case, of:206 will use the pair-link to reach the server via of:205/12. This correct behavior is depicted below.

Next lets assume that the link between eth1 and of:206 is back up and running. But at the same time, lets assume that due to the first issue discovered above (Delayed discovery of 2nd connect-point), this newly activated connect point has not yet been discovered by ONOS – ie. ONOS knows that of:206/29 is UP, but it has not discovered that that the dual-homed server is reachable on that port.

At this point, the incoming traffic at of:206 is still routed via the pair link to of:205. This is not an issue.

However, the bond port at the server has discovered that the slave eth1 is UP and active, and so it moves the outgoing traffic back to the original hash out of eth1.

The issue is that, for some reason, the outgoing traffic is now dropped at the switch of:206. This is depicted below.

The flows and groups we expect to see are all normal and should work. We cannot replicate this behavior with emulated software switches. Therefore we suspect there is some issue in OF-DPA resulting in the packet drops.

Action Item: Discuss with Broadcom

Workaround in this release: For some strange reason, normal forwarding behavior resumes if we force the discovery of the 2nd connect-point in the switch.

With this woraround, traffic flow will return to the normal behavior shown in the first picture in this section.

Dual-homed hosts should not be configured

Currently in ONOS, configured hosts are not updated when the connectPoint is lost. This is not a problem with single-homed hosts because there is no other way to reach them anyway if their connectPoint goes down.  But in dual-homed scenarios, the controller should take corrective action if one of the connectPoints go down – the trigger for this event does not happen when the dual-homed host's connect points are configured (not discovered).

Action Item: We will fix this in a future release.

Workaround in this release: Avoid configuring dual-homed hosts. Let ONOS discover the two connect-points via ARP/NDP/DHCP.