ONOS 1.11.1 comes with support for dual external routers connected to dual-ToRs in various configurations.

As mentioned before, the Trellis vRouter is NOT a software router - it is not a router in a VM. Previously (in ONOS 1.10.4), the external router viewed the entire Trellis fabric as 1 big router.

In ONOS 1.11.1, we added support for dual external routers, which view the Trellis fabric as 2 individual routers, as shown above.

As before the vRouter control plane is implemented as a combination of Quagga, which peers with the upstream routers, and ONOS which listens to Quagga (via FPM) and programs the underlying fabric. In dual-router scenarios, there are two instances of Quagga required. 

As before the hardware fabric serves as the data-plane of vRouter. In dual-router scenarios, the external routers MUST be connected to paired-ToRs. In future releases, we will explore connecting the external routers to any ToRs (paired or unpaired).

Configuration

Lets consider the simpler case where the external routers are each connected to a single leaf in a ToR pair.

The figure on the left below shows the logical view. The figure on the right shows the physical connectivity.

One of the upstream routers is connected to of:205 and the other is connected to of:206. Note that of:205 and of:206 are paired-ToRs.

The ToRs are connected via a physical port to separate Quagga VMs or containers. These Quagga instances can be placed in any compute node. They do not need to be in the same server, and are only shown to be co-located for simplicity.

The two Quagga instances do  NOT talk to each other.

Switchport configuration

in the ToRs follow the same rules as described in the single-router case. 

In the example shown above, the switch port config would look like this:

...
	"of:0000000000000205/1" : {
	    "interfaces" : [{
			"ips" : [ "10.0.100.3/29", "2000::6403/125" ],
			"vlan-untagged": 100,
			"name" : "internet-router-1"
	    }]
	},

	"of:0000000000000205/48" : {
	    "interfaces" : [{
			"ips" : [ "10.0.100.3/29", "2000::6403/125" ],
			"vlan-untagged": 100,
			"name" : "quagga-1"
	    }]
	},

	"of:0000000000000206/1" : {
	    "interfaces" : [{
			"ips" : [ "10.0.200.3/29", "2000::6503/125" ],
			"vlan-untagged": 200,
			"name" : "internet-router-2"
	    }]
	},

	"of:0000000000000206/48" : {
	    "interfaces" : [{
			"ips" : [ "10.0.200.3/29", "2000::6503/125" ],
			"vlan-untagged": 200,
			"name" : "quagga2"
	    }]
	},
...

Note that in the example shown above, switch of:205 uses VLAN 100 for bridging the peering session between Quagga1 and ExtRouter1, while switch of:205 uses VLAN 200 to do the same for the other peering session. But since these vlans and bridging domains are defined on different switches, the VLAN ids could have been the same.

This philosophy is consistent with the fabric use of bridging. 

Quagga configuration

Each instance of Quagga can be downloaded and deployed the same way as described before. Configuring Quagga also follows the same rules described in the single-router case. 

There are a few things to remember

1. The two Zebra instances should point to two different ONOS instances for their FPM connections. For example Zebra in Quagga1 could point to ONOS instance with 

   "fpm connection ip 10.6.0.1 port 262", while the other Zebra should point to a different ONOS instance with "fpm connection ip 10.6.0.2 port 2620". It does not matter which ONOS instances they point to as long as they are different.

2. The two Quagga BGP sessions should appear to come from different routers but still use the same AS number – i.e. the two Quaggas' belong to the same AS, the one used to represent the entire Trellis infrastructure. 
3. The two upstream routers can belong to the same or different AS, but these AS numbers should be different from the one used to represent the Trellis AS.
4. Typically both Quagga instances advertise the same routes to the upstream. These prefixes belonging to various infrastructure nodes in the deployment should be reachable from either of the leaf switches connected to the upstream routers.
5. The upstream routers may or may not advertise the same routes. Trellis will ensure that traffic directed to a route reachable only one upstream router is directed to the appropriate leaf.

Dual-ToRs connected to Both External Routers

Now lets consider the more-complicated but more fault-tolerant case of each Quagga instance peering with BOTH external routers.

Again the logical view is shown on the left and the phycial view on the right

First lets talk about the physical connectivity

• Quagga instance 1 peers with external router R1 via port 1 on switch of:205
• Quagga instance 1 peers with external router R2 via port 2 on switch of:205
  
  

Similarly

• Quagga instance 2 peers with external router R1 via port 2 on switch of:206
• Quagga instance 2 peers with external router R2 via port 1 on switch of:206

To distinguish between the two peering sessions in the same physical switch, say of:205, the physical ports 1 and 2 need to be configured in different vlans and subnets. For example, port 1 on of:205 is (untagged) in vlan 100, while port 2 is in vlan 101.

Note that peering for Quagga1 and R1 happens with IPs in the 10.0.100.0/29 subnet, and for Quagga 1 and R2 in the 10.0.101.0/29 subnet.

Furthermore, port 48 on of:205 carries both peering sessions to Quagga1. Thus port 48 should now be configured as a trunk port (vlan-tagged) with both vlans and both subnets.

Finally the Quagga interface on the VM now needs sub-interface configuration for each VLAN id.

Similar configuration concepts apply to IPv6 as well. Here is a look at the switch port config in ONOS for of:205

...
	"of:0000000000000205/1" : {
	    "interfaces" : [{
			"ips" : [ "10.0.100.3/29", "2000::6403/125" ],
			"vlan-untagged": 100,
			"name" : "internet-router1"
	    }]
	},


	"of:0000000000000205/2" : {
	    "interfaces" : [{
			"ips" : [ "10.0.101.3/29", "2000::7403/125" ],
			"vlan-untagged": 101,
			"name" : "internet-router2"
	    }]
	},
	"of:0000000000000205/48" : {
	    "interfaces" : [{
			"ips" : [ "10.0.100.3/29", "2000::6403/125", "10.0.101.3/29", "2000::7403/125" ],
			"vlan-tagged": [100, 101]
			"name" : "quagga1"
	    }]

	},
...