CORD : Tunneling Upstream Network Through Head Node Management Network

This page describes POD configuration settings / processes to use when attempting to tunnel the fabric uplink connection over the management network using the head node to NAT that uplink traffic.

Currently, CORD only works with a single leaf configuration. This is because the fabric requires a subnet per leaf, but the NFV orchestration has not yet been updated to support multiple subnets when deploying NFVs.

Overview

Break the fabric bond interface on the head node, so that the head node will have two fabric interfaces
Configure vRouter to route to the secondary fabric interface on the head node
Update fabric configuration to support secondary fabric interface on head node

Pre-Configuration

Before you invoke the ./gradlew -PdeployConfig=podX.yml fetch buildImages publish deploy command, there are a few changes that should be made to the existing source code.

It is unclear that these changes should be checked into the source code tree, which is why they remain here as manual edits

XOS address pool ranges set in $CORD_ROOT/build/platform-install/roles/xos-install/templates/cord-services.yaml.j2, should be changes from a subnet of the default fabric network 10.6.1.0/24 to each having its own /24 subnet.

Address Pool Changes

    addresses_vsg:                                                       addresses_vsg:
        type: tosca.nodes.AddressPool                                      type: tosca.nodes.AddressPool
        properties:                                                        properties:
          addresses: 10.6.1.128/26                                |          addresses: 10.7.1.0/24
          gateway_ip: 10.6.1.129                                  |          gateway_ip: 10.7.1.1
          gateway_mac: 02:42:0a:06:01:01                                     gateway_mac: 02:42:0a:06:01:01
                                                                   
      addresses_public:                                                  addresses_public:
        type: tosca.nodes.AddressPool                                      type: tosca.nodes.AddressPool
        properties:                                                        properties:
          addresses: 10.6.1.192/26                                |          addresses: 10.8.1.0/26
          gateway_ip: 10.6.1.193                                  |          gateway_ip: 10.8.1.1
          gateway_mac: 02:42:0a:06:01:01                                     gateway_mac: 02:42:0a:06:01:01

Head Node - Post Deployment

After the head node is deployed, but before any compute nodes have been booted, the networking on the head node needs to be updated. Specifically the following changes should be made in /etc/network/interfaces files:

Remove the second NIC from the fabric interface bond. This means deleting the the lines in the file under the interface definition that pertain to bonding
Set this interface with no IP address and to manual configuration. It should look something like:
Secondary Fabric Interface
```
auto p1p2
iface p1p2 inet manual
```
Create a new bridge interface and assign the secondary fabric interface into that bridge as a port
Secondary Fabric Bridge
```
auto br-wan
iface br-wan inet static
  address 10.5.1.1
  network 10.5.1.0
  netmask 255.255.255.0
  broadcast 10.5.1.255
  bridge_ports p1p2
```
The subnet 10.5.1.0/24 can be any subnet, but must be different than the primary subnet for the fabric.
After updating the /etc/network/interfaces file, you should convince the head node to accept these network changes. This can be accomplished in various ways, including rebooting the head node, or issuing the appropriate ip / ifconfig commands. If you reboot the head node you will likely have to re-deploy XOS.
If you end up deploying XOS, you want to be sure to disable network configuration in the deployment configuration file so that the changes made to the head node configuration are not overwritten. This is done by enabling the skipTags in the deployment configuration file for the interface_config option
```
skipTags:
```
```
  - 'interface_config'
```
Configure static arp rule on head node to to the switch IP for the secondary fabric interface, i.e. the x.x.x.254 address on the subnet. The MAC address used for the arp should be that of the leaf switch to which the head node is connected.
```
arp -s 10.5.1.254 cc:37:ab:7c:b9:d6
```

Boot Compute Nodes and Switches

As CORD only works with a single leaf, be sure that all compute nodes are attached to a single leaf when they bo

Fabric Configuration - Post Compute Node Boot

Once the head node is deployed, but before booting any compute nodes, the leaf spine fabric should be configures. Below is a sample of the fabric configuration utilized on the example POD. Important highlights of this configuration are:

lines 49 - 50: Add the IPs that will be used for the created vSGs and vCPEs. This really should be all IPs in the range 10.7.1.2 to 10.7.1.253 inclusive, but need only be the IPs actually used as you demonstrate CORD
lines 63 - 70: A host block for the secondary fabric interface on the head node
lines 91 - 101: Define the secondary interface on the head node as the upstream router

This sample POD was using the subnet 10.2.1.0/24 for the fabric subnet, not 10.6.1.0/24 which is often used as the default.

Sample Fabric Configuration

{
  "devices": {
    "of:0000cc37ab7cbe68": {
      "segmentrouting": {
        "name": "device-cc37ab7cbe68",
        "nodeSid": 100,
        "routerIp": "10.2.0.14",
        "routerMac": "cc:37:ab:7c:be:68",
        "isEdgeRouter": false,
        "adjacencySids": []
      }
    },
    "of:0000cc37ab7cbfee": {
      "segmentrouting": {
        "name": "device-cc37ab7cbfee",
        "nodeSid": 101,
        "routerIp": "10.2.0.12",
        "routerMac": "cc:37:ab:7c:bf:ee",
        "isEdgeRouter": false,
        "adjacencySids": []
      }
    },
    "of:0000cc37ab7cb9d6": {
      "segmentrouting": {
        "name": "device-cc37ab7cb9d6",
        "nodeSid": 102,
        "routerIp": "10.2.0.10",
        "routerMac": "cc:37:ab:7c:b9:d6",
        "isEdgeRouter": true,
        "adjacencySids": []
      }
    },
    "of:0000cc37ab7cbada": {
      "segmentrouting": {
        "name": "device-cc37ab7cbada",
        "nodeSid": 103,
        "routerIp": "10.2.0.5",
        "routerMac": "cc:37:ab:7c:ba:da",
        "isEdgeRouter": true,
        "adjacencySids": []
      }
    }
  },
  "hosts": {
    "3C:FD:FE:9E:8A:88/-1": {
      "basic": {
        "ips": [
          "10.2.1.2",
          "10.7.1.2",
	      "10.7.1.3"
        ],
        "location": "of:0000cc37ab7cb9d6/2"
      }
    },
    "3C:FD:FE:9E:93:10/-1": {
      "basic": {
        "ips": [
          "10.2.1.1"
        ],
        "location": "of:0000cc37ab7cb9d6/1"
      }
    },
    "3C:FD:FE:9E:93:11/-1": {
      "basic": {
        "ips": [
          "10.5.1.1"
        ],
        "location": "of:0000cc37ab7cb9d6/3"
      }
    }
  },
  "ports": {
    "of:0000cc37ab7cb9d6/2": {
      "interfaces": [
        {
          "ips": [
            "10.2.1.254/24"
          ]
        }
      ]
    },
    "of:0000cc37ab7cb9d6/1": {
      "interfaces": [
        {
          "ips": [
            "10.2.1.254/24"
          ]
        }
      ]
    },
    "of:0000cc37ab7cb9d6/3": {
      "interfaces": [
        {
          "name": "upstream",
          "ips": [
            "10.5.1.254/24"
          ],
          "mac" : "cc:37:ab:7c:b9:d6"
        }
      ]
    }
  },
  "links": {},
  "apps": {
    "org.onosproject.router": {
      "router": {
        "controlPlaneConnectPoint": "of:0000cc37ab7cb9d6/5",
        "ospfEnabled": "true",
        "interfaces": [
          "upstream"
        ]
      }
    },
    "org.onosproject.segmentrouting": {
      "segmentrouting": {
        "suppressHostByPort": [
          "of:0000cc37ab7cb9d6/3"
        ],
        "suppressHostByProvider": [
          "org.onosproject.provider.host"
        ],
        "suppressSubnet": [
          "of:0000cc37ab7cb9d6/3"
        ],
        "vRouterId": "of:0000cc37ab7cb9d6",
        "vRouterMacs" : [
          "02:42:0a:06:01:01"
        ]
      }
    }
  }
}

Restart ONOS Fabric Applications

ssh -p 8101 karaf@localhost (password: karaf)
onos> app deactivate org.onosproject.segmentrouting
onos> app deactivate org.onosproject.vrouter
onos> app activate org.onosproject.segmentrouting
onos> app activate org.onosproject.vrouter

Add Router to VRouter

onos> route-add 0.0.0.0/0 10.5.1.1

Add Default Route on Compute Nodes

On each compute node, remove any default route and add the leaf IP as the default route

ssh <compute-node>
sudo ip route del default
sudo ip route add default via 10.2.1.254