This page describes CORD's support for service composition and the role VTN plays in service composition. This document takes a “top down” perspective. It starts with the requirements that XOS puts on VTN and then defines the XOS/VTN interface. For more information about VTN’s internals and its relationship to the CORD fabric, see the Trellis documentation.

Service Composition

A given configuration of CORD consists of a set of Services organized in a Service Graph. The edges in this graph represent Tenant dependencies, with a client Service being a tenant of the provider Service. This Tenant relationship both establishes the right of the client service to acquire a service instance from the provider service (this is part of the operator-specified security policy), and specifies how the two services are to be interconnected in the network data plane. VTN plays a key role in this interconnection.

The data model defines a Service as a set of nested object classes, which we informally represent as follows (where Foo[ ] indicates a “set of objects of type Foo”): 

Service = (Controller, Slices[ ])

Slice = (Instances[ ], Networks[ ])

ServiceDependency = (Service_Cons, Service_Prov, ConnectMethod, ConnectService, ConnectNetwork, ConnectAddress)

The ServiceDependency model effectively defines an edge in the Service Graph, connecting a consumer service (Service_Cons) to a provider service (Service_Prov). The other four Connect* fields describe how the two Services' respective Networks are to be interconnected in the data plane (as described below). Even though each Service can have multiple Slices, and each Slice can have multiple Networks, it is common that there is one (data) Network per Slice and one Slice per Service, so to simplify the exposition, the rest of this document makes that assumption. In general, establishing a ServiceDependency relationship requires specifying which Network of which Slice of each Service is to be interconnected.

VTN-Provided Networks

Each Service is connected to one or two Management Networks and one or more Data Networks. VTN implements both types of networks.

VTN defines two management networks that instances can join: 

• MANAGEMENT_LOCAL: This puts the instance on the 172.27.0.0/24 network, which is limited to the local compute node. The compute node's root context is always 172.27.0.1 on this local network. Synchronizers currently use this network to SSH into an Instance to configure it. They SSH to the compute node's root context, and then from there into the Instance. 

• MANAGEMENT_HOST: This puts the instance on the 10.1.0.0/24 network that does span compute nodes and does offer end-to-end connectivity to the head node. This network currently runs over the physical management network. 

These two management networks are completely independent. A Slice can choose to participate in either of them, neither of them, or both of them. In the latter case, each instance would have two management interfaces, on with a 172.27.0.0/24 address and one with a 10.1.0.0/24 address. Instances on different compute nodes can talk to each other on MANAGEMENT_HOST, but they cannot talk to each other on MANAGEMENT_LOCAL. Note that these two management networks are entirely configurable: 172.27.0.0/24 and 10.1.0.0/24 are what been set for CORD-in-a-Box but need not necessarily be the same on a multi-node POD.

The rest of this page focuses on the Data Network, which connects all the instances in a Service to each other (by default), but optionally can also connect that network to other networks in CORD. VTN is responsible for both implementing the base (default) Data Network and for splicing that network to other networks. These “other” networks are provided by both VTN (i.e., the other network is some other Service’s base Data Network) and by other CORD services (e.g., vOLT, vRouter, vCarrierEthernet).

When first created, each Data Network is Private by default, analogous to a private network in OpenStack (i.e., it connects only the Instances in the Service). The network can remain private, or it can be connected onto one or more other networks. These other networks can themselves be the Data Network of some other Service, but it is also possible to connect a Service’s Data Network to the public Internet, making the Instances Internet-routable. This framing of how to connect a Service to the public Internet is unusual (i.e., one connects to a public network by augmenting an existing a Private network), but it is helpful to view the public Internet being just like any other network in CORD – it is provided by some service. In the case of the public Internet, this service is currently provided by vRouter.

Note that in an earlier implementation, an Instance “appears” to connect directly to a Public network (as opposed to first connecting to a private network and then splicing that network to a public network), but that only worked because of a race condition involving Neutron. Should Neutron have had the opportunity to come up before the public network is connected, a private address rather than a public address would be assigned to each Instance's Port. Our new approach is that (a) the private network be established as a first step, and (b) it is possible to assign a second address (in this case public) to each Port. This allows a private network to be made public at any time, and in general, being able to assign multiple addresses to a Port is a requirement.

Interconnecting Networks 

VTN programs the underlying software switches (e.g., OvS) to forward packets to/from the Ports of the Service’s Instances. Connecting a network onto an existing Data Network means Instances in the two networks can exchange packets according to the parameters of the splicing operation (see below) and it may result in a new address being assigned to each Port.

A Service’s Data Network is interconnected to other networks as a consequence or the corresponding Services being composed in the Service Graph. XOS implements the Service Graph, which drives the VTN’s activity. There are four cases, depending on whether the two services being composed are implemented in the network control plane or the network data plane. We start with the example most people assume (the two services implement VNFs in the data plane), and then show how the same principle generalizes to other possible compositions.

When a dependency is established between a pair of data plane services, denoted S_A → S_B, it causes the Data Networks of A and B to be interconnected according to attribute’s assigned to their respective Data Networks. (Recall to simplify the discussion we are making the assumption that each Service has one Slice and each Slice has one Network. In practice, it is necessary to specify which of the Service’s Networks are being interconnected.) Two attribute vectors are (currently) defined:

• Direct vs Indirect

• Unidirectional vs Bidirectional

The first defines whether A uses Direct addressing for the Instances of B (using a unique address for each Instance) or Indirect addressing (using a Service-wide address, where VTN implements load balancing by forwarding the packet to a specific Instance). The second defines whether communication is Unidirectional (A can send packets to B but not vice versa) or Bidirectional (A and B can send packets to each other).

Technically, these attributes are associated with Service B’s Network rather than with the interconnection of A to B. When A interconnects with B, the properties of B’s Data Network dictate the terms of the interconnection. But as we expand the capability, it may be possible that attributes of the interconnection will be specified with the interconnection model rather than one or other of the networks being interconnected.

The above corresponds to a service implemented by one set of Instances being composed with a service implemented by another set of Instances. This results in two VTN-implemented Data Networks being connected together. Because VTN allocates a disjoint block of addresses drawn from a common private address space to each Data Network, this means a new block of addresses becomes visible to (routable from) each Service; each Instance continues to have the same private IP address assigned to its Ports.

In general, however, one or both of S_A and S_B might be “control plane” Services, in which case it implements a network rather than uses a network. Two examples are vOLT and vRouter, meaning we have two “mixed” service interconnections:

• vOLT → vSG (a control plane service connects to a data plane service)

• vSG → vRouter (a data plane service connects to a control plane service)

What this means is that rather than VTN splicing together two VTN-based Data Networks, we are asking VTN to connect some “other” network to the VTN-defined Data Network. 

Similarly, this generalizes to account for other Networks-as-a-Service; e.g.,

• vSG → vCarrierEthernet (a container connects to a metro-area virtual network on demand service)

Finally, although VTN does not yet support this case, one can imagine a situation where one control plane service connects to another control plane service, in which case VTN will need to interconnect two networks (neither of which VTN implements itself). For example:

• vOLT → vRouter (or alternatively, vSG → vRouter, where vSG is SDN-based rather than implemented by a container)

Looking across this set of examples, there appears to be two subcases. In the first, when interconnecting two VTN-based networks, the result is basically the union of the two original networks (with restrictions). In the second, when interconnecting a Service to some ONOS-provided network, the result is to dynamically add the Service Instances to that new network, with the side-effect of the instances being assigned a new address on that network. These two subcases can be traced back to the two roles VTN plays: (1) it connects instances to networks, and (2) it provides a private network for a set of instances.

Previous Mechanism 

The previous mechanism (in v1.0) by which VTN knows how to connect networks was specialized, hard-coded, and configured at the granularity of Neutron ports:

• vOLT → vSG: The neutron port is named after the s-tag. For example, a port named "stag-123" will receive vOLT packets with s-tag 123.

• vSG → vRouter: The allowed_address_pairs attribute is set for the port to indicate which public addresses it has. For example allowed_address_pairs=[{"ip_address": "10.168.0.2", "mac_address": "02:42:0a:a8:00:02"}] would put vRouter traffic for 10.168.0.2 onto the private network. A private network can have several public addresses specified in the allowed_address_pairs attribute.

One design challenge addressed by this version is how to generalize this implementation so it is not hard-coded for these specific Services. Another way to say this is that we have redesigned VTN to be a general-purpose Instance-to-Network, Network-to-Instance, Instance-to-Instance, and Network-to-Network connector, where the Instance on one side or the other can be a VM or a container, and the Network on one side or the other can be any ONOS-defined network (including a VTN-based private network).

Interfaces

This section defines the two interfaces (VTN-provided and XOS-provided) necessary to support service composition. XOS invokes the VTN-provided API to interconnect networks belonging to two composed services. VTN invokes the XOS-provided API to restore interconnection state (e.g., if VTN restarts), but this interface is not involved in typical XOS/VTN interaction. The following diagram shows the relationship between XOS, OpenStack Neutron, and the various sub-systems of VTN.

In a configuration that includes OpenStack, XOS indirectly calls VTN via Neutron. In this case, Neutron's ML2 plugin informs OpenStack Nova about the virtual network connecting a set of instances. XOS can also create a virtual network by directly calling VTN without Neutron's involvement (e.g,. to interconnect Docker containers).

Because VTN provides a CLI to purge its internal state, it uses the XOS-provided API to resync with XOS. This VTN-to-XOS interface is not shown in the figure.

XOS Provided API

ServicePorts

GET api/service/vtn/servicePorts list service ports including port details 

GET api/service/vtn/servicePorts/{port_id}  show service port details

Service Port Details

{
   "servicePorts": [
       {
           "id": "55d1b71f-efe2-455d-a78c-e48e9d3a4094"
       },
       {
           "id": "7098f0a8-cdcb-4bce-9706-05aecacb784b",
           "vlan_id": 222,
           "floating_address_pairs": [
               {
                   "ip_address": "20.0.0.3",
                   "mac_address" : "fa:16:3e:ca:05:9c"
               }
           ]
       }
   ]
}

ServiceNetworks

GET api/service/vtn/serviceNetworks list service networks including the details 

GET api/service/vtn/serviceNetworks/{network_id}  show service network details

Service Network Details

• PRIVATE: virtual network for the instances in the same service

• PUBLIC: externally accessible network

• MANAGEMENT_LOCAL: instance management network which does not span compute nodes, only accessible from the host machine

• MANAGEMENT_HOST: real management network which spans compute and head nodes

• ACCESS_AGENT: network for access agent infrastructure service

Example json response: api/service/vtn/serviceNetworks

{
    "serviceNetworks": [
        {
            "id": "8ab38619-327e-47ce-9304-2c595c1b6708",
            "type": "public",
            "providerNetworks": [],
            "tenantNetworks": []
        },
        {
            "id": "a14d7a6b-dffb-4271-8a17-5d715b362d1e",
            "type": "private",
            "providerNetworks": [],
            "tenantNetworks": [
                {
                    "id": "71cc8c93-f809-42ff-b1d6-0c8d92c6cd2b",
                    "bidirectional": true
                }
            ]
        }
    ]
}

Example json response: api/service/vtn/serviceNetworks/a14d7a6b-dffb-4271-8a17-5d715b362d1e

{
    "service": {
        "id": "a14d7a6b-dffb-4271-8a17-5d715b362d1e",
        "type": "private",
        "providerNetworks": [],
        "tenantNetworks": [
            {
                "id": "71cc8c93-f809-42ff-b1d6-0c8d92c6cd2b",
                "bidirectional": true
            }
        ]
    }
}

VTN Provided API

ServicePorts

POST onos/cordvtn/servicePorts create a service port 

GET onos/cordvtn/servicePorts list service ports including service port details

GET onos/cordvtn/servicePorts/{port_id} show service port details 

DELETE onos/cordvtn/servicePorts/{port_id} delete a service port

Service Port Details

* fields are mandatory for creating a new service port.

{
   "ServicePort":{
      "id":"b8ba3d85-1dec-49f9-8503-6f1b90399152",
      "network_id":"0f30807c-fce0-4fa5-9c36-99ff6d4d0ae1",
      "name":"tapb8ba3d85-1d",
      "vlan_id":222,
      "mac_address":"FA:16:3E:56:77:45",
      "ip_address":"10.0.2.2",
      "floating_address_pairs":[
         {
            "ip_address":"10.6.1.131",
            "mac_address":"02:42:0A:06:01:83"
         },
         {
            "ip_address":"10.6.1.130",
            "mac_address":"02:42:0A:06:01:82"
         }
      ]
   }
}

POST onos/cordvtn/serviceNetworks create a service network

GET onos/cordvtn/serviceNetworks list service networks including the details

GET onos/cordvtn/serviceNetworks/{network_id}  show service network details

PUT onos/cordvtn/serviceNetworks/{network_id} update a service network dependencies 

DELETE onos/cordvtn/serviceNetworks/{network_id} delete a service network

Service Network Details

* fields are mandatory for creating a new service network

Example json request: onos/cordvtn/serviceNetworks

{
   "ServiceNetwork":{
      "id":"e4974238-448c-4b5c-9a45-b27c9477eb6a",
      "name":"mysite_one-access",
      "type":"PRIVATE",
      "segment_id":1018,
      "subnet":"10.0.3.0/24",
      "service_ip":"10.0.3.1",
      "providers":[
         {
             "id": "71cc8c93-f809-42ff-b1d6-0c8d92c6cd2b",
             "bidirectional": true
         }
      ]
   }
}

XOS and the ServiceDependency Model 

The ServiceDependency model (formerly the Tenant model) defines the tenant/provider relationship between services. It serves two purposes. One is to specify a dependency between a pair of services. This how the operator establishes that one service is permitted to acquire an instance of the another service. The second is to specify the means by which the two services connect to each other. This is what triggers VTN setting up the data path through the underlying network infrastructure, or the use of some other network service like vRouter.

The service graph is defined by a set of services (nodes) and tenancy relationships (edges). The existence of an edge between Services A and B means A is permitted to be a tenant of B. Four new fields will be added to the Tenancy object: 

ServiceDependency.ConnectMethod = [DefaultPrivate | DefaultPublic | Service | None] 

ServiceDependencyt.ConnectService = <ServiceID>

ServiceDependency.ConnectNetwork = <NetworkID>

ServiceDependency.ConnectAddress = IP:Port

The ConnectMethod field states which type of connection is to be performed. It can be set to DefaultPrivate, DefaultPublic, Service, or None. DefaultPrivate and DefaultPublic indicate that XOS will choose a connection method that XOS deems as appropriate (VTN or vRouter in the common R-CORD case). Service indicates that a specific service should be used. None indicates that no network connectivity will be provided.

The ConnectService field is used when ConnectMethod==Service and specifies the service to use for connection. For example, VTN, vRouter, vCarrier Ethernet, etc. Both the provider and the subscriber service must be tenants of the service. For example, if ConnectMethod==vRouter, then both provider and subscriber must be tenants of vRouter. ConnectService may be set to None in the case where ConnectMethod is DefaultPrivate, DefaultPublic, or None. 

The ConnectNetwork field specifies the network that should be used. It is the ID of a network that is provided by a slice belonging to the provider service. ConnectNetwork may be set to None in the case that no network is required, for example, when connecting using the public Internet.

The ConnectAddress field specifies the IP Address and Port where the subscribing service should contact the providing service. This is optional. Some services may have more complicated semantics than can be distilled to a single IP:Port. (Note that although VTN already offers a similar feature, it makes sense for services that connect publicly, such as CDN, to be able to specify what address the subscriber should connect to.)

Example Service Graphs

This section walks through a sequence of example Service Graphs as a way of explaining the implications of the models described above. One of the main takeaways from the following discussion is that our current characterization of services – blue is for control plane services and  red is for data plane services – is too simplistic. There are other dimensions to the service space. As described below, the Tenancy relationships need to explicitly define the relationship between services; it cannot implicitly depend on the control-vs-data plane characterization.

Simple R-CORD Service Graph

 Example 1: Canonical service graph in R-CORD.

The first example depicts the canonical service graph for R-CORD. It includes the network-to-instance and instance-to-network interconnections described above. It also shows an example TenantRoot (Subscriber) model that anchors a service graph; this corresponds to a principal that initially requests tenancy from CORD. The Subscriber → vOLT edge represents a ServiceDependency model, exactly like any other edge in the service graph; the only difference being that the tenant corresponds to a user rather than a service. (We drop the TenantRoot in all the other examples in this section.)

Building Block Services

 Example 2: Including building block services. 

The second example shows the tenancy relationships between the vOLT → vSG → vRouter chain from before, with two of the underlying building block services: Monitoring and ONOS. In the case of the dependency on Monitoring, this is so all the other services can send metrics to the Monitoring service. This would use the Monitoring service’s VTN-based network, so even though vOLT, vRouter, and ONOS control the network (they are blue in the figure), they also run in a scalable set of instances, and these instances need to communicate with the instances that implement the Monitoring service.

Similarly, the two edges vOLT → ONOS and vRouter → ONOS signify that both vOLT and vRouter are tenants of ONOS, but there is no data plane connection since tenancy is implemented by loading the corresponding control applications into the ONOS instances. (Technically, there is a “data plane” connecting vOLT and vRouter to ONOS, but it is implemented internally by the JVM, and so it not visible to XOS.) Had it been the case that vOLT and vRouter ran in a separate scalable set of instances, their data plane connection to ONOS would have been via the VTN-based private network connecting those services. As it is, the ConnectMethod is likely set to “None” in the current design.

Generalized Service Graph

 Example 3: Generalized Example (general shape).

The third example shows the general expected shape of a CORD service graph. There are two things of note. First, a CORD service graph always has one or more “access side” (left-most) control-based services (e.g., vOLT, vG.Fast) and one or more “Internet side” (right-most) services (e.g., vRouter, vCE). Typically, a subgraph of data plane services define the middle of the service graph, but there will never be a data plane (red) service at the left-most or right-most edge. In practice, vSG is always VNF1 today, but we expect that limitation to eventually go away. Also, the VNF1 → VNF2 relationship is implemented by VTN in the data plane.

Tenancy and Data Plane 

 Example 4: Both tenancy and data plane.

The fourth example shows a VNF dependency (VNF2 → VNF3) where both are also connected to a common network service (vRouter). This means that if the VNF2 → VNF3 is labelled ConnectMethod=DefaultPublic, then vRouter implements the data plane connectivity between the instances in these two services. (If it is labelled ConnectMethod=DefaultPrivate, then VTN would implement the connection in the data plane.) Although not shown in this example, if VNF2 → VNF3 had been labelled ConnectMethod=Service and ConnectService=vCE, then vCE would have implemented the data plane connection, but only if both VNF2 and VNF3 area also tenants of that service. Trying to connect two services with a third service should throw a “configuration-time exception” if the two services are not also tenants of the third service.

Arbitrary Control Plane Services

 Example 5: Arbitrary Control Plane Services.

The fifth example is speculative (not yet supported), and includes a “third party” control plane service (vMagic). Behavior is essentially the same as before (in this case, VNF1 and VNF2 can elect to use vMagic as their ConnectMethod), but it does serve to highlight that not all control plane services are implemented by the network infrastructure (i.e., the hardware fabric and the software switch running in the kernel on each node). Instead, the data plane might be implemented in a software switch running in an instance. This implies both the vMagic control plane (perhaps an ONOS-hosted app) and the vMagic data plane (some implementation of a software switch) are implemented in compute instances, in which case the VNF1 → vMagic and VNF2 → vMagic interconnections are implemented by VTN’s instance-to-instance interconnection mechanism.

This example highlights the need to do a better job characterizing “infrastructure” and “non-infrastructure” services. Equating vMagic with vRouter or VTN does not seem appropriate, and our control plane versus data plane characterization also breaks down. It is probably more accurate to think of vMagic as a data plane service (and show it in red) even though internally it organizes itself as decoupled control plane and data plane subsystems. (From an XOS perspective, these two subsystems would be implemented in two different Slices.)

Here is an attempt at a set of definitions:

• Infrastructure Data Plane – All switching infrastructure, including both hardware switches and “root” software switches (OvS) on each server, for which there can be only one controller. Today, there is a separate controller for each (ONOS-CORD for the OvS-based overlay and ONOS-Fabric for the hardware-based switching fabric), but this is an implementation artifact and not fundamental. The key is that other (e.g., third-party) controllers are not allowed to share control of the infrastructure data plane.

• Infrastructure Services – The set of control plane services services, and corresponding ONOS apps, that manage the Infrastructure Data Plane. Today these include VTN, Fabric, and vRouter. We are now referring to the Infrastructure Data Plane plus the collection Infrastructure Services as Trellis.

• Control Plane Services – The set of services implemented by an ONOS control app managing either the Infrastructure Data Plane or some other hardware-based data plane (e.g., vOLT controlling the access devices). Control Plane Services are a superset of the Infrastructure Services.

• Data Plane Services – The set of services implemented by a scalable set of compute instances. Internally, such services can be implemented as a homogeneous set of instances (i.e., they instances host the same image) or they may be organized into a control-plane half (running in one Slice) and a data-plane half (running in another Slice). By this definition, vMagic from above is a Data Plane Service.

The open question is whether there are other software-based data planes – beyond the privileged OvS half of the Infrastructure Data Plane – that can be jointly shared by third-party Control Plane Services. If all possible Data Planes are single-tenant, then any service that wants to organize itself into control-plane and data-plane halves (where the data-plane is implemented by some software switch running  in one or more compute instances) is best viewed as a compound Data Plane Service.

Network-to-Network Composition

 Example 6: Control-to-Control Composition.

The sixth example is also speculative. It assumes vSG is implemented by an ONOS application (we assume in the infrastructure rather than as a third party service as in the previous example). In this case, VTN will need to be extended to interconnect pairs of virtual networks, the exact method (and VTN AP) still TBD.