CORD : Trellis Troubleshooting Tool (T3)

Overview
Supported packet types
Activation and use
Command Line Interface
REST API

Overview

The Trellis Troubleshooting Tools is an ONOS application intended to help you troubleshoot traffic-related problems in your network.

The T3 tool works both on simulated OVS-based topologies but also on real HW, basing it's internal decisions on the driver assigned to the device.

T3 also currently supports dual links between leaf and spine and properly handles the difference between deferred and immediate rules.

Dual Homed hosts are also supported in T3.

Supported packet types

T3 supports, both IPv4 and Ipv6

L3 Unicast
L2 Unicast
L2 Broadcast
L3 Multicast
LLDP and BDDP
ARP
DHCP

Activation and use

To activate this tool on your ONOS environment you need to

onos> app activate org.onosproject.t3

Command Line Interface

Troubleshoot CLI command

As of right now the tools provides a CLI command (troubleshoot). As arguments to the CLI command you have to pass arguments describing the packet you want to trace through the network.

Command Help

onos> t3-troubleshoot --help
DESCRIPTION
        onos:troubleshoot

	troubleshoots flows and groups between source and destination

SYNTAX
        onos:troubleshoot [options]

OPTIONS
        -sm, --srcMac
                Source MAC
        -j, --json
                Output JSON
        --help
                Display this help message
        -v, --verbose
                Outputs complete path
        -ipp, --ipProto
                IP Proto
        -s, --srcIp
                Source IP
        -dm, --dstMac
                Destination MAC
        -ml, --mplsLabel
                Mpls label of incoming packet
        -vv, --veryverbose
                Outputs flows and groups for every device
        -udps, --udpSrc
                UDP Source
        -mb, --mplsBos
                MPLS BOS
        -et, --ethType
                ETH Type
                (defaults to ipv4)
        -sp, --srcPort
                Source Port
        -dtp, --dstTcpPort
                destination TCP Port
        -udpd, --udpDst
                UDP Destination
        -stp, --srcTcpPort
                Source TCP Port
        -d, --dstIp
                Destination IP
        -vid, --vlanId
                Vlan of incoming packet
                (defaults to None)

Plain Verbosity Level

Let's look at an example.

onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None

with this command we are asking the t3 tool to trace a packet with a Source IP of 10.0.2.1/32 input port of:0000000000000204/3 source MAC 00:AA:00:00:00:01 ethType Ipv4 Source TCP Port 12345 Destination IP 10.0.3.1/32 destination MAC 00:AA:00:00:00:01, Destination TCP Port 80 and VlanId None.

This command has a verbosity level of 0 and will output just the paths that the packet might take inside the network. In the example deployment taken from here https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py the output will be:

Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Paths
[of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
[of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
Result:
Reached required destination Host
Reached required destination Host

Middle Verbosity Level

There are also two other verbosity levels.

onos> t3-troubleshoot -v -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None

prints flows selectors and groupId that are met along the path

Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
flowId=a20000bd57cb49, selector=[]
flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
flowId=a2000038b487e7, selector=[]
Groups
groupId=GroupId{id=0x70000010}
groupId=GroupId{id=0x9200000d}
groupId=GroupId{id=0x9000000c}
groupId=GroupId{id=0xffe0002}
groupId=GroupId{id=0x9200000f}
groupId=GroupId{id=0x9000000e}
groupId=GroupId{id=0xffe0001}
Output through of:0000000000000204/2
Device of:0000000000000227
Input from of:0000000000000227/1
Flows
flowId=a200005577bcd6, selector=[]
flowId=900000498457b2, selector=[IN_PORT:1, VLAN_VID:None]
flowId=900000f87746c9, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
flowId=a200005bfd6ed7, selector=[]
flowId=9000009f3d554f, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
flowId=a20000e601fc35, selector=[]
Groups
groupId=GroupId{id=0x70000017}
groupId=GroupId{id=0x20000016}
groupId=GroupId{id=0xffe0002}
Output through of:0000000000000227/2
Device of:0000000000000205
Input from of:0000000000000205/2
Flows
flowId=a200008d542817, selector=[]
flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
flowId=a2000042d8e6b5, selector=[]
Groups
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
Output through of:0000000000000205/3
Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
flowId=a20000bd57cb49, selector=[]
flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
flowId=a2000038b487e7, selector=[]
Groups
groupId=GroupId{id=0x70000010}
groupId=GroupId{id=0x9200000d}
groupId=GroupId{id=0x9000000c}
groupId=GroupId{id=0xffe0002}
groupId=GroupId{id=0x9200000f}
groupId=GroupId{id=0x9000000e}
groupId=GroupId{id=0xffe0001}
Output through of:0000000000000204/1
Device of:0000000000000226
Input from of:0000000000000226/1
Flows
flowId=a20000e9b27d24, selector=[]
flowId=9000004bd76105, selector=[IN_PORT:1, VLAN_VID:None]
flowId=90000069f274d1, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
flowId=a200000ec7529e, selector=[]
flowId=90000050f9a350, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
flowId=a20000b15cada6, selector=[]
Groups
groupId=GroupId{id=0x7000000b}
groupId=GroupId{id=0x20000009}
groupId=GroupId{id=0xffe0002}
Output through of:0000000000000226/2
Device of:0000000000000205
Input from of:0000000000000205/1
Flows
flowId=a200008d542817, selector=[]
flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
flowId=a2000042d8e6b5, selector=[]
Groups
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
groupId=GroupId{id=0x2000002a}
groupId=GroupId{id=0x1e0003}
Output through of:0000000000000205/3
Result:
Reached required destination Host
Reached required destination Host

High Verbosity Level

The most verbose output you can get is

onos> t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None

This verbosity level prints all the flows and groups that are met along the path

Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
Output through of:0000000000000204/2
Device of:0000000000000227
Input from of:0000000000000227/1
Flows
ADDED, bytes=702318, packets=8511, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=343130, packets=4234, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=754, packets=7, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
ADDED, bytes=17566, packets=57, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
ADDED, bytes=510, packets=5, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x70000017], transition=TABLE:60]
ADDED, bytes=17566, packets=57, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x70000017, state=ADDED, type=SELECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=2
id=0x70000017, bucket=1, bytes=754, packets=7, actions=[GROUP:0x20000016]
id=0x20000016, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x20000016, bucket=1, bytes=754, packets=7, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:27, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=754, packets=7, actions=[VLAN_POP, OUTPUT:2]
Output through of:0000000000000227/2
Device of:0000000000000205
Input from of:0000000000000205/2
Flows
ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
Output through of:0000000000000205/3
Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
Device of:0000000000000204
Input from of:0000000000000204/3
Flows
ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
Output through of:0000000000000204/1
Device of:0000000000000226
Input from of:0000000000000226/1
Flows
ADDED, bytes=701910, packets=8507, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=358984, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=16608, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
ADDED, bytes=17158, packets=53, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
ADDED, bytes=16608, packets=48, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x7000000b], transition=TABLE:60]
ADDED, bytes=17158, packets=53, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x7000000b, state=ADDED, type=SELECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=2
id=0x7000000b, bucket=1, bytes=16608, packets=48, actions=[GROUP:0x20000009]
id=0x20000009, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0x20000009, bucket=1, bytes=16608, packets=48, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:26, VLAN_ID:4094, GROUP:0xffe0002]
id=0xffe0002, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
id=0xffe0002, bucket=1, bytes=16608, packets=48, actions=[VLAN_POP, OUTPUT:2]
Output through of:0000000000000226/2
Device of:0000000000000205
Input from of:0000000000000205/1
Flows
ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
Groups
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
Output through of:0000000000000205/3
Result:
Reached required destination Host
Reached required destination Host

Bridging example

In this example we trace a packet that should be bridged between two hosts in the same subnet

onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:AA:00:00:00:02 -vid None
Tracing packet: [IN_PORT:3, ETH_DST:00:AA:00:00:00:02, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Paths
[of:0000000000000204/3, of:0000000000000204/4]
Result:
Reached required destination Host

Failure Detection Example

Obviously failures can happen along one of the paths that the packet can take.

In our sample we removed a flow in of:0000000000000226 thus disabling that possible path for our initial packet. The output of our trace would then look something like

onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
Paths
[of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
[of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1]
Result:
Reached required destination Host
No output out of device of:0000000000000226 Packet is dropped

Notice that on the second path that was previously going through of:0000000000000226 now the packet is dropped due to the rule we removed.

Command Line Samples

These sample commands refer to the mininet environment found here:

https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py

L2 Unicast IPv4

t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.2.2 -dm 00:AA:00:00:00:02

L2 Unicast IPv6

t3-troubleshoot -vv -s 2000::301  -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::302 -dm 00:BB:00:00:00:04 -et ipv6

L3 Unicast IPv4

t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.1 -dm 00:00:00:00:02:04

L3 Unicast IPv6

t3-troubleshoot -vv -s 2000::301  -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::202 -dm 00:00:00:00:02:05 -et ipv6

ARP

t3-troubleshoot -sp of:0000000000000204/3 -et ARP

DHCP

t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.253 -dm 00:00:00:00:02:04 -ipp 17 -udps 67 -udpd 67

LLDP

t3-troubleshoot -sp of:0000000000000204/3 -et lldp

BDDP

t3-troubleshoot -sp of:0000000000000204/3 -et bddp

LLDP from CONTROLLER

T3 is also capable of tracing packets coming in from the CONTROLLER. We use the same input port with the possibility to give CONTROLLER as logical port number. This is a mere convenience to carry the information inside T3 and do the proper flooding to all the ports of the given device. This case is not treated like all the rest and the input port here is a logical entity that has no match on any physical element on the device.

t3-troubleshoot -sp of:0000000000000204/CONTROLLER -et lldp

Multicast

T3 is also capable of tracing multicast packets.

t3-troubleshoot -vv -sp of:0000000000000205/9 -et ipv4 -d 224.0.0.1 -dm 01:00:5e:00:00:01

Troubleshoot Simple CLI command

T3 also offers a simplified version of the command to be able to ping between two given hosts without having to specify all the information. This commands bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast. If that is not present the user needs to input this information manually recurring to the the previously shown command. Same verbosity levels and output as the troubleshoot command are possible.

onos> t3-troubleshoot-simple --help
DESCRIPTION
        onos:troubleshoot-simple

	Given two hosts troubleshoots flows and groups between them, in case of segment routing

SYNTAX
        onos:troubleshoot-simple [options] one two

ARGUMENTS
        one
                One host ID
        two
                Another host ID

OPTIONS
        -vv, --veryverbose
                Outputs flows and groups for every device
        -j, --json
                Output JSON
        --help
                Display this help message
        -et, --ethType
                ETH Type
                (defaults to ipv4)
        -v, --verbose
                Outputs complete path

Troubleshoot Simple CLI examples

Examples are given for the same trellis.py script.

L2 Unicast

Hosts are under the same leaf, only interface configuration is needed.

t3-troubleshoot-simple -v -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None

L3 Multicast

Hosts are under different leafs, also segment routing configuration config is needed for the leaf's routerMac.

t3-troubleshoot-simple -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:03/None

Troubleshoot PingAll CLI command

T3 also offers a ping-all command to be able to test ping between any given hosts of an EthType. This commands bases itself on top of the simplified Host to Host command. It has two verbosity levels, one just with Host and Destination with the result message of the Trace and one that shows also the packet and the paths.

onos> t3-troubleshoot-pingall --help
DESCRIPTION
        onos:t3-troubleshoot-pingall

	Traces a ping between all hosts in the system of a given ETH type

SYNTAX
        onos:t3-troubleshoot-pingall [options]

OPTIONS
        -j, --json
                Output JSON
        --help
                Display this help message
        -et, --ethType
                ETH Type
                (defaults to ipv4)
        -v, --verbose
                Outputs trace for each host to host combination

Troubleshoot PingAll examples

IPv4 hosts

t3-troubleshoot-pingall

IPv6 hosts

t3-troubleshoot-pingall -et IPV6

Troubleshoot MCast CLI command

T3 also offers a mcast command queries the system for all the multicast routes given a certain vlanId. It prints all the traces given a certain group, source and source connect point. It has two "standard" verbosity levels.

onos> t3-troubleshoot-mcast --help
DESCRIPTION
        onos:t3-troubleshoot-mcast
 
 Traces all the mcast routes present in the system
 
SYNTAX
        onos:t3-troubleshoot-mcast [options]
 
OPTIONS
        -j, --json
                Output JSON
        --help
                Display this help message
        -vv, --veryverbose
                Outputs details of every trace
        -vid, --vlanId
                Vlan of incoming packet
                (defaults to None)
        -v, --verbose
                Outputs trace for each host to host combination

REST API

T3 now also offers a way to execute the troubleshoot through REST API endpoints.

You can find the description and test the API at the ONOS docs endpoint here:

<ONOS_IP>:8181/onos/v1/docs/#/t3

The rest API offered are a non verbose and a verbose version of the troubleshoot simple command. It bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast. The use need to specify just the Source Host, the destination host and the ETH type.

curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>'

curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>/verbose'