Old Version
This page contains information that describes how to install OFDPA 3.0 EA4 with ONL 2.x, which is the version supported in 4th release of CORD (Shared Delusion).
Introduction
In this wiki page, we are going to show you how to setup the hardware switches for CORD Fabric.
Here we focus on Accton 6712 (32 X 40G) switches, which are the switches we use to build the CORD Pod in ON.Lab.
In principle, you could use other switch models (1G, 10G, 40G, 100G) in the fabric as long as they support the OF-DPA and ONL versions listed here.
Wire Up
Connect power cable, console port and management network.
C1. Install Open Networking Linux (ONL)
The switches are shipped with Open Networking Install Environment (ONIE) bootloader. After booting up, we should see the ONIE prompt from console.
Here we assume that the management port on the switch already has Internet access. (via DHCP)
Enter ONIE
Has no ONL or ONL 1.x installed
The way to install ONL 2.x is the same as ONL 1.x. However, if you have no ONL installed or have older version of ONL, you might find it tricky to (re)install a newer version. Here's the instruction:
- Plug in the console cable and reboot the switch
- (If your bootloader is grub) When you see the boot menu, select ONIE -> ONIE: Rescue
- (If your bootloader is uboot) When you see Hit any key to stop autoboot instead of the boot menu, press any key and then enter run onie_rescue to enter ONIE rescue mode.
Has ONL 2.x installed
It would be much more easier to reboot into ONIE if you have a previously installed ONL 2. Simply run
onl-onie-boot-mode install reboot
Available ONIE modes are: {install,rescue,uninstall,update,embed,diag,none}
This can be helpful when you are in a normal ONL and want to upgrade your system.
Install ONL
At the ONIE prompt, we need to get and install ONL. Depending on the switch model you are using, we need to use different installers from here:
For powerpc based 5710, use ONL-2.0.0-ONL-OS-DEB8-2016-12-22.1828-604af0c-PPC-INSTALLED-INSTALLER
For x86 based 5712 and 6712, use ONL-2.0.0-ONL-OS-DEB8-2016-12-22.1828-604af0c-AMD64-INSTALLED-INSTALLER
In the console, type
ONIE:/ # wget <installer-url> ONIE:/ # sh <installer-filename>
to fetch and install the latest ONL.
The switch will automatically reboot into ONL after installation. Default login credential of ONL is: root/onl
We might want to configure a fixed IP address for the management interface.
First edit /etc/network/interfaces and configure ma1, which is the management interface.
auto ma1 iface ma1 inet static address 10.128.10.128 netmask 255.255.0.0 gateway 10.128.0.1 dns-nameservers 192.168.1.1 8.8.8.8
C2. Install OF-DPA package
Please download the OFDPA 3.0 EA4 release debian package from:
For x86 based switches (eg. Accton 5712, 6712, 7712)
For powerpc based switches (eg. Accton 5710)
Copy the package to /mnt/onl/data
$ scp ofdpa_3.0.4.0+accton1.0~1-1_amd64.deb 10.128.10.128:/mnt/onl/data
Install the OF-DPA package.
# dpkg -i --force-overwrite /mnt/onl/data/ofdpa_3.0.4.0+accton1.0~1-1_amd64.deb
C3. (Optional) Switch Port Config
By default, all 32 ports are running in 1x40G mode. The /etc/accton/ofdpa.conf need to be modified if we want to break out 1x40G into 4x10G.
port_mode_1=4x10g # front port 1
Restart ofdpa service after modifying the config
# killall -15 ofagentapp # launcher ofagentapp -t <controller_ip1> -t <controller_ip2> -t <controller_ip3> -i <dpid>
C4. Connect Switch to ONOS
Start OFDPA and OpenFlow agent
The way to start OFDPA 3.0 EA3 is quite different from 3.0 EA0 and below.
The OFDPA software and the Indigo agent are now a single process. To start it, run
launcher ofagentapp -t <controller_ip1> -t <controller_ip2> -t <controller_ip3> -i <dpid>
To run it in debug mode:
launcher ofagentapp -t <controller_ip1> -t <controller_ip2> -t <controller_ip3> -i <dpid> -d4 -c1 -c2 -c3 -c4 -c5 -a1
To stop the agent, simply use "killall -15 ofagentapp"
More ofagentapp options can be found by running "ofagentapp --help"
Start OFDPA and OpenFlow agent with listen option
An optional -l parameter can be added for switch to listen on specific IP:port. The switch will listen on given IP:port and allow us to use tools like ovs-ofctl or dpctl to connect and control the switch.
launcher ofagentapp -t <controller_ip> -i <dpid> -l <ip>:<port>
To connect to the switch using ovs-ofctl, you can use commands such as
ovs-ofctl -O OpenFlow13 show tcp:<ip>:<port> ovs-ofctl -O OpenFlow13 dump-flows tcp:<ip>:<port> ovs-ofctl -O OpenFlow13 dump-groups tcp:<ip>:<port> ovs-ofctl -O OpenFlow13 add-flow tcp:<ip>:<port> table=60,priority=40000,eth_type=0x0800,ip_dst=55.55.55.55,actions=controller
For more command, please see ovs-ofctl --help or the post OpenvSwitch ovs-ofctl and OF-DPA
C5. Config Network Interface on the Host
We need to configure the IP address and route on the host.
Configure the IP address and subnet mask
Make sure the IP address and the subnet mask on the fabric network interface of the host is consistent with the information in the Network Configuration host section.
For example, you can runip addr add 10.0.0.1/24 dev mlx0
Configure the default route
Make sure you change the default route of the host to the interface IP of the leaf switch it connects to.
For example, you can runip route add default via 10.0.0.254
In the case that you want to keep default route through the management network, you need to add routes to all other subnets in the network one by one.
Verification
After the setup is completed, you should be able to ping from one host to another.
Other Useful Information
ONL Boot Mode
OFDPA 2 offers two boot mode: INSTALLED and SWI.
In INSTALLED mode, ONL mounts the root filesystem from /dev/sdb7 which is a flash drive that persists everything even during a power cycle.
In SWI mode, ONL will load the root filesystem from a read-only image (*.swi) and put the modified files in an copy-on-write overlay filesystem. The change will not be persisted during a power cycle.
The pros and cons are obvious. INSTALLED mode is more convenient while SWI mode is more error-safe.
The boot mode will be determined by which image we used install ONL. We use the installed mode in this instruction for simplicity. But we can still change it after the installation. Here's the instruction:
mount /mnt/onl/boot -o remount,rw sed -i 's/BOOTMODE=INSTALLED/BOOTMODE=SWI/g' /mnt/onl/boot/boot-config reboot
Persistence mechanism
INSTALLED mode already persists everything for you. This section is mainly for SWI mode.
The mechanism to persist files on the switch is different from ONL 1.x to ONL 2.x. Following are the steps required to persist a file (e.g. OFDPA package) in ONL 2.x.
- Files put in /mnt/onl/data will be persisted automatically.
- To install a deb package during start-up, put the deb file under /mnt/onl/data/install-debs folder and create a plain text file /mnt/onl/data/install-debs/list with all the filename of the deb (e.g. ofdpa-3.0-ea0_3.0.0.0+accton1.0-1_amd64.deb)
- To execute some command during start-up, put the command in /mnt/onl/data/rc.boot
# set static ip and route (dhcp by default) echo 'ip addr add 10.128.0.203/16 dev ma1' >> /mnt/onl/data/rc.boot echo 'ip route add default via 10.128.0.1' >> /mnt/onl/data/rc.boot # grant executable permission chmod a+x /mnt/onl/data/rc.boot
See PersistWorkflow.md for more detail.
Build a customized ONL image
Sometimes we need to build our own ONL image from source to include some extra files. (The most common case is a modified /etc/network/interfaces)
The instruction of build process has already been well-documented in ONL repo Building.md. But you might find it a little bit tricky to inject some extra files.
To do that, you need to put the files under $ONL_ROOT/builds/any/rootfs/jessie/common/overlay/ and then run the make command again.
Useful Commands
There are some useful commands under /usr/bin/
client_cfg_purge client_debugcomp client_drivshell client_flowtable_dump client_meter_dump client_port_table_dump client_tunnel_dump client_classcolortable_dump client_debuglvl client_event client_grouptable_dump client_oam_dump client_queue_config
Recovery from a faulty ONL install
If, for some reason, the ONL install process fails, you may be brought to the grub rescue prompt upon reboot. You may or may not find system files, or even basic grub rescue commands (i.e. 'help'). There are two options for returning to a (known working) ONIE prompt.
- Reinstaller - Accton provides a ISO for reinstalling the firmware to factory default. This can be acquired from Accton by registering for their online support, and applying for it here.
Once you have the ISO, you can either use PXE or a live USB to boot the switch into the image. Manually boot to ONIE - If the files are there, you can manually configure GRUB to load the necessary modules to boot back into ONIE.
# This should be the partition on the ONIE partition on the AS6712. # It might be hd1 or hd0. Change appropriately. grub rescue> ls (hd0,gpt2)/ ./ ../ lost+found/ onie/ grub/ grubenv # These instructions will load the kernel and ONIE initrd. grub rescue> set prefix=(hd0,gpt2)/grub grub rescue> set root=(hd0,gpt2) grub rescue> insmod normal grub rescue> insmod linux grub rescue> ls /onie ./ ../ vmlinuz-3.2.35-onie initrd.img-3.2.35-onie tools/ grub/ grub.d/ config/ # if the ",115200n8" is omitted, baud rate defaults to 9600. grub rescue> linux /onie/vmlinuz-3.2.35-onie console=tty0 console=ttyS1,115200n8 grub rescue> initrd /onie/initrd.img-3.2.35-onie # The system will not boot into ONIE and you can recover or re-install from there. grub rescue> boot
Once back at the ONIE prompt, you can try to install ONL by following the same steps again.