In the previous article #003 "Are you P4 compliant ?" we exposed a setup where RARE/freeRouter was controlling BMv2 P4 dataplane called simple_switch_grpc. In this article we replace the open source BMv2 target by a commercial virtual target provided by INTEL/BAREFOOT. As a side note, we will show that this setup can be integrated with real networks. (with inherent software limitations) 

Requirement

  • Basic Linux/Unix knowledge
  • Basic networking knowledge

Overview

I'm repeating the core message from #003: For those who are not familiar with data plane programming and especially with P4, "P4 is a domain-specific programming language for specifying the behaviour of the dataplanes of network-forwarding elements." (from p4.org) in short it helps you to write a "program specifying how a switch processes packets".

Article objective

In this article we'll using freeRouter setup deployed in #003 and replace bmv2/simple_switch_grpc providing freeRouter P4Lang's dataplane by INTEL BAREFOOT/bf_switchd. Actually the effective dataplane is ensured by INTEL/BAREFOOT virtual bf_switchd model running RARE P4 program called: bf_router.p4.

Diagram

[ #003 ] - Cookbook

In our example we will use the OpenNetworkLinux KVM image (ONL9) this is the recommended build from INTEL/BAREFOOT for SDE-9.2.0.

and we add a bridge network interface to or laptop RJ45 connection.



mkdir -p ~/freeRouter/bin ~/freeRouter/lib ~/freeRouter/etc ~/freeRouter/log
cd ~/freeRouter/lib
wget http://freerouter.nop.hu/rtr.jar


╭─[11:11:54]floui@debian ~ 
╰─➤ tree freeRouter
freeRouter
├── bin   # binary files      
├── etc   # configuration files      
├── lib   # library files      
└── log   # log files      






wget freerouter.nop.hu/rtr.tar



tar xvf rtr.tar -C ~/freeRouter/bin/


For those you would like to rebuild these binaries you can find the compilation shell script in freeRouter cloned git repository in: ~/freeRouter/src/native/c.sh



In that section, you'll need to get access to INTEL/BAREFOOT Software Development Environment. For Research & Academia institution, you can apply here in order to become a FASTER member and access to INTEL/BAREFOOT resources. You can find here, a document installing INTEL/BAREFOOT SDE on ONL for a WEDGE100BF32X system. In our case, we are setting up the following environment:

  • ONL9 as VM guest with kernel 8192 Mb of RAM abd 2 vCPU
  • SDE 9.2.0
  • VirtualBox is running on MACOSX host

Just for the sake of example, SDE 9.2.0 is installed in root home directory:

export SDE=/root/bf-sde-9.2.0
export SDE_INSTALL=/root/bf-sde-9.2.0/install
export PATH=$PATH:$SDE_INSTALL/bin:$SDE/tools

bitbucket is a private repository. It is currently being reworked in order to make it public as per INTEL/BAREFOOT decision to grant P4 code related to TOFINO architecture. It is thus inaccessible for now but will be soon opened.

cd ~/
git clone https://bitbucket.software.geant.org/scm/rare/rare.git


root@bionic:~/rare/p4src# p4_build.sh -I /root/rare/p4src/ -DHAVE_MPLS /root/rare/p4src/bf_router.p4 
Using SDE          /root/bf-sde-9.2.0
Using SDE_INSTALL /root/bf-sde-9.2.0/install
Using SDE version bf-sde-9.2.0

OS Name: Ubuntu 18.04.4 LTS
This system has 8GB of RAM and 1 CPU(s)
Parallelization:  Recommended: -j1   Actual: -j1

Compiling for p4_16/tna
P4 compiler path:    /root/bf-sde-9.2.0/install/bin/p4c
P4 compiler version: 9.2.0 (SHA: 639d9ec) (p4c-based)
Build Dir: /root/bf-sde-9.2.0/build/p4-build/bf_router
 Logs Dir: /root/bf-sde-9.2.0/logs/p4-build/bf_router

  Building bf_router        CLEAR CONFIGURE MAKE INSTALL ... DONE



FreeRouter uses 2 configuration files in order to run, let's write these configuration files for R1 in ~/freeRouter/etc

int eth0 eth 0000.1111.00fb 127.0.0.1 22710 127.0.0.1 22709
tcp2vrf 2323 v1 23
tcp2vrf 9080 v1 9080


hostname p4-freerouter
buggy
!
!
vrf definition v1
 exit
!
interface ethernet0
 description freerouter@P4_CPU_PORT[veth251]
 no shutdown
 no log-link-change
 exit
!
interface sdn1
 description freerouter@sdn1[enp0s3]
 mtu 9000
 macaddr 0072.3e18.1b6f
 vrf forwarding v1
 ipv4 address 192.168.0.131 255.255.255.0
 ipv6 address 2a01:e0a:159:2850::666 ffff:ffff:ffff:ffff::
 ipv6 enable
 no shutdown
 no log-link-change
 exit
!
!
!
!
!
!
!
!
!
!
!
!
!
!
server telnet tel
 security protocol telnet
 no exec authorization
 no login authentication
 vrf v1
 exit
!
server p4lang p4
 export-vrf v1 1
 export-port sdn1 0 10
 interconnect ethernet0
 vrf v1
 exit
!
client tcp-checksum transmit
!
end




echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6

ip link add veth251 type veth peer name veth250
ip link set veth250  up 
ip link set veth251  up 

ifconfig enp0s3 promisc
ifconfig veth250 promisc
ifconfig veth251 promisc

ip link set dev veth250 up mtu 10240
ip link set dev veth251 up mtu 10240
ip link set dev enp0s3 up mtu 10240
export TOE_OPTIONS="rx tx sg tso ufo gso gro lro rxvlan txvlan rxhash"

for TOE_OPTION in $TOE_OPTIONS; do
    /sbin/ethtool --offload veth250 "$TOE_OPTION" off &> /dev/null
    /sbin/ethtool --offload veth251 "$TOE_OPTION" off &> /dev/null
    /sbin/ethtool --offload enp0s3 "$TOE_OPTION" off &> /dev/null

sudo ip link add veth251 type veth peer name veth250
sudo ip link set veth250 up  
sudo ip link set veth251 up  


╭─[6:06:13]floui@debian ~/freeRouter  
╰─➤  java -jar lib/rtr.jar routersc etc/p4-freerouter-hw.txt etc/p4-freerouter-sw.txt
info cfg.cfgInit.doInit:cfgInit.java:556 booting
info cfg.cfgInit.doInit:cfgInit.java:680 initializing hardware
info cfg.cfgInit.doInit:cfgInit.java:687 applying defaults
info cfg.cfgInit.doInit:cfgInit.java:695 applying configuration
info cfg.cfgInit.doInit:cfgInit.java:721 done
welcome
line ready
freerouter#                   


╭─[1:00:53]floui@debian[1]  ~/freeRouter/bin  
╰─➤  sudo ./pcapInt.bin veth251 22709 127.0.0.1 22710 127.0.0.1
binded to local port 127.0.0.1 22709.
will send to 127.0.0.1 22710.
pcap version: libpcap version 1.8.1
opening interface veth251 with pcap1.x api
serving others
> 


export P4_RARE_ROOT=/home/floui/RARE/02-PE-labs
sudo simple_switch_grpc --log-file $P4_RARE_ROOT/run/log/p4-freerouter.log \                                               
                        -i 1@enp0s9 \                        
                        -i 64@veth250 \                                                
                        --thrift-port 9090 --nanolog ipc://$P4_RARE_ROOT/run/bm-0-log.ipc --device-id 0 $P4_RARE_ROOT/build/simple_switch_grpc.json \
                        -- --grpc-server-addr 127.0.0.1:50051 > $P4_RARE_ROOT/run/log/p4-freerouter.out 2>&1 &                  


╭─[2:07:10]floui@debian[1]  ~/RARE/02-PE-labs/p4src  ‹master*› 
╰─➤  ./forwarder.py
rx:  ['myaddr4_add', '224.0.0.0/4', '0', '1', '\n']
rx:  ['myaddr4_add', '255.255.255.255/32', '0', '1', '\n']
rx:  ['myaddr6_add', 'ff00::/8', '0', '1', '\n']
rx:  ['myaddr4_add', '192.168.1.0/24', '-1', '1', '\n']
rx:  ['myaddr4_add', '192.168.1.131/32', '-1', '1', '\n']
rx:  ['myaddr6_add', 'fd7d:a59c:650b::/60', '-1', '1', '\n']
rx:  ['myaddr6_add', 'fd7d:a59c:650b::666/128', '-1', '1', '\n']
rx:  ['myaddr6_add', 'fe80::/64', '-1', '1', '\n']
rx:  ['mylabel6_add', '270549', '1', '\n']
rx:  ['mylabel4_add', '606864', '1', '\n']
rx:  ['state', '1', '1', '0', '\n']
rx:  ['mtu', '1', '9000', '\n']
rx:  ['portvrf_add', '1', '1', '\n']
rx:  ['neigh4_add', '14252', '192.168.1.1', '34:ce:00:67:18:c2', '1', '00:72:3e:18:1b:6f', '1', '\n']
rx:  ['neigh4_add', '52194', '192.168.1.143', '9c:eb:e8:d5:2c:51', '1', '00:72:3e:18:1b:6f', '1', '\n']
rx:  ['keepalive', '\n']
rx:  ['keepalive', '\n']
...


Verification


╭─[7:07:41]floui@debian[1]  ~/freeRouter/etc  
╰─➤  telnet localhost 2323
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
welcome
line ready
freerouter#




p4-freerouter#sh run                                                           
hostname p4-freerouter
buggy
!
vrf definition v1
 exit
!
interface ethernet0
 description freerouter@P4_CPU_PORT[veth251]
 no shutdown
 no log-link-change
 exit
!
interface sdn1
 description freerouter@sdn1[enp0s9]
 mtu 9000
 macaddr 0072.3e18.1b6f
 vrf forwarding v1
 ipv4 address 192.168.1.131 255.255.255.0
 ipv6 address fd7d:a59c:650b::666 ffff:ffff:ffff:fff0::
 ipv6 enable
 no shutdown
 no log-link-change
 exit
!
!
!
!
!
!
!
!
!
!
!
!
!
!
server telnet tel
 security protocol telnet
 no exec authorization
 no login authentication
 vrf v1
 exit
!
server p4lang p4
 export-vrf v1 1
 export-port sdn1 1 0
 interconnect ethernet0
 vrf v1
 exit
!
client tcp-checksum transmit
!
end


p4-freerouter#show interfaces summary                                          
interface  state  tx     rx    drop
ethernet0  up     10616  9243  0
sdn1       up     10340  9069  0




p4-freerouter#ping 192.168.1.131 /vrf v1                                       
pinging 192.168.1.131, src=null, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/0/0/0
p4-freerouter#ping 192.168.1.1 /vrf v1                                         
pinging 192.168.1.1, src=null, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=3/5/10/28


p4-freerouter#sh ipv4 arp sdn1                                                 
mac             address        time      static
34ce.0067.18c2  192.168.1.1    00:00:35  false
9ceb.e8d5.2c51  192.168.1.143  00:00:35  false
6420.0c65.437b  192.168.1.173  00:01:35  false                                                     

p4-freerouter#ping fd7d:a59c:650b::666 /vrf v1                                 
pinging fd7d:a59c:650b::666, src=null, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/0/0/1
p4-freerouter#ping fd7d:a59c:650b::1 /vrf v1                                   
pinging fd7d:a59c:650b::1, src=null, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=2/2/3/13                                                                                                       


p4-freerouter#show ipv6 neighbors sdn1                                         
mac             address                    time      static  router
34ce.0067.18c2  fd7d:a59c:650b::1          00:01:22  false   false
9ceb.e8d5.2c51  fe80::10e6:87a7:6a9:f14a   00:01:22  false   false
34ce.0067.18c2  fe80::36ce:ff:fe67:18c2    00:01:22  false   false
b6be.fdcf.d0f9  fe80::b4be:fdff:fecf:d0f9  00:01:22  false   false


p4-freerouter#ssh 192.168.1.1 /vrf v1 /user root                               
 - connecting to 192.168.1.1 22
password: ***************
                
 - securing connection



BusyBox v1.28.4 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 18.06.2, r7676-cddd7b4c77
 -----------------------------------------------------
root@OpenWrt:~# 


p4-freerouter#ssh fd7d:a59c:650b::1 /vrf v1 /user root                         
 - connecting to fd7d:a59c:650b::1 22
password: ***************
                
 - securing connection



BusyBox v1.28.4 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 18.06.2, r7676-cddd7b4c77
 -----------------------------------------------------


Conclusion

In this article you:

  • FreeRouter is using UNIX socket in order to forward packet dedicated to control plane + dataplane communication.

This essential paradigm is used to ensure communication between freeRouter and BMv2 P4 dataplane. It is ensured by pcapInt binary from freeRouter net-tools that will bind freeRouter socket (veth251@locathost:22710) to a virtual network interface (veth250@localhost:22709)  connected to CPU_PORT 64.

  • freeRouter control plane and dataplane communication is enabled by RARE forwarder.py 

forwarder.py is a simple python script based on GRPC P4Runtime python library.

freeRouter is doing all the control plane route computation and write/modify/remove message entry via P4Runtime so that P4 entries are created/modified/removed accordingly from P4 tables

While BMv2 target is a very good choice for packet processing algorithm validation, it is not an ideal target for production use. We will see in next articles how we can reach a higher rate throughput related required by use cases defined by network operators.