Guide on data plane debugging
Change-Id: I6f9d962fc203e953986a07f3a445fae0d62b5cd5
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 1fa42c2..729acab 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -157,3 +157,66 @@
## RESET Successful
At this point you should be again in a clean state with all the configuration properly pushed.
+
+## Data Plane Debugging
+So you have successfully set up the EVC in the CORD UI. And yet when
+you try to ping, traffic does not seem to go through. Here we will
+walk you through the different steps to check if and where the data
+path is failing.
+
+1. First you want to check the flow rules on the MicroSemi SFP.
+To do that, log into the ONOS CORD instance on the pod. You can use
+the UI (port 8182) or the CLI (port 8102). You should see a flow rule
+that matches on the c-tag (you specified it when creating the EVC) and
+pushes the s-tag (this is automatically allocated) on top. A second rule
+reverses this process: match on s-tag and pop it off. This is all we
+can verify for this part of the data path, as the Microsemi driver
+does not support port or flow stats (thus we cannot confirm if the
+packets are actually hitting the device).
+
+2. The next step is to verify the Ethernet edge (Centec in our case)
+ switch. This one is also controlled by the same ONOS CORD
+ instance from the previous step. First, check if a meter has been
+ installed on the switch; the parameters should match the bandwidth
+ profile you selected when provisioning the EVC. Next, you should
+ also see a flow rule that matches on the s-tag and outputs to the
+ fabric port. A second flow rule does the same for the reverse path.
+ You can also look at the port counters of the Ethernet edge and
+ confirm that both the input port (where the SFP is connected) and
+ the output port (where the fabric is connected) are steadily incrementing
+ as you'd expect from your ping.
+
+3. To check the fabric data path, start by connecting to the ONOS
+ FABRIC instance. This time you will need to connect to the CLI
+ (port 8101). First check if the VLAN cross connect is properly
+ installed, by checking the output of the ```netcfg```
+ command. Search for the *xconnect* configuration for
+ *org.onosproject.segmentrouting*, then verify (1) the given vlan
+ corresponds to the s-tag, and (2) the given ports connect the
+ fabric to the Ethernet edge and to the transport/next
+ CO. Afterwards, check the output of the ```flows``` command, where
+ you can see how the VLAN cross connect gets instantiated in the
+ switch's tables. Again, check the input/output ports and the VLAN
+ ID. You can also relate this to the output of the ```groups```
+ command. An example output is given below.
+
+```
+onos> flows
+deviceId=of:0000cc37ab7cc2fa, flowRuleCount=7
+ id=7300003f50706b, state=ADDED, bytes=0, packets=0, duration=14, liveType=UNKNOWN, priority=32768, tableId=10, appId=org.onosproject.segmentrouting, payLoad=null, selector=[IN_PORT:37, VLAN_VID:1], treatment=DefaultTrafficTreatment{immediate=[], deferred=[], transition=TABLE:20, meter=None, cleared=false, metadata=null}
+ id=7300004524fea3, state=ADDED, bytes=0, packets=0, duration=14, liveType=UNKNOWN, priority=32768, tableId=10, appId=org.onosproject.segmentrouting, payLoad=null, selector=[IN_PORT:30, VLAN_VID:1], treatment=DefaultTrafficTreatment{immediate=[], deferred=[], transition=TABLE:20, meter=None, cleared=false, metadata=null}
+ id=730000009feee7, state=ADDED, bytes=0, packets=0, duration=13, liveType=UNKNOWN, priority=1000, tableId=50, appId=org.onosproject.segmentrouting, payLoad=null, selector=[VLAN_VID:1], treatment=DefaultTrafficTreatment{immediate=[], deferred=[GROUP:0x40010000], transition=TABLE:60, meter=None, cleared=false, metadata=null}
+ id=1000082927ca9, state=ADDED, bytes=283894, packets=3336, duration=4911, liveType=UNKNOWN, priority=40000, tableId=60, appId=org.onosproject.core, payLoad=null, selector=[ETH_TYPE:lldp], treatment=DefaultTrafficTreatment{immediate=[OUTPUT:CONTROLLER], deferred=[], transition=None, meter=None, cleared=true, metadata=null}
+ id=10000a168723c, state=ADDED, bytes=269365, packets=3169, duration=4911, liveType=UNKNOWN, priority=40000, tableId=60, appId=org.onosproject.core, payLoad=null, selector=[ETH_TYPE:bddp], treatment=DefaultTrafficTreatment{immediate=[OUTPUT:CONTROLLER], deferred=[], transition=None, meter=None, cleared=true, metadata=null}
+ id=73000030665163, state=ADDED, bytes=0, packets=0, duration=4911, liveType=UNKNOWN, priority=40000, tableId=60, appId=org.onosproject.segmentrouting, payLoad=null, selector=[ETH_TYPE:ipv6, IP_PROTO:58], treatment=DefaultTrafficTreatment{immediate=[OUTPUT:CONTROLLER], deferred=[], transition=None, meter=None, cleared=false, metadata=null}
+ id=73000030db30ed, state=ADDED, bytes=0, packets=0, duration=4911, liveType=UNKNOWN, priority=40000, tableId=60, appId=org.onosproject.segmentrouting, payLoad=null, selector=[ETH_TYPE:arp], treatment=DefaultTrafficTreatment{immediate=[OUTPUT:CONTROLLER], deferred=[], transition=None, meter=None, cleared=false, metadata=null}
+onos> groups
+deviceId=of:0000cc37ab7cc2fa, groupCount=3
+ id=0x1001e, state=ADDED, type=INDIRECT, bytes=0, packets=0, appId=org.onosproject.segmentrouting
+ id=0x1001e, bucket=1, bytes=0, packets=0, actions=[OUTPUT:30]
+ id=0x10025, state=ADDED, type=INDIRECT, bytes=0, packets=0, appId=org.onosproject.segmentrouting
+ id=0x10025, bucket=1, bytes=0, packets=0, actions=[OUTPUT:37]
+ id=0x40010000, state=ADDED, type=ALL, bytes=0, packets=0, appId=org.onosproject.segmentrouting
+ id=0x40010000, bucket=1, bytes=0, packets=0, actions=[GROUP:0x10025]
+ id=0x40010000, bucket=2, bytes=0, packets=0, actions=[GROUP:0x1001e]
+```