Adding initial trouble shooting E-CORD guide
Change-Id: Ib2faab5da61fbcf0a03be4b7523f667a7e7dd7f7
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
new file mode 100644
index 0000000..1fa42c2
--- /dev/null
+++ b/docs/troubleshooting.md
@@ -0,0 +1,159 @@
+# Troubleshooting
+
+Having troubles deploying or running E-CORD? You have come to the right place.
+
+## Reset the nodes
+
+Through these steps we will bring all your pods back to a clean state.
+You will need the reset scripts under:
+```
+CORD_ROOT/orchestration/profiles/ecord/examples/
+```
+### Global node
+Let's start with the global node.
+
+#### Reset Step
+* SSH into the global node
+* execute these commands
+ ```
+ cd CORD_ROOT/orchestration/profiles/ecord/examples/
+ chmod +x reset-global.sh
+ ./reset-global.sh YOUR_GLOBAL_NODE_IP YOUR_PROPER_PATH_TO_THE_GLOBAL_JSON_CONFIG_FILE
+ ```
+ an example of the last command is:
+ ```
+ ./reset-global.sh 127.0.0.1 ~/global-onos-fabric.json
+ ```
+
+#### Check Step
+Once the reset step is finished we can check if the state is what we expect.
+
+##### XOS
+* Login into the XOS UI at:
+ ```
+ http://<YOUR_GLOBAL_NODE_IP>/xos
+ ```
+* Click on the *vNaaS* tab on the left
+* Click on *Ethernet Virtual Lines* and make sure nothing is there
+* Click on *Bandwidth Profiles* and make sure there is at least one bandwidth profile
+* Click on *User Network Interfaces* and make sure there that your UNIs are present
+
+##### ONOS
+* Log into ONOS_CORD: `ssh -p 8102 onos@onos-cord`
+* In the ONOS CLI (onos>) verify that apps are loaded: `apps -a -s`
+ The list of applications enabled should include:
+ ```
+ org.opencord.ce.api
+ org.opencord.ce.global
+ ```
+
+
+### Local Node(s)
+Let's continue with all your local pods.
+
+#### Reset Step
+* SSH into the local head node
+* exectue these commands
+ ```
+ cd CORD_ROOT/orchestration/profiles/ecord/examples/
+ chmod +x reset-local.sh
+ ./reset-local.sh YOUR_GLOBAL_NODE_IP YOUR_PATH_TO_ONOS_FABRIC_JSON_CONFIG_FILE YOUR_PATH_TO_THE_ONOS_CORD_JSON_CONFIG_FILE
+ ```
+ an example of the last one is:
+ ```
+ ./reset-local.sh 127.0.0.1 ~/local-1-onos-fabric.json ~/local-1-onos-cord.json
+ ```
+
+#### Check Step
+Once the reset step is finished we can check if the state is what we expect.
+
+##### ONOS Fabric
+Let's start with the ONOS_CORD.
+* Log into ONOS_Fabric: `ssh -p 8101 onos@onos-fabric`
+* In the ONOS CLI (onos>) verify that apps are loaded: `apps -a -s`
+ The list of applications enabled should include:
+ ```
+ org.onosproject.segmentrouting
+ org.opencord.ce.api
+ org.opencord.ce.local.bigswitch
+ org.opencord.ce.local.channel.http
+ org.opencord.ce.local.fabric
+ ```
+* Check that the BigSwitchManager configuration is properly set by typing
+ ```
+ cfg get org.opencord.ce.local.bigswitch.BigSwitchManager
+ ```
+ The output should be:
+ ```
+ org.opencord.ce.local.bigswitch.BigSwitchManager
+ name=domainId, type=string, value=<YOUR_LOCAL_NODE_IP>-fabric-onos, defaultValue=local-domain, description=Domain ID where this ONOS is running
+ ```
+
+##### ONOS CORD
+Let's now take a look at the ONOS_CORD.
+
+* Log into ONOS_CORD: `ssh -p 8102 onos@onos-cord`
+* Verify that apps are loaded: `apps -a -s`
+ The list of applications enabled should include:
+ ```
+ org.onosproject.drivers.microsemi
+ org.onosproject.cfm
+ org.opencord.ce.api
+ org.opencord.ce.local.bigswitch
+ org.opencord.ce.local.channel.http
+ org.opencord.ce.local.vee
+ ```
+
+* Verify that at least 3 devices, a netconf one and two openflow are presentbyt typing `devices` in the ONOS CLI.
+ An example is:
+ ```
+ onos> devices
+ id=netconf:10.6.0.160:830, available=true, local-status=connected 34s ago, role=MASTER, type=SWITCH, mfr=Microsemi, hw=EA1000, sw=2.6.33-arm1-MSEA1000--00392-ge9ee017.1.0.0.96-dirty, serial=SJC162300029 , driver=microsemi-netconf, ipaddress=10.6.0.160, locType=geo, name=netconf:10.6.0.160:830, port=830, protocol=NETCONF
+ id=of:0000001e08095936, available=true, local-status=connected 1m53s ago, role=MASTER, type=SWITCH, mfr=2004-2015 Centec Networks Inc, hw=48T4X, sw=3.1.16.17.alpha, serial=E101ZB141004, driver=default, channelId=10.6.0.150:48073, managementAddress=10.6.0.150, protocol=OF_13
+ id=of:00003cfdfe9df039, available=true, local-status=connected 1m53s ago, role=MASTER, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.3.2, serial=None, driver=ovs, channelId=10.6.0.14:34793, managementAddress=10.6.0.14, protocol=OF_13
+ ```
+* Check that the BigswitchManagerConfiguration is properly set by typing
+ ```
+ cfg get org.opencord.ce.local.bigswitch.BigSwitchManager
+ ```
+ The output should be:
+ ```
+ org.opencord.ce.local.bigswitch.BigSwitchManager
+ name=domainId, type=string, value=<YOUR_LOCAL_NODE_IP>-fabric-onos, defaultValue=local-domain, description=Domain ID where this ONOS is running
+ ```
+* Check that the BigswitchManagerConfiguration is properly set by typing
+ ```
+ cfg get org.onosproject.netconf.ctl.impl.NetconfControllerImpl
+ ```
+ The output should be:
+ ```
+ org.onosproject.netconf.ctl.impl.NetconfControllerImpl
+ name=sshLibrary, type=string, value=apache_mina, defaultValue=apache_mina, description=Ssh Llbrary instead of Apache Mina (i.e. ethz-ssh2
+ name=netconfConnectTimeout, type=integer, value=120, defaultValue=5, description=Time (in seconds) to wait for a NETCONF connect.
+ name=netconfReplyTimeout, type=integer, value=120, defaultValue=5, description=Time (in seconds) waiting for a NetConf reply
+ ```
+
+### UNI Configuration assurance
+Before pushing the UNI there is one last set of checks to make to be sure the state is properly configured:
+* SSH into the global node
+* Log in the global ONOS: `ssh -p 8102 onos@onos-cord`
+* type `bigswitch-ports`
+* find the port where *mefPortType=UNI* and write down its port number.
+ An example is
+ ```
+ onos> bigswitch-ports
+ DefaultPortDescription{number=1, isEnabled=true, type=FIBER, portSpeed=1000, annotations={interlinkId=EE-1-to-fabric, portName=eth-0-50, mefPortType=INNI, portMac=00:1e:08:09:59:69, domainId=10.90.1.30-cord-onos}}
+ DefaultPortDescription{number=2, isEnabled=true, type=FIBER, portSpeed=1000, annotations={portName=Optics, mefPortType=UNI, domainId=10.90.1.30-cord-onos}}
+ ```
+ Where UNI port is number 2
+* log into the XOS UI.
+ ```
+ http://<YOUR_GLOBAL_NODE_IP>/xos
+ ```
+* Click on the vNaaS tab on the left
+* Click on User Network Interfaces
+* Please make sure the port number (the number after the / in the Cpe id) matches what you wrote down.
+ If not, press the hour glass at the far right and edit the number after the / making it match yours. Then on the bottom right click the save and you are done.
+
+## RESET Successful
+At this point you should be again in a clean state with all the configuration properly pushed.
diff --git a/examples/reset-global b/examples/reset-global
new file mode 100644
index 0000000..1c8ec0e
--- /dev/null
+++ b/examples/reset-global
@@ -0,0 +1,34 @@
+#!/bin/bash
+# -----------------------------------------------------------------------------
+# Resets the E-CORD Global Pod
+# -----------------------------------------------------------------------------
+
+password=`cat /opt/credentials/xosadmin@opencord.org`
+ip=${1:-localhost}
+cord_json=${2:-wrong-path}
+
+docker exec -i ecordglobal_xos_db_1 psql <<-EOF
+
+\connect xos
+
+DELETE FROM vnaas_eline;
+
+EOF
+
+echo Resetting onos_cord
+cd /opt/onos_cord
+
+docker-compose down
+
+sleep 5
+
+docker-compose up -d
+
+sleep 20
+
+echo sending tosca file
+python /opt/cord/build/platform-install/scripts/run_tosca.py 9000 xosadmin@opencord.org ${password} /opt/cord_profile/carrierethernet-global-app.yaml
+
+echo sending json config onos_cord
+
+curl -X POST -H "content-type:application/json" http://${ip}:8182/onos/v1/network/configuration -d @${cord_json} --user onos:rocks
diff --git a/examples/reset-local b/examples/reset-local
new file mode 100644
index 0000000..87d5969
--- /dev/null
+++ b/examples/reset-local
@@ -0,0 +1,52 @@
+#!/bin/bash
+# -----------------------------------------------------------------------------
+# Resets a E-CORD Local Pod
+# -----------------------------------------------------------------------------
+
+password=`cat /opt/credentials/xosadmin@opencord.org`
+ip=${1:-localhost}
+fabric_json=${2:-wrong-path}
+cord_json=${3:-wrong-path}
+
+echo Resetting onos_fabric
+cd /opt/onos_fabric
+
+docker-compose down
+
+sleep 5
+
+docker-compose up -d
+
+
+echo Resetting onos_cord
+cd /opt/onos_cord
+
+docker-compose down
+
+sleep 5
+
+docker-compose up -d
+
+sleep 20
+
+echo sending tosca file
+python /opt/cord/build/platform-install/scripts/run_tosca.py 9000 xosadmin@opencord.org ${password} /opt/cord_profile/fabric-service.yaml
+python /opt/cord/build/platform-install/scripts/run_tosca.py 9000 xosadmin@opencord.org ${password} /opt/cord_profile/vtn-service.yaml
+
+sleep 2
+
+python /opt/cord/build/platform-install/scripts/run_tosca.py 9000 xosadmin@opencord.org ${password} /opt/cord_profile/carrierethernet-local-app.yaml
+
+sleep 60
+
+echo sending json config onos_fabric
+
+curl -X POST -H "content-type:application/json" http://${ip}:8181/onos/v1/network/configuration -d @${fabric_json} --user onos:rocks
+
+echo sending json config onos_cord
+
+curl -X POST -H "content-type:application/json" http://${ip}:8182/onos/v1/network/configuration -d @${cord_json} --user onos:rocks
+
+sleep 45
+
+curl -X POST -H "content-type:application/json" http://${ip}:8182/onos/v1/network/configuration -d @${cord_json} --user onos:rocks
\ No newline at end of file