Merge "Runtime configuration guide"
diff --git a/README.md b/README.md
index 8f18a6e..86214fa 100644
--- a/README.md
+++ b/README.md
@@ -17,9 +17,10 @@
consider running a complete system entirely emulated in software using
[SEBA-in-a-Box](./profiles/seba/siab-overview.md).
-If you are anxious to jump straight to a [Quick Start](quickstart.md)
-procedure that brings up a subset of the CORD platform running
-on your laptop (without a subscriber data plane), that too is an option.
+If you prefer a gentle walk through of process bringing up a subset
+of the CORD platform running on your lapto (e.g., to get an
+introduction to all the moving parts in CORD) then jumping to the
+[Quick Start](quickstart.md) page is also an option.
Finally, if you want to get a broader lay-of-the-land, you
-might step back and start with an [Overview](overview.md).
+might step back and read the [Overview](overview.md).
diff --git a/SUMMARY.md b/SUMMARY.md
index 30fb9c5..6c31ff3 100644
--- a/SUMMARY.md
+++ b/SUMMARY.md
@@ -17,13 +17,11 @@
* [Single Node](prereqs/k8s-single-node.md)
* [Multi-Node](prereqs/k8s-multi-node.md)
* [Helm](prereqs/helm.md)
- * [Optional Packages](prereqs/optional.md)
- * [OpenStack](prereqs/openstack-helm.md)
* [Fabric Software Setup](fabric-setup.md)
* [Install Platform](platform.md)
* [Install Profile](profiles.md)
* [Offline Install](offline-install.md)
- * [Attach containers to external NICs](operating_cord/veth_intf.md)
+ * [Attach Container to a NIC](operating_cord/veth_intf.md)
* [Operations Guide](operating_cord/operating_cord.md)
* [General Info](operating_cord/general.md)
* [GUI](operating_cord/gui.md)
@@ -40,11 +38,7 @@
* [Getting the Source Code](developer/getting_the_code.md)
* [Modeling Services](developer/xos-intro.md)
* [Developer Workflows](developer/workflows.md)
- * [Working on R-CORD Without an OLT/ONU](developer/configuration_rcord.md)
* [Building Docker Images](developer/imagebuilder.md)
- * [Example Services](examples/examples.md)
- * [SimpleExampleService](simpleexampleservice/simple-example-service.md)
- * [ExampleService](exampleservice/example-service.md)
* [GUI Development](xos-gui/developer/README.md)
* [Quickstart](xos-gui/developer/quickstart.md)
* [GUI Extensions](xos-gui/developer/gui_extensions.md)
@@ -86,11 +80,11 @@
* [Try a new version of *](profiles/seba/operate/k8s-helm-ops.md)
* [Known Issues](profiles/seba/known-issues.md)
* Troubleshoot (FAQ)
- * [I can't ping](profiles/seba/troubleshoot/no-ping.md)
* [No ONUs discovered](profiles/seba/troubleshoot/no-onus.md)
+ * [No AttWorkflowDriver service instances](profiles/seba/troubleshoot/no-att-si.md)
* [I can't authenticate](profiles/seba/troubleshoot/no-aaa.md)
* [DHCP is not working](profiles/seba/troubleshoot/no-dhcp.md)
- * [No AttWorkflowDriver service instances](profiles/seba/troubleshoot/no-att-si.md)
+ * [I can't ping](profiles/seba/troubleshoot/no-ping.md)
* [M-CORD](profiles/mcord/README.md)
* Install
* [Install M-CORD Charts](profiles/mcord/install.md)
@@ -114,6 +108,7 @@
* [Kafka](charts/kafka.md)
* [Base Kubernetes](charts/base-kubernetes.md)
* [Base OpenStack](charts/base-openstack.md)
+ * [OpenStack](prereqs/openstack-helm.md)
* [VTN Setup](prereqs/vtn-setup.md)
* [R-CORD](charts/rcord.md)
* [M-CORD](charts/mcord.md)
@@ -124,3 +119,4 @@
* [PONNET](charts/ponnet.md)
* [PONSIM v2](charts/ponsimv2.md)
* [PONSIM Pod](charts/ponsim-pod.md)
+
diff --git a/developer/workflows.md b/developer/workflows.md
index f57f946..99b39f1 100644
--- a/developer/workflows.md
+++ b/developer/workflows.md
@@ -23,26 +23,19 @@
Once both Helm and Minikube are installed, you can deploy the
core components of XOS, along with the services that make
-up, for example, the R-CORD profile. This uses images published
+up, for example, the SEBA profile. This uses images published
on DockerHub:
```shell
cd ~/cord/helm-charts
```
-In this folder you can choose from the different charts which one to deploy.
-For example to deploy R-CORD you can follow [this guide](../profiles/rcord/install.md)
-
-### Deploy a Single Instance of Kafka
-
-Some profiles require a `kafka` message bus to work properly.
-If you need to deploy it for development purposes, a single instance
-deployment will be enough. You can do so as follows:
-
-```shell
-helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
-install --name cord-kafka incubator/kafka -f examples/kafka-single.yaml
-```
+In this folder you can choose from the different charts which one to
+deploy. For example to deploy SEBA you can follow
+[these instructions](../profiles/seba/install.md). Alternatively, if
+you are working on a new profile or a new service that is not part of
+any existing profile, you can install just the
+[CORD Platform](../platform.md).
## Making and Deploying Changes
@@ -93,14 +86,14 @@
> the containers, you can just delete the corresponding POD and kubernetes will
> restart it with the new image.
-## Pushing Changes to a docker registry
+## Pushing Changes to a Docker Registry
If you have a remote POD that you want to test your changes on, you
need to push your docker images to a docker registry that can be accessed
from the POD.
The way we recommend doing this is via a private docker registry.
-You can find more informations about what a
-docker registry is in the [offline installation section](../offline-install.md).
+You can find more information about what a docker registry is in the
+[offline installation section](../offline-install.md).
{% include "/partials/push-images-to-registry.md" %}
diff --git a/offline-install.md b/offline-install.md
index dc209ef..3ead38b 100644
--- a/offline-install.md
+++ b/offline-install.md
@@ -1,8 +1,10 @@
# Offline Install
-Often, CORD PODs' (management networks) don't have access to Internet.
+In many cases, a CORD POD (specifically, its management network) does
+not have access to Internet.
-This section of the guide provides guidelines, best-practices and examples to deploy the CORD/SEBA software without Internet connectivity.
+This section provides guidelines, best-practices, and examples to
+deploy CORD software without Internet connectivity.
> NOTE: The guide assumes that the Operating Systems (both on servers and on network devices) and Kubernetes are already installed and running.
@@ -10,39 +12,57 @@
* When the CORD POD has no access to Internet, so artifacts used for the installation (i.e. Docker images) cannot be downloaded directly to the POD.
-* While developing, you may want to test your changes pushing artifacts to the POD, before uploading them to the official docker repository.
+* While developing, you may want to test your changes by pushing artifacts to the POD, before uploading them to the official docker repository.
-## Target Infrastructure, requirements overview
+## Target Infrastructure / Requirements Overview
Your target infrastructure (where the POD runs) needs
-* A **local Docker Registry** where to push your Docker images (previously pulled from the web). If you don't have one, follow the notes below to deploy with helm a local Docker registry on top of your existing Kubernetes cluster.
+* **Local Docker Registry:** To push your Docker images
+ (previously pulled from the web). If you don't have one, follow the
+ notes below to use Helm to deploy a local Docker registry on top
+ of your existing Kubernetes cluster.
> More informations about docker registries can be found at <https://docs.docker.com/registry/>.
-* A **local webserver** to host the ONOS applications (.oar files), that are instead normally downloaded from Sonatype. If you don't have one, follow the notes below to quickly deploy a webserver on top of your existing Kubernetes cluster.
+* **Local Webserver:** To host the ONOS applications (.oar files),
+ that are instead normally downloaded from Sonatype. If you don't
+ have one, follow the notes below to quickly deploy a webserver on
+ top of your existing Kubernetes cluster.
+
+* **Kubernetes Servers Default Gateway:** For `kube-dns` to
+ work, a default route (even pointing to a non-exisiting/working
+ gateway) needs to be set on the machines hosting Kubernetes. This is
+ something related to Kubernetes, not to CORD.
-* **Kubernetes servers default gateway**: In order for kube-dns to work, a default route (even pointing to a non-exisiting/working gateway) needs to be set on the machines hosting Kubernetes. This is something related to Kubernetes, not to CORD.
-
-## Prepare the offline installation
+## Prepare the Offline Installation
This should be done from a machine that has access to Internet.
-* Clone the *helm-charts* repository
+* Add the *cord* repository to the list of your local repositories and download the repository index.
```shell
-git clone https://gerrit.opencord.org/helm-charts
+helm repo add cord https://charts.opencord.org
+helm repo update
```
-Next steps largely depend on the type of profile you want to install.
+* Add other third-party helm repositories used by CORD and pull external dependencies.
-* Add external helm repositories and pull external dependencies.
+* Fetch locally all the charts from the remote repositories
-* Fetch helm-charts not available locally.
+```shell
+helm fetch name-of-the-repo/name-of-the-chart --untar
+```
-* Modify the CORD helm charts to instruct kubernetes to pull the images from your local registry (where images will be pushed), instead of DockerHub. One option is to modify the *values.yaml* in each chart. A better option consists in extending the charts, rather than directly modifying them. This way, the original configuration can be kept as is, just overriding some values as needed. You can do this by writing your additional configuration yaml file, and parsing it as needed, adding `-f my-additional-config.yml` while using the helm install/upgrade commands. The full CORD helm charts reference documentation is available [here](../charts/helm.md).
+* Fetch (where needed) the chart dependencies
-* Download the ONOS applications (OAR files). Informations about the oar applications used can be found here: <https://github.com/opencord/helm-charts/blob/master/xos-profiles/att-workflow/values.yaml>
+```shell
+helm dep update name-of-the-repo/name-of-the-chart
+```
+
+* Create a file to override the default values of the charts, in order to instruct Kubernetes to pull the Docker images from your local registry (where images will be pushed) instead of DockerHub, and the ONOS images from the local webserver. One option is to modify the *values.yaml* files of each chart. A better option consists in extending the charts, rather than directly modifying them. This way, the original configuration can be kept as is, and just some values can be override as needed. You can do this by writing your additional configuration yaml file, and parsing it adding `-f my-additional-config.yml` while using the helm install/upgrade commands. The full CORD helm charts reference documentation is available [here](../charts/helm.md).
+
+* Download the ONOS applications (OAR files) used in your profile. For SEBA, this can be found here: <https://github.com/opencord/helm-charts/blob/master/xos-profiles/seba-services/values.yaml>
* Pull from DockerHub all the Docker images that need to be used on your POD. The *automation-tools* repository has a *images_from_charts.sh* utility inside the *developer* folder that can help you to get all the image names given the helm-chart repository and a list of chart names. More informations in the sections below.
@@ -50,41 +70,61 @@
* Optionally download the OpenOLT driver deb files from your vendor website (i.e. EdgeCore).
-* Optionally, save as tar files the Docker images downloaded. This can be useful if you'll use a different machine to upload the images on the local registry running in your infrastructure. To do that, for each image use the Docker command.
+* Optionally, save the Docker images downloaded as tar files. This can be useful if you'll use a different machine to upload the images on the local registry running in your infrastructure. To do that, for each image use the Docker command.
```shell
docker save IMAGE_NAME:TAG > FILE_NAME.tar
```
-* If the artifacts need to be deployed to the target infrastructure from a different machine, save the helm-charts directory, the ONOS applications, the docker images downloaded and the additional helm charts variable extension file.
+* If the artifacts need to be deployed to the target infrastructure
+ from a different machine, save the helm-charts directory, the ONOS
+ applications, the docker images downloaded and the additional helm
+ charts variable extension file.
-## Deploy the artifacts to your infrastructure and install CORD/SEBA
+## Deploy the Artifacts to Your Infrastructure
-This should not require any Internet connectivity. To deploy the artifacts to your POD, do the following from machine that has access to your Kubernetes cluster:
+This should not require any Internet connectivity. To deploy the
+artifacts to your POD, do the following from machine that has access
+to your Kubernetes cluster:
-* Optionally, if at the previous step you saved the Docker images on an external hard drive as .tar files, restore them in the deployment machine Docker registry. For each image (file), use the docker command
+* Optionally, if at the previous step you saved the Docker images on
+ an external hard drive as .tar files, restore them in the deployment
+ machine Docker registry. For each image (file), use the Docker
+ command:
```shell
docker load < FILE_NAME.tar
```
-* Tag and push your Docker images to the local Docker registry running in your infrastructure. More info on this can be found in the paragraph below.
+* Tag and push your Docker images to the local Docker registry running
+ in your infrastructure. More info on this can be found in the
+ paragraph below.
-* Copy the ONOS applications to your local web server. The procedure largely varies from the web server you run, its configuration, and what ONOS applications you need.
+* Copy the ONOS applications to your local web server. The procedure
+ largely varies from the web server you run, its configuration, and
+ what ONOS applications you need.
-* Deploy CORD/SEBA using the helm charts. Remember to load with the *-f* option the additional configuration file to extend the helm charts, if any.
+* Deploy CORD using the helm charts. Remember to load with the
+ *-f* option the additional configuration file to extend the helm
+ charts, if any.
{% include "/partials/push-images-to-registry.md" %}
-## Optional packages
+## Optional Packages
-### Install a Docker Registry using helm
+### Install a Docker Registry Using Helm
-If you don't have a local Docker registry deployed in your infrastructure, you can install an **insecure** one using the official Kubernetes helm-chart.
+If you don't have a local Docker registry deployed in your
+infrastructure, you can install an **insecure** one using the official
+Kubernetes helm-chart.
+
+Since this specific docker registry is packaged as a kubernetes pod, shipped with helm, you'll need Internet connectivity to install it.
> **Note:** *Insecure* registries can be used for development, POCs or lab trials. **You should not use this in production.** There are planty of documents online that guide you through secure registries setup.
-The following command deploys the registry and exposes the port *30500*. (You may want to change it with any value that fit your deployment needs).
+The following command deploys the registry and exposes the port
+*30500*. (You may want to change it with any value that fit your
+deployment needs).
```shell
helm install stable/docker-registry --set service.nodePort=30500,service.type=NodePort -n docker-registry
@@ -96,9 +136,11 @@
curl -X GET http://KUBERNETES_IP:30500/v2/_catalog
```
-### Install a local web server using helm (optional)
+### Install a Local Webserver Using Helm (optional)
-If you don't have a local web server that can be accessed from the POD, you can easily install one on top of your existing Kubernetes cluster.
+If you don't have a local web server that can be accessed from the
+POD, you can easily install one on top of your existing Kubernetes
+cluster.
```shell
# From the helm-charts directory, while preparing the offline install
@@ -106,24 +148,30 @@
helm fetch bitnami/nginx --untar
# Then, while deploying offline
-helm install -n mavenrepo --set service.nodePorts.http=30160,service.type=NodePort nginx
+helm install -n mavenrepo --set service.type=NodePort --set service.nodePorts.http=30160 bitnami/nginx
```
-The webserver will be up in few seconds and you'll be able to reach the root web page using the IP of one of your Kubernetes nodes, port *30160*. For example, you can do:
+The webserver will be up in few seconds and you'll be able to reach
+the root web page using the IP of one of your Kubernetes nodes, port
+*30160*. For example, you can do:
```shell
wget KUBERNETES_IP:30160
```
-OAR images can be copied to the document root of the web server using the *kubectl cp* command. For example:
+OAR images can be copied to the document root of the web server using
+the *kubectl cp* command. For example:
```shell
kubectl cp my-onos-app.oar `kubectl get pods | grep mavenrepo | awk '{print $1;}'`:/opt/bitnami/nginx/html
```
-## Example: offline SEBA install
+## Example: Offline SEBA Install
-The following section provides an exemplary list of commands to perform an offline SEBA POD installation. Please, note that some command details (i.e. chart names, image names, tools) may have changed over time.
+The following section provides an exemplary list of commands to
+perform an offline SEBA POD installation. Please, note that some
+command details (i.e. chart names, image names, tools) may have
+changed over time.
### Assumptions
@@ -143,49 +191,37 @@
* The IP address of the machine hosting Kubernetes is 192.168.0.100.
-### Prepare the installation
+### Prepare the Installation
```shell
-# Clone repos
+# Clone the automation-tools repo
git clone https://gerrit.opencord.org/automation-tools
-git clone https://gerrit.opencord.org/helm-charts
-# Copy automation scripts in the right place
-cp automation-tools/developer/images_from_charts.sh helm-charts
-cp automation-tools/developer/pull_images.sh helm-charts
-cp automation-tools/developer/tag_and_push.sh helm-charts
-
-cd helm-charts
-
-# Add online helm repositories and update dependencies
-helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
-helm repo add rook-beta https://charts.rook.io/beta
+# Add the online helm repositories and update indexes
+helm repo add cord https://charts.opencord.org
+helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
helm repo add bitnami https://charts.bitnami.com/bitnami
-helm dep update voltha
-helm dep update xos-core
-helm dep update xos-profiles/att-workflow
-helm dep update xos-profiles/base-kubernetes
-helm dep update nem-monitoring
-helm dep update logging
-helm dep update storage/rook-operator
+helm repo update
-# Fetch helm-charts not available locally
+# Fetch helm charts
+helm fetch cord/cord-platform --version 6.1.0 --untar
+helm fetch cord/seba --version 1.0.0 --untar
+helm fetch cord/att-workflow --version 1.0.0 --untar
helm fetch stable/docker-registry --untar
-helm fetch --version 0.8.0 stable/etcd-operator --untar
-helm fetch --version 0.8.8 incubator/kafka --untar
helm fetch bitnami/nginx --untar
-# Update Kafka dependencies
-helm dep update kafka
+# Update chart dependencies
+helm dep update cord-platform
+helm dep update seba
# For demo, install the local, helm-based Docker Registry on the remote POD (this will require POD connectivity to download the docker registry image)
helm install stable/docker-registry --set service.nodePort=30500,service.type=NodePort -n docker-registry
# For demo, install the local web-server to host ONOS images
-helm install -n mavenrepo bitnami/nginx
+helm install -n mavenrepo --set service.type=NodePort --set service.nodePorts.http=30160 bitnami/nginx
# Identify images form the official helm charts and pull images from DockerHub. If you see some "skipped value for filters" warning that's fine
-bash images_from_charts.sh kafka etcd-cluster etcd-operator voltha onos xos-core xos-profiles/att-workflow xos-profiles/base-kubernetes nem-monitoring logging storage/rook-operator | bash pull_images.sh > images
+bash automation-tools/developer/images_from_charts.sh cord-platform seba seba/charts/voltha/charts/etcd-cluster att-workflow | automation-tools/developer/bash pull_images.sh > images
# Download ONOS apps
curl -L "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.opencord&a=olt-app&v=2.1.0-SNAPSHOT&e=oar" > olt.oar
@@ -194,93 +230,101 @@
curl -L "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.opencord&a=aaa&v=1.8.0-SNAPSHOT&e=oar" > aaa.oar
curl -L "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.opencord&a=kafka&v=1.0.0-SNAPSHOT&e=oar" > kafka.oar
-# Create file to extend the helm charts (call it extend.yaml)
+# Create file to override the default helm charts values (call it extend.yaml)
global:
registry: 192.168.0.100:30500/
-image: 192.168.0.100:30500/confluentinc/cp-kafka
-imageTag: 4.1.2-2
+# CORD platform overrides
+kafka:
+ image: 192.168.0.100:30500/confluentinc/cp-kafka
+ imageTag: 4.1.2-2
+ configurationOverrides:
+ zookeeper.connection.timeout.ms: 60000
+ zookeeper.session.timeout.ms: 60000
-zookeeper:
- image:
- repository: 192.168.0.100:30500/gcr.io/google_samples/k8szk
+ zookeeper:
+ image:
+ repository: 192.168.0.100:30500/gcr.io/google_samples/k8szk
-etcd-cluster:
- spec:
- repository: 192.168.0.100:30500/quay.io/coreos/etcd
- pod:
- busyboxImage: 192.168.0.100:30500/busybox:1.28.1-glibc
-
-etcdOperator:
- image:
- repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
-backupOperator:
- image:
- repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
-restoreOperator:
- image:
- repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
-
-grafana:
- image:
- repository: 192.168.0.100:30500/grafana/grafana
- sidecar:
- image: 192.168.0.100:30500/kiwigrid/k8s-sidecar:0.0.3
-
-prometheus:
- server:
+logging:
+ elasticsearch:
image:
- repository: 192.168.0.100:30500/prom/prometheus
- alertmanager:
- image:
- repository: 192.168.0.100:30500/prom/alertmanager
- configmapReload:
- image:
- repository: 192.168.0.100:30500/jimmidyson/configmap-reload
- kubeStateMetrics:
- image:
- repository: 192.168.0.100:30500/quay.io/coreos/kube-state-metrics
- nodeExporter:
- image:
- repository: 192.168.0.100:30500/prom/node-exporter
- pushgateway:
- image:
- repository: 192.168.0.100:30500/prom/pushgateway
- initChownData:
- image:
+ repository: 192.168.0.100:30500/docker.elastic.co/elasticsearch/elasticsearch-oss
+ initImage:
repository: 192.168.0.100:30500/busybox
-elasticsearch:
- image:
- repository: 192.168.0.100:30500/docker.elastic.co/elasticsearch/elasticsearch-oss
- initImage:
- repository: 192.168.0.100:30500/busybox
+ kibana:
+ image:
+ repository: 192.168.0.100:30500/docker.elastic.co/kibana/kibana-oss
-kibana:
- image:
- repository: 192.168.0.100:30500/docker.elastic.co/kibana/kibana-oss
+ logstash:
+ image:
+ repository: 192.168.0.100:30500/docker.elastic.co/logstash/logstash-oss
-logstash:
- image:
- repository: 192.168.0.100:30500/docker.elastic.co/logstash/logstash-oss
+nem-monitoring:
+ grafana:
+ image:
+ repository: 192.168.0.100:30500/grafana/grafana
+ sidecar:
+ image: 192.168.0.100:30500/kiwigrid/k8s-sidecar:0.0.3
-oltAppUrl: http://192.168.0.100:30160/olt.oar
-sadisAppUrl: http://192.168.0.100:30160/sadis.oar
-dhcpL2RelayAppUrl: http://192.168.0.100:30160/dhcpl2relay.oar
-aaaAppUrl: http://192.168.0.100:30160/aaa.oar
-kafkaAppUrl: http://192.168.0.100:30160/kafka.oar
+ prometheus:
+ server:
+ image:
+ repository: 192.168.0.100:30500/prom/prometheus
+ alertmanager:
+ image:
+ repository: 192.168.0.100:30500/prom/alertmanager
+ configmapReload:
+ image:
+ repository: 192.168.0.100:30500/jimmidyson/configmap-reload
+ kubeStateMetrics:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/kube-state-metrics
+ nodeExporter:
+ image:
+ repository: 192.168.0.100:30500/prom/node-exporter
+ pushgateway:
+ image:
+ repository: 192.168.0.100:30500/prom/pushgateway
+ initChownData:
+ image:
+ repository: 192.168.0.100:30500/busybox
+
+# SEBA specific overrides
+voltha:
+ etcd-cluster:
+ spec:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd
+ pod:
+ busyboxImage: 192.168.0.100:30500/busybox:1.28.1-glibc
+
+ etcdOperator:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
+ backupOperator:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
+ restoreOperator:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
+
+seba-services:
+ oltAppUrl: http://192.168.0.100:30160/olt.oar
+ sadisAppUrl: http://192.168.0.100:30160/sadis.oar
+ dhcpL2RelayAppUrl: http://192.168.0.100:30160/dhcpl2relay.oar
+ aaaAppUrl: http://192.168.0.100:30160/aaa.oar
+ kafkaAppUrl: http://192.168.0.100:30160/kafka.oar
# Download the openolt.deb driver installation file from the vendor website (command varies)
scp/wget... openolt.deb
```
-### Offline deployment
+### Offline Deployment
```shell
-cd helm-charts
-
# Tag and push the images to the local Docker registry
-cat images | bash tag_and_push.sh -r 192.168.0.100:30500
+cat images | bash automation-tools/developer/tag_and_push.sh -r 192.168.0.100:30500
# Copy the ONOS applications to the local web server
MAVEN_REPO=$(kubectl get pods | grep mavenrepo | awk '{print $1;}')
@@ -290,16 +334,10 @@
kubectl cp aaa.oar $MAVEN_REPO:/opt/bitnami/nginx/html
kubectl cp kafka.oar $MAVEN_REPO:/opt/bitnami/nginx/html
-# Install SEBA
-helm install -n etcd-operator -f extend.yaml --version 0.8.0 etcd-operator
-helm install -n cord-kafka -f examples/kafka-single.yaml -f extend.yaml --version 0.8.8 kafka
-helm install -n voltha -f extend.yaml voltha
-helm install -n onos -f configs/onos.yaml -f extend.yaml onos
-helm install -n xos-core -f extend.yaml xos-core
-helm install -n att-workflow -f extend.yaml xos-profiles/att-workflow
-helm install -n base-kubernetes -f extend.yaml xos-profiles/base-kubernetes
-helm install -n nem-monitoring -f extend.yaml nem-monitoring
-helm install -n logging -f extend.yaml logging
+# Install the CORD platform, the SEBA profile and the ATT workflow
+helm install -n cord-platform -f extend.yaml cord-platform
+helm install -n seba -f extend.yaml seba
+helm install -n att-workflow -f extend.yaml att-workflow
# On the OLT, copy, install and run openolt.deb
scp openolt.deb root@192.168.0.200:
diff --git a/operating_cord/veth_intf.md b/operating_cord/veth_intf.md
index 4ccf526..4bbe2cb 100644
--- a/operating_cord/veth_intf.md
+++ b/operating_cord/veth_intf.md
@@ -1,18 +1,27 @@
-# Manually connect containers to a network card
+# Attach Container to a NIC
-Sometimes you may need to attach some containers NICs to the network cards of the machines hosting them, for example to run some data plane traffic through them.
+Sometimes you may need to attach some container's virtual network
+interface to the NIC of the machine hosting it, for example, to run
+some data plane traffic through the container.
-Although CORD doesn't fully support this natively there are some (hackish) ways to do this manually.
+Although CORD doesn't fully support this natively, there are some
+(hackish) ways to do this manually.
## Create a bridge and a veth
-The easiest way to do this is to skip Kubernetes and directly attach the Docker container link it to the host network interface, through a Virtual Ethernet Interface Pair (veth pair).
+The easiest way to do this is to skip Kubernetes and directly attach
+the Docker container link it to the host network interface through a
+Virtual Ethernet Interface Pair (veth pair).
Let's see how.
-For completeness, let's assume you're running a three nodes Kubernetes deployment, and that you're trying to attach a container *already deployed* called *vcore-5b4c5478f-lxrpb* to a physical interface *eth1* (already existing on one of the three hosts, running your container). The virtual interface inside the container will be called *eth2*.
+Let's assume you're running a three node Kubernetes deployment, and
+that you're trying to attach a container *already deployed* called
+*vcore-5b4c5478f-lxrpb* to a physical interface *eth1* (already
+existing on one of the three hosts, running your container). The
+virtual interface inside the container will be called *eth2*.
-You got the name of the container running
+You get the name of the container by running:
```shell
$ kubectl get pods [-n NAMESPACE]
@@ -20,16 +29,20 @@
vcore-5b4c5478f-lxrpb 1/1 Running 1 7d
```
-Find out on which of the three nodes the container has been deployed
+To find out on which of the three nodes the container has been
+deployed on, type:
```shell
$ kubectl describe pod vcore-5b4c5478f-lxrpb | grep Node
Node: node3/10.90.0.103
Node-Selectors: <none>
```
-As you can see from the first line, the container has been deployed by Kubernetes on the Docker daemon running on node 3 (this is just an example). In this case, with IP *10.90.0.103*.
-Let's SSH into the node and let's look for the specific Docker container ID
+As you can see from the first line, the container has been deployed by
+Kubernetes on the Docker daemon running on node 3 (this is just an
+example), in this case, with IP *10.90.0.103*.
+
+Next, SSH into the node and look for the specific Docker container ID:
```shell
$ container_id=$(sudo docker ps | grep vcore-5b4c5478f-lxrpb | head -n 1 | awk '{print $1}')
@@ -42,31 +55,32 @@
sudo ip link set eth1 down
```
-Create a veth called *veth0* and let's add to it the new virtual interface *eth2*
+Create a veth called *veth0* and add to it the new virtual interface *eth2*:
```shell
sudo ip link add veth0 type veth peer name eth2
```
-Add the virtual network interface *eth2* to the container namespace
+Now add the virtual network interface *eth2* to the container namespace:
```shell
sudo ip link set eth2 netns ${container_id}
```
-Bring up the virtual interface
+Bring up the virtual interface:
```shell
sudo ip netns exec ${container_id} ip link set eth2 up
```
-Bring up *veth0*
+Bring up *veth0*:
```shell
sudo ip link set veth0 up
```
-Create a bridge named *br1*. Add *veth0* to it and the host interface *eth1*
+Create a bridge named *br1*, and add *veth0* to it and the host
+interface *eth1*:
```shell
sudo ip link add br1 type bridge
@@ -75,14 +89,15 @@
```
-Bring up again the host interface and the bridge
+Bring up again the host interface and the bridge:
```shell
sudo ip link set eth1 up
sudo ip link set br1 up
```
-At this point, you should see an additional interface *eth2* inside the container
+At this point, you should see an additional interface *eth2* inside
+the container:
```shell
$ kubectl exec -it vcore-5b4c5478f-lxrpb /bin/bash
@@ -96,9 +111,10 @@
link/ether d6:84:33:2f:8c:92 brd ff:ff:ff:ff:ff:ff
```
-## Cleanup (remove veth and bridge)
+## Cleanup (remove bridge and veth)
-As a follow up of the previous example, let's now try to delete what has been created so far, to bring the system back to the original state.
+To delete the connection and bring the system back to the original
+state, execute the following:
```shell
ip link set veth0 down
diff --git a/platform.md b/platform.md
index f66fc1b..d14697c 100644
--- a/platform.md
+++ b/platform.md
@@ -3,7 +3,7 @@
Once the prerequisites have been met, the next step to installing CORD is
to bring up the Helm charts for the platform components.
-## CORD Platform as a whole
+## CORD Platform as a Whole
To install the CORD Platform you can use the corresponding chart:
@@ -11,14 +11,14 @@
helm install -n cord-platform cord/cord-platform --version=6.1.0
```
-## CORD Platform as separate components
+## CORD Platform as Separate Components
-The main reason to install the CORD Platform by installing its standalone components
-is if you're developing on it and you need granular control.
-
-There are the components included in the `cord-platform` chart:
+Sometimes it his helpful (for example, when developing) to install the
+individual components that make up the CORD Platform one at a time.
+The following are the individual components included in the
+`cord-platform` chart:
- [ONOS](./charts/onos.md#onos-manages-fabric--voltha)
-- [xos-core](./charts/xos-core.md)
-- [cord-kafka](./charts/kafka.md)
-- [logging-monitoring](./charts/logging-monitoring.md)
+- [XOS](./charts/xos-core.md)
+- [Kafka](./charts/kafka.md)
+- [Logging-Monitoring](./charts/logging-monitoring.md)
diff --git a/prereqs/hardware.md b/prereqs/hardware.md
index a790698..deafaaf 100644
--- a/prereqs/hardware.md
+++ b/prereqs/hardware.md
@@ -29,7 +29,7 @@
emulate the fabric in software (e.g., using Mininet), but this applies
only to specific use-cases.
-* **Access Devices**: At the moment, both R-CORD and M-CORD work
+* **Access Devices**: At the moment, SEBA and M-CORD work
with very specific access devices, as described below. We strongly
recommend using these tested devices.
@@ -72,7 +72,7 @@
* Robofiber QSFP-40G-03C QSFP+ 40G direct attach passive
copper cable, 3m length - S/N: QSFP-40G-03C
-* **R-CORD Access Devices and Optics**
+* **SEBA Access Devices and Optics**
* **GPON**
* **OLT**: Celestica CLS Ruby S1010 (experimental, only top-down provisioning is supported - through manual customizations)
* Compatible **OLT optics**
diff --git a/prereqs/helm.md b/prereqs/helm.md
index b3a6cd0..c75080f 100644
--- a/prereqs/helm.md
+++ b/prereqs/helm.md
@@ -42,5 +42,6 @@
## Next Step
Once you are done, you are ready to deploy CORD components using their
-helm charts! See [Bringing Up CORD](../profiles/intro.md). For more detailed
-information, see the [helm chart reference guide](../charts/helm.md).
+helm charts! Start by brining up the
+[CORD Platform](../platform.md). For more detailed information, see
+the [Helm Reference](../charts/helm.md).
diff --git a/prereqs/openstack-helm.md b/prereqs/openstack-helm.md
index aefa6f1..b3941c7 100644
--- a/prereqs/openstack-helm.md
+++ b/prereqs/openstack-helm.md
@@ -4,7 +4,8 @@
project can be used to install a set of Kubernetes nodes as OpenStack
compute nodes, with the OpenStack control services (nova, neutron,
keystone, glance, etc.) running as containers on Kubernetes. This is
-necessary, for example, to run the M-CORD profile.
+an easy way to bring up an OpenStack cluster that can be controlled
+by loading the [Base OpenStack](../charts/base-openstack.md) chart.
Instructions for installing `openstack-helm` on a single node or a
multi-node cluster can be found at
diff --git a/prereqs/optional.md b/prereqs/optional.md
deleted file mode 100644
index e19dbe3..0000000
--- a/prereqs/optional.md
+++ /dev/null
@@ -1,7 +0,0 @@
-# Optional Packages
-
-Although not required, you may want to install the following packages:
-
-* **OpenStack:** If you need to include OpenStack in your deployment,
- so you can bring up VMs on your POD, you will need to following the
- [OpenStack deployment](openstack-helm.md) guide.
diff --git a/prereqs/software.md b/prereqs/software.md
index 03b0a07..971a296 100644
--- a/prereqs/software.md
+++ b/prereqs/software.md
@@ -4,11 +4,3 @@
pretty much any Kubernetes environment. It is your choice how
to install Kubernetes, although this section describes automation
scripts we have found useful.
-
-> **Note:** M-CORD is the exception since its components still depend on
-> OpenStack, which is in turn deployed as a set of Kubernetes containers
-> using the [openstack-helm](https://github.com/openstack/openstack-helm)
-> project. Successfully installing the OpenStack Helm charts requires
-> some additional system configuration besides just installing Kubernetes
-> and Helm. You can find more informations about this in the
-> [OpenStack Support](./openstack-helm.md) installation section.
diff --git a/profiles/seba/configuration.md b/profiles/seba/configuration.md
index a98975c..780ea1a 100644
--- a/profiles/seba/configuration.md
+++ b/profiles/seba/configuration.md
@@ -128,7 +128,7 @@
device_type: openolt
host: 10.90.0.122
port: 9191
- switch_datapath_id: of:0000000000000002 # the openflow switch to with the OLT is connected
+ switch_datapath_id: of:0000000000000002 # the openflow switch to which the OLT is connected
switch_port: "1" # the port on the switch on which the OLT is connected
outer_tpid: "0x8100"
uplink: "65536"
diff --git a/profiles/seba/install.md b/profiles/seba/install.md
index c3c8dc8..f32b61f 100644
--- a/profiles/seba/install.md
+++ b/profiles/seba/install.md
@@ -8,9 +8,16 @@
In order to run SEBA you need to have the [CORD Platform](../../platform.md) installed.
-### SEBA as a whole
+Specifically, wait for the EtcdCluster CustomResourceDefinitions to
+appear in Kubernetes:
-To install the SEBA Profile you can use the corresponding chart:
+```shell
+kubectl get crd | grep etcd
+```
+
+Once the CRDs are present, proceed with the `seba` chart installation.
+
+### SEBA as a whole
```shell
helm install -n seba cord/seba --version=1.0.0
@@ -21,7 +28,7 @@
The main reason to install the SEBA Profile by installing its standalone
components is if you're developing on it and you need granular control.
-There are the components included in the `seba` chart:
+These are the components included in the `seba` chart:
- [VOLTHA and etcd-operator](../../charts/voltha.md)
- [seba-services](../../charts/seba-services.md)
diff --git a/profiles/seba/siab-overview.md b/profiles/seba/siab-overview.md
index a45c2fe..030a5b9 100644
--- a/profiles/seba/siab-overview.md
+++ b/profiles/seba/siab-overview.md
@@ -1,4 +1,4 @@
-# SEBA-in-a-Box
+# SEBA-in-a-Box Overview
If you do not have the [prerequisite hardware](../../prereqs/hardware.md) to
install SEBA we have you covered. You can get a complete CORD system together
diff --git a/profiles/seba/troubleshoot/no-aaa.md b/profiles/seba/troubleshoot/no-aaa.md
index cbb591f..61ab49a 100644
--- a/profiles/seba/troubleshoot/no-aaa.md
+++ b/profiles/seba/troubleshoot/no-aaa.md
@@ -1 +1,175 @@
# I can't authenticate my RGs
+
+If the RG cannot authenticate, there can be a number of issues with the system.
+On the one hand, it could truly be an authentication issue. For that it is best
+to check your RADIUS server's logs. On the other hand, it could be an issue with
+how authentication packets are flowing through the system.
+
+To quickly recap, EAPOL packets from the RG reach the RADIUS server using this path:
+
+RG --> ONU --> OLT --> trapped to VOLTHA --> packet-in to ONOS AAA app --> RADIUS server
+
+## Check VOLTHA and the PON
+
+The first thing to check - is VOLTHA receiving trapped EAPOL packets at all?
+Unfortunately, the only way to do this currently is to check VOLTHA logs, by streaming
+it using kubectl and grepping for EAPOL ethertype 0x888e.
+
+From a terminal where you have kubectl client, try
+
+```shell
+cord@node1:~$ kubectl logs -f -n voltha vcore-0 | grep -E "packet_indication|packet-in" | grep 888e
+
+20180912T003237.453 DEBUG MainThread adapter_agent.send_packet_in {adapter_name: openolt, logical_port_no: 16, logical_device_id: 000100000a5a0097, packet: 0180c200000390e2ba82fa8281000ffb888e01000009020100090175736572000000000000000000000000000000000000000000000000000000000000000000, event: send-packet-in, instance_id: compose_voltha_1_1536712228, vcore_id: 0001}
+```
+
+Notice that the packet is displayed in hex, and includes the ethertype right after
+the VLAN tag (with ethertype 0x8100 and VLAN id 0x0ffb)
+
+If you don't see this packet, there is something wrong with the EAPOL trap flow in the OLT
+or the PON setup itself (i.e. OLT + ONU). Check the state of the devices on the
+VOLTHA cli.
+
+Connect to the [VOLTHA CLI](../../../charts/voltha.md#accessing-the-voltha-cli)
+and check the devices:
+
+```shell
+(voltha) devices
+Devices:
++------------------+-------------------+------+------------------+------------------+-------------+-------------+----------------+----------------+------------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
+| id | type | root | parent_id | serial_number | admin_state | oper_status | connect_status | parent_port_no | host_and_port | reason | proxy_address.device_id | proxy_address.channel_id | proxy_address.onu_id | proxy_address.onu_session_id |
++------------------+-------------------+------+------------------+------------------+-------------+-------------+----------------+----------------+------------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
+| 00012d28315ddb79 | openolt | True | 000100000a5a007a | 10.90.0.122:9191 | ENABLED | ACTIVE | REACHABLE | | 10.90.0.122:9191 | | | | | |
+| 0001d18bedd13517 | brcm_openomci_onu | True | 00012d28315ddb79 | ALPHe3d1cfe3 | ENABLED | ACTIVE | REACHABLE | 536870912 | | initial-mib-downloaded | 00012d28315ddb79 | | 1 | 1 |
+| 00011c399faa957d | brcm_openomci_onu | True | 00012d28315ddb79 | ALPHe3d1cf9d | ENABLED | DISCOVERED | REACHABLE | 536870912 | | starting-omci | 00012d28315ddb79 | | 2 | 2 |
++------------------+-------------------+------+------------------+------------------+-------------+-------------+----------------+----------------+------------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
+```
+Both your OLT and ONU should be enabled, active and reachable.
+Next check the flows in the logical device that represents the pon.
+
+```shell
+(voltha) logical_device 000100000a5a007a
+(logical device 000100000a5a007a) flows
+Logical Device 000100000a5a007a (type: n/a)
+Flows (12):
++----------+----------+-----------+---------+----------+----------+----------+---------+---------+--------------+--------------+----------+-----------+------------+------------+
+| table_id | priority | cookie | in_port | vlan_vid | eth_type | ip_proto | udp_src | udp_dst | metadata | set_vlan_vid | pop_vlan | push_vlan | output | goto-table |
++----------+----------+-----------+---------+----------+----------+----------+---------+---------+--------------+--------------+----------+-----------+------------+------------+
+| 0 | 10000 | ~1b322a1e | 65536 | | 88CC | | | | | | | | CONTROLLER | |
+| 0 | 10000 | ~1ec4282a | 65536 | | 800 | 17 | 67 | 68 | | | | | CONTROLLER | |
+| 0 | 10000 | ~e863949e | 30736 | | 888E | | | | | | | | CONTROLLER | |
+-snip-
+```
+Notice that the EAPOL flow 0x888e corresponds to the UNI port 30736 and has a
+directive to send to CONTROLLER. Check 'ports' on the logical device to ensure
+that the UNI port corresponds to the correct ONU (via Serial Number) that connects
+to your RG.
+
+```shell
+(voltha) logical_device 000100000a5a007a
+(logical device 000100000a5a007a) ports
+Logical device ports:
++-----------+------------------+----------------+-----------+------------------+------------------------------+---------------+----------------+---------------+---------------------+------------------------+
+| id | device_id | device_port_no | root_port | ofp_port.port_no | ofp_port.hw_addr | ofp_port.name | ofp_port.state | ofp_port.curr | ofp_port.curr_speed | ofp_port_stats.port_no |
++-----------+------------------+----------------+-----------+------------------+------------------------------+---------------+----------------+---------------+---------------------+------------------------+
+| nni-65536 | 00012d28315ddb79 | 65536 | True | 65536 | [0L, 0L, 0L, 1L, 0L, 0L] | nni-65536 | 4 | 4128 | 32 | 65536 |
+| uni-30736 | 0001867d6f014a8d | 30736 | | 30736 | [8L, 0L, 0L, 15L, 120L, 16L] | ISKT71e80080 | 4 | 4160 | 64 | |
+| uni-64 | 0001a331b09f048b | 64 | | 64 | [8L, 0L, 0L, 0L, 0L, 64L] | ALPHe3d1ced5 | 4 | 4160 | 64 | |
+| uni-32 | 00011c399faa957d | 32 | | 32 | [8L, 0L, 0L, 0L, 0L, 32L] | ALPHe3d1cf9d | 4 | 4160 | 64 | |
+-snip-
+```
+You should also check the flows in the corresponding physical device object that
+represents the OLT
+
+```shell
+(voltha) device 00012d28315ddb79
+(device 00012d28315ddb79) flows
+Device 00012d28315ddb79 (type: openolt)
+Flows (15):
++----------+----------+-----------+------------+----------+----------+----------+---------+---------+--------------+--------------+----------+-----------+------------+------------+
+| table_id | priority | cookie | in_port | vlan_vid | eth_type | ip_proto | udp_src | udp_dst | metadata | set_vlan_vid | pop_vlan | push_vlan | output | goto-table |
++----------+----------+-----------+------------+----------+----------+----------+---------+---------+--------------+--------------+----------+-----------+------------+------------+
+| 0 | 10000 | ~7e524bb2 | 30736 | 4091 | 888E | | | | | | | | CONTROLLER | |
+| 0 | 10000 | ~7e524bb2 | CONTROLLER | 3999 | | | | | | | | | 30736 | |
+-snip-
+```
+
+Notice that the 1 EAPOL flow in the logical device, becomes 2 flows in the physical device,
+one from UNI port to CONTROLLER and the other from CONTROLLER to UNI port. The former
+shows the ethertype 0x888e, and the latter does not, but both are necessary. Furthermore,
+the VLANs in the display are default VLANs used by the openolt adaptor - don't worry
+about them.
+
+If all this looks good, there is a possibility that your RG is connected to the
+ONU on the wrong LAN port. On the ONU there are typically 4 LAN ports marked LAN 1
+through LAN 4. Your RG should be connected to LAN 1. Sometimes the internal
+naming could be reversed, so you could try switching the RG to LAN 4.
+
+For more low level ONU debugging consider:
+<https://guide.opencord.org/openolt/#why-does-the-broadcom-onu-not-forward-eapol-packets>
+
+
+## Check ONOS AAA app state
+
+If EAPOL packets are indeed making their way through VOLTHA, then check what's
+happening in ONOS. At this time we do not have counters for AAA state machine transactions.
+
+Connect to the [ONOS CLI](../../../charts/onos.md#accessing-the-onos-cli)
+and check the logs
+
+```shell
+onos> log:tail
+-snip-
+2018-12-19 23:01:36,788 | INFO | 33.102.179:59154 | StateMachine$Authorized | 185 - org.opencord.aaa - 1.8.0 | Moving from AUTHORIZED state to STARTED state.
+2018-12-19 23:01:36,789 | INFO | 33.102.179:59154 | AaaManager | 185 - org.opencord.aaa - 1.8.0 | Auth event STARTED for of:000000000a5a007a/30736
+2018-12-19 23:01:36,789 | INFO | 33.102.179:59154 | StateMachine | 185 - org.opencord.aaa - 1.8.0 | Current State 1
+2018-12-19 23:01:36,906 | INFO | 33.102.179:59154 | StateMachine$Started | 185 - org.opencord.aaa - 1.8.0 | Moving from STARTED state to PENDING state.
+2018-12-19 23:01:36,906 | INFO | 33.102.179:59154 | AaaManager | 185 - org.opencord.aaa - 1.8.0 | Auth event REQUESTED for of:000000000a5a007a/30736
+2018-12-19 23:01:36,906 | INFO | 33.102.179:59154 | StateMachine | 185 - org.opencord.aaa - 1.8.0 | Current State 2
+2018-12-19 23:01:37,017 | INFO | AAA-radius-0 | AaaManager | 185 - org.opencord.aaa - 1.8.0 | Send EAP success message to supplicant 90:E2:BA:82:F9:75
+2018-12-19 23:01:37,017 | INFO | AAA-radius-0 | StateMachine$Pending | 185 - org.opencord.aaa - 1.8.0 | Moving from PENDING state to AUTHORIZED state.
+2018-12-19 23:01:37,018 | INFO | AAA-radius-0 | StateMachine | 185 - org.opencord.aaa - 1.8.0 | Current State 3
+2018-12-19 23:01:37,018 | INFO | AAA-radius-0 | AaaManager | 185 - org.opencord.aaa - 1.8.0 | Auth event APPROVED for of:000000000a5a007a/30736
+-snip-
+```
+
+In a working authentication transaction, you should see the logs above for the RG
+on the UNI port `of:000000000a5a007a/30736` where the `of:000000000a5a007a` is the
+logical device representation of the PON provided by VOLTHA to ONOS, and `30736`
+is the UNI port. You can also check the state in the 'aaa-users' command
+
+```shell
+nos> aaa-users
+UserName=user,CurrentState=AUTHORIZED,DeviceId=of:000000000a5a007a,MAC=90:E2:BA:82:F9:75,PortNumber=30736,SubscriberId=PON 1/1/04/1:1.1.1
+```
+
+In the non-working case, you should see WARNINGs or ERRORs. There is one possibility
+where you may not see any errors - this typically happens in State 1 of the AAA
+state machine. Essentially, the RG (supplicant) sends an EAPOL_START message to the
+Authenticator (ONOS), and the AAA app sends an EAPOL_IDENTITY_REQUEST back to the
+RG. If this packet never reaches the RG, then the state machine will not proceed.
+
+Go back to VOLTHA and ensure that the packet is making its way back to the OLT.
+
+```shell
+cord@node1:~$ kubectl logs -f -n voltha vcore-0 | grep -E "sending-packet-to-ONU" | grep 888e
+
+20180912T003238.392 DEBUG MainThread openolt_device.packet_out {ip: 10.90.0.151:9191, id: 00018e736871ac1c, packet: 90e2ba82fa8200000000100181000ffb888e010000040302000400000000000000000000000000000000000000000000000000000000000000000000, egress_port: 16, onu_id: 1, intf_id: 0, event: sending-packet-to-ONU, instance_id: compose_voltha_1_1536712228, vcore_id: 0001}
+```
+
+You should see an EAPOL packet (0x888e) being sent to the ONU in the logs. If you do see this,
+consider doing a tcpdump in your RG to ensure that this packet is making it all the
+way to the RG.
+
+## Connectivity to RADIUS server
+
+If instead you see in the ONOS logs that the state machine is stuck in State 2, it means
+that the EAPOL_IDENTITY_REQUEST successfully reached the RG, and the RG responded with
+an EAPOL_IDENTITY_REPLY which reached ONOS, causing the state machine to move on from
+STARTED to PENDING state. At this point, the identity reply is packaged into a
+RADIUS packet and sent to the RADIUS server by the AAA app.
+
+The last hop from the AAA app in ONOS to the RADIUS server happens over the
+management network. In a deployment it is entirely possible that the RADIUS server is
+outside the SEBA pod, which means that you need to ensure that the POD has connectivity
+to the server and that you have correctly configured details of the server in the pod.
diff --git a/profiles/seba/troubleshoot/no-att-si.md b/profiles/seba/troubleshoot/no-att-si.md
index 82d170f..c810f69 100644
--- a/profiles/seba/troubleshoot/no-att-si.md
+++ b/profiles/seba/troubleshoot/no-att-si.md
@@ -106,7 +106,7 @@
root@kafkacat-7f45f65485-2lgp8:/# kafkacat -b cord-platform-kafka -t onu.events
```
-If evetyhing is correct you should see:
+If everything is correct you should see:
```shell
{"timestamp":"2018-12-19T22:34:00.755Z","status":"activated","serial_number":"ALPHe3d1cfe3","uni_port_id":16,"of_dpid":"of:000000000a5a007a"}
diff --git a/profiles/seba/troubleshoot/no-dhcp.md b/profiles/seba/troubleshoot/no-dhcp.md
index 63c1675..e2b9eea 100644
--- a/profiles/seba/troubleshoot/no-dhcp.md
+++ b/profiles/seba/troubleshoot/no-dhcp.md
@@ -1 +1,178 @@
# RG is not getting IP from DHCP
+
+To debug a failing DHCP process, it is necessary to understand the packet flow
+through the system, the relevant flows in the hardware, and the expected logs.
+
+## DHCP Relay Process
+
+It is important to understand that DHCP will not succeed in the AT&T workflow, if
+the RG has not been authenticated. Once an RG successfully authenticates, flows
+that trap DHCP packets to VOLTHA and ONOS, as well as the subscribers's VLAN tags
+are programmed in the OLT as well as the AGG switch.
+
+Next it is important to understand that there are two ways the ONOS DHCP L2 Relay app
+can operate. DHCP packets received from the RG can be relayed to the server using
+either the OLT's uplink port (that connects to the AGG switch), or they can be
+relayed directly to the AGG switch's uplink port (that connects to the BNG/DHCP-server).
+By default, a SEBA pod uses the latter mechanism.
+
+
+
+
+
+The configuration for the DHCP relay app can be seen in the XOS GUI
+
+
+
+As shown, the SEBA pod's default configuration for dhcpl2relay sets the
+`useOltUplinkForServerPktInOut` flag to false, and provides the location of the
+server attachment on port 32 of the AGG switch (`of:0000000000000002/32`).
+Note: if you wish to try the OLT uplink for DHCP, set the flag to true, and remove
+the `dhcpServerConnectPoint` field.
+
+Finally, it is also useful to understand that DHCP packets are sent to the DHCP
+server QinQ tagged with the subscribers S and C VLAN tags (both 0x8100 ethertype).
+So if the server is not configured to accept double tagged packets, you won't get
+a DHCP reply.
+
+## Relevant Flows
+
+The DHCP packet-in/out from the OLT and the AGG switch depends on a few key flows.
+
+### OLT DHCP flows
+
+Connect to the [VOLTHA CLI](../../../charts/voltha.md#accessing-the-voltha-cli)
+and check the logical device flows
+
+```shell
+(voltha) logical_device 000100000a5a007a
+(logical device 000100000a5a007a) flows
+Logical Device 000100000a5a007a (type: n/a)
+Flows (12):
++----------+----------+-----------+---------+----------+----------+----------+---------+---------+--------------+--------------+----------+-----------+------------+------------+
+| table_id | priority | cookie | in_port | vlan_vid | eth_type | ip_proto | udp_src | udp_dst | metadata | set_vlan_vid | pop_vlan | push_vlan | output | goto-table |
++----------+----------+-----------+---------+----------+----------+----------+---------+---------+--------------+--------------+----------+-----------+------------+------------+
+-snip- |
+| 0 | 10000 | ~1ec4282a | 65536 | | 800 | 17 | 67 | 68 | | | | | CONTROLLER | |
+-snip-
+| 0 | 10000 | ~68a65ba0 | 30736 | | 800 | 17 | 68 | 67 | | | | | CONTROLLER | |
+```
+
+Note that the second flow above is necessary because it traps DHCP packets from the
+client (RG) - input port `30736` which is the UNI portnumber and `udp_src` port 68.
+The first flow above is not really required as it is meant to trap DHCP packets from
+the server on the OLT NNI port (in_port=65536). This is because in the SEBA POD,
+these packets from the server will never reach the OLT NNI port as they will be trapped
+earlier at the AGG switch uplink port.
+
+### AGG Switch DHCP flows
+
+There is only 1 flow that is essential for DHCP relay in the AGG switch.
+
+Connect to the [ONOS CLI](../../../charts/onos.md#accessing-the-onos-cli)
+and check the flows in the agg switch
+
+```shell
+onos> flows -s any of:0000000000000002 60
+deviceId=of:0000000000000002, flowRuleCount=12
+ ADDED, bytes=0, packets=0, table=60, priority=61000, selector=[IN_PORT:32, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:67, UDP_DST:68], treatment=[immediate=[OUTPUT:CONTROLLER], clearDeferred]
+-snip-
+```
+
+Ensure that you are using the correct AGG switch dpid, for example `of:0000000000000002`,
+and you are asking for the flows from Table 60 (the ACL table). As shown above,
+this flow matches on DHCP packets with UDP src port 67, and incoming port 32 on the
+AGG switch, which is where the DHCP server (BNG) is connected.
+
+## DHCP relay logs
+
+If all the flows look correct, you may wish to check ONOS logs for the DHCP transactions.
+First it will be useful to set the logging to TRACE for the dhcpl2relay app.
+From the ONOS CLI, use the following commands
+
+```shell
+onos> log:set TRACE org.opencord.dhcpl2relay
+onos> log:list
+Logger | Level
+--------------------------------
+ROOT | INFO
+org.opencord.aaa | DEBUG
+org.opencord.dhcpl2relay | TRACE
+```
+
+Next, check the logs by grepping for the app logs
+
+```shell
+onos> log:tail | grep DhcpL2Relay
+
+2018-12-20 22:42:06,464 | INFO | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Received DHCP Packet of type DHCPDISCOVER from of:000000000a5a007a/30736
+2018-12-20 22:42:06,474 | TRACE | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | DHCP packet received from client at of:000000000a5a007a/30736
+2018-12-20 22:42:07,597 | DEBUG | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | option82data [id:ISKT71e80080,cTag:222,sTag:222,nasPortId:PON 1/1/04/1:1.1.1,uplinkPort:-1,slot:-1,hardwareIdentifier:null,ipaddress:null,nasId:null,circuitId:foo2,remoteId:bar2]
+2018-12-20 22:42:07,600 | INFO | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Finished processing packet.. relaying to dhcpServer
+2018-12-20 22:42:07,603 | TRACE | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Relaying packet to dhcp server at of:0000000000000002/32
+2018-12-20 22:42:08,000 | INFO | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Received DHCP Packet of type DHCPOFFER from of:0000000000000002/32
+2018-12-20 22:42:08,002 | TRACE | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | DHCP packet received from server at of:0000000000000002/32
+2018-12-20 22:42:08,026 | INFO | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Finished processing packet.. relaying to client
+2018-12-20 22:42:08,028 | INFO | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Sending DHCP packet to client at of:000000000a5a007a/30736
+2018-12-20 22:42:08,029 | TRACE | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Relaying packet to dhcp client at of:000000000a5a007a/30736
+2018-12-20 22:42:08,088 | INFO | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Received DHCP Packet of type DHCPREQUEST from of:000000000a5a007a/30736
+2018-12-20 22:42:08,088 | TRACE | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | DHCP packet received from client at of:000000000a5a007a/30736
+2018-12-20 22:42:08,092 | DEBUG | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | option82data [id:ISKT71e80080,cTag:222,sTag:222,nasPortId:PON 1/1/04/1:1.1.1,uplinkPort:-1,slot:-1,hardwareIdentifier:null,ipaddress:null,nasId:null,circuitId:foo2,remoteId:bar2]
+2018-12-20 22:42:08,093 | INFO | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Finished processing packet.. relaying to dhcpServer
+2018-12-20 22:42:08,095 | TRACE | 33.102.179:59154 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Relaying packet to dhcp server at of:0000000000000002/32
+2018-12-20 22:42:08,148 | INFO | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Received DHCP Packet of type DHCPACK from of:0000000000000002/32
+2018-12-20 22:42:08,148 | TRACE | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | DHCP packet received from server at of:0000000000000002/32
+2018-12-20 22:42:08,153 | INFO | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Finished processing packet.. relaying to client
+2018-12-20 22:42:08,158 | INFO | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Sending DHCP packet to client at of:000000000a5a007a/30736
+2018-12-20 22:42:08,159 | TRACE | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Relaying packet to dhcp client at of:000000000a5a007a/30736
+```
+
+Notice the app receives DHCPDISCOVER and DHCPREQUEST packets from the client RG at
+`of:000000000a5a007a/30736` and relays them to the server at `of:0000000000000002/32`.
+
+Similarly, the app receives DHCPOFFER and DHCPACK packets from `of:0000000000000002/32`
+and relays them to the client at `of:000000000a5a007a/30736`.
+
+Also note that the option-82 data is added to the packet before relaying to the
+server. This data, related to the subscriber is top down provisioned by the [operator
+in the AT&T workflow](../configuration.md). Without this configuration, DHCP will
+not work.
+
+If you wish to see the packets, check the logs (in TRACE) without
+the grep in the previous command. For example,
+
+```shell
+onos> log:tail
+-snip-
+2018-12-20 22:42:08,148 | INFO | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | Received DHCP Packet of type DHCPACK from of:0000000000000002/32
+2018-12-20 22:42:08,148 | TRACE | 0.90.0.120:46211 | DhcpL2Relay | 186 - org.opencord.dhcpl2relay - 1.5.0 | DHCP packet received from server at of:0000000000000002/32
+ip
+dl_qinqVlan: 222
+dl_qinqVlan_pcp: 0
+dl_vlan: 222
+dl_vlan_pcp: 0
+dl_src: 00900b237d11
+dl_dst: 90e2ba82f975
+nw_src: 10.22.2.254
+nw_dst: 10.22.2.10
+nw_tos: 16
+nw_proto: 17
+tp_src: 67
+tp_dst: 68
+-snip-
+```
+
+shows the DHCP ACK packet received from the server with outer S-tag 222 (`dl_qinqVlan`)
+and the inner C-tag 222 (`dl_vlan`). Note that packets received from the client
+will only have the C-tag.
+
+You can also check the DHCP allocation on the ONOS CLI
+
+```shell
+onos> dhcpl2relay-allocations
+SubscriberId=PON 1/1/04/1:1.1.1,ConnectPoint=of:000000000a5a007a/30736,State=DHCPACK,MAC=90:E2:BA:82:F9:75,CircuitId=None,IP Allocated=10.22.2.10,Allocation Timestamp=2018-12-20T22:42:08.153Z
+```
+
+and in the XOS GUI, the relevant AAA authentication and DHCP status.
+
+
diff --git a/profiles/seba/troubleshoot/no-onus.md b/profiles/seba/troubleshoot/no-onus.md
index 5412c69..02297db 100644
--- a/profiles/seba/troubleshoot/no-onus.md
+++ b/profiles/seba/troubleshoot/no-onus.md
@@ -1 +1,68 @@
# ONUs are not getting discovered
+
+If ONUs are not getting discovered you can debug it following these steps.
+
+## Check ONOS CLI
+
+Check if the ONU is displayed in the ``ports`` command in the ONOS CLI. The portName corresponds to the serial number of the ONU.
+```shell
+onos> ports
+id=of:0000626273696d62, available=true, local-status=connected 4m4s ago, role=MASTER, type=SWITCH, mfr=VOLTHA Project, hw=, sw=, serial=bbsim:50060, chassis=626273696d62, driver=voltha, channelId=172.17.0.16:43266, managementAddress=172.17.0.16, protocol=OF_13
+ port=2064, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:01:08:10, portName=BBSM00000100
+```
+
+If the port corresponding to the ONU is not displayed or has ``adminState=disabled``, then check the VOLTHA CLI.
+
+## Check VOLTHA CLI
+
+Check if the ONUs shows up in the ``devices`` in VOLTHA CLI, and it is represented in the ``ports`` of the ``logical_device`` in the VOLTHA CLI.
+
+```shell
+(voltha) devices
+Devices:
++------------------+-------------------+------+------------------+---------------+-------------+-------------+----------------+----------------+---------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
+| id | type | root | parent_id | serial_number | admin_state | oper_status | connect_status | parent_port_no | host_and_port | reason | proxy_address.device_id | proxy_address.channel_id | proxy_address.onu_id | proxy_address.onu_session_id |
++------------------+-------------------+------+------------------+---------------+-------------+-------------+----------------+----------------+---------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
+| 000129f21f7b4032 | openolt | True | 0001626273696d62 | bbsim:50060 | ENABLED | ACTIVE | REACHABLE | | bbsim:50060 | | | | | |
+| 0001969090d72daf | brcm_openomci_onu | True | 000129f21f7b4032 | BBSM00000100 | ENABLED | ACTIVE | REACHABLE | 536870913 | | initial-mib-downloaded | 000129f21f7b4032 | 1 | 1 | 1 |
++------------------+-------------------+------+------------------+---------------+-------------+-------------+----------------+----------------+---------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
+(voltha) logical_devices
+Logical devices:
++------------------+------------------+------------------+-----------------+---------------------------+--------------------------+
+| id | datapath_id | root_device_id | desc.serial_num | switch_features.n_buffers | switch_features.n_tables |
++------------------+------------------+------------------+-----------------+---------------------------+--------------------------+
+| 0001626273696d62 | 0000626273696d62 | 000129f21f7b4032 | bbsim:50060 | 256 | 2 |
++------------------+------------------+------------------+-----------------+---------------------------+--------------------------+
+(voltha) logical_device 0001626273696d62
+(logical device 0001626273696d62) ports
+Logical device ports:
++-----------+------------------+----------------+-----------+------------------+----------------------------+---------------+----------------+---------------+---------------------+------------------------+
+| id | device_id | device_port_no | root_port | ofp_port.port_no | ofp_port.hw_addr | ofp_port.name | ofp_port.state | ofp_port.curr | ofp_port.curr_speed | ofp_port_stats.port_no |
++-----------+------------------+----------------+-----------+------------------+----------------------------+---------------+----------------+---------------+---------------------+------------------------+
+| nni-65536 | 000129f21f7b4032 | 65536 | True | 65536 | [0L, 0L, 0L, 1L, 0L, 0L] | nni-65536 | 4 | 4128 | 32 | 65536 |
+| uni-2064 | 0001969090d72daf | 2064 | | 2064 | [8L, 0L, 0L, 1L, 8L, 16L] | BBSM00000100 | 4 | 4160 | 64 | |
++-----------+------------------+----------------+-----------+------------------+----------------------------+---------------+----------------+---------------+---------------------+------------------------+
+```
+
+If the port corresponding to the ONU is not displayed then check the physical OLT. If the port corresponding to the UNI appears in the ``logical_device``, then check the port status of the ONU device:
+
+```shell
+(voltha) device 0001969090d72daf
+(device 0001969090d72daf) ports
+Device ports:
++---------+----------+--------------+-------------+-------------+------------------+-------------------------------------------------------+
+| port_no | label | type | admin_state | oper_status | device_id | peers |
++---------+----------+--------------+-------------+-------------+------------------+-------------------------------------------------------+
+| 100 | PON port | PON_ONU | ENABLED | ACTIVE | 0001969090d72daf | [{'port_no': 2064, 'device_id': u'000129f21f7b4032'}] |
+| 2064 | uni-2064 | ETHERNET_UNI | ENABLED | ACTIVE | 0001969090d72daf | |
++---------+----------+--------------+-------------+-------------+------------------+-------------------------------------------------------+
+```
+If the ``oper_state`` of the port is not ``ACTIVE`` then check the following.
+
+### ONU ``oper_state`` is DISCOVERED
+
+A common reason for the ONU's ``oper_state`` to remain in the ``DISCOVERED`` state is if the ONU's vendor-id is not recognized by any of the ONU adapters in VOLTHA.
+
+### ONU ``oper_state`` is UNKNOWN
+
+The ONU can get into this state if it has been admin disabled.
diff --git a/profiles/seba/troubleshoot/screenshots/dhcp-config.png b/profiles/seba/troubleshoot/screenshots/dhcp-config.png
new file mode 100644
index 0000000..3a0d789
--- /dev/null
+++ b/profiles/seba/troubleshoot/screenshots/dhcp-config.png
Binary files differ
diff --git a/profiles/seba/troubleshoot/screenshots/dhcp-state.png b/profiles/seba/troubleshoot/screenshots/dhcp-state.png
new file mode 100644
index 0000000..ba1eccc
--- /dev/null
+++ b/profiles/seba/troubleshoot/screenshots/dhcp-state.png
Binary files differ
diff --git a/profiles/seba/troubleshoot/screenshots/olt-uplink.png b/profiles/seba/troubleshoot/screenshots/olt-uplink.png
new file mode 100644
index 0000000..a5beb60
--- /dev/null
+++ b/profiles/seba/troubleshoot/screenshots/olt-uplink.png
Binary files differ
diff --git a/profiles/seba/troubleshoot/screenshots/switch-uplink.png b/profiles/seba/troubleshoot/screenshots/switch-uplink.png
new file mode 100644
index 0000000..a01d834
--- /dev/null
+++ b/profiles/seba/troubleshoot/screenshots/switch-uplink.png
Binary files differ
diff --git a/profiles/seba/workflows/att-configure.md b/profiles/seba/workflows/att-configure.md
index 4426858..df04375 100644
--- a/profiles/seba/workflows/att-configure.md
+++ b/profiles/seba/workflows/att-configure.md
@@ -5,7 +5,7 @@
## Whitelist population
-> NOTE: the whitelist can be populated at any time.
+> NOTE: the whitelist can be populated at any time.
> It doesn't need to be done upfront.
To configure the ONU whitelist, you can use this TOSCA:
@@ -40,7 +40,7 @@
For instructions on how to push TOSCA into a CORD POD, please
refer to this [guide](../../../xos-tosca/README.md).
-### Where to find the required informations
+### Where to find the required information
#### Serial Number
