Adding offline installation docs
Change-Id: I1108311a20e78fd20651f30d8b5e95ace859c8c6
diff --git a/SUMMARY.md b/SUMMARY.md
index c9819fd..5ad8044 100644
--- a/SUMMARY.md
+++ b/SUMMARY.md
@@ -15,7 +15,6 @@
* [Multi-Node](prereqs/k8s-multi-node.md)
* [Helm](prereqs/helm.md)
* [Optional Packages](prereqs/optional.md)
- * [Docker Registry](prereqs/docker-registry.md)
* [OpenStack](prereqs/openstack-helm.md)
* [Fabric Software Setup](fabric-setup.md)
* [Bringing Up CORD](profiles/intro.md)
@@ -25,6 +24,7 @@
* [Emulated OLT/ONU](profiles/rcord/emulate.md)
* [M-CORD](profiles/mcord/install.md)
* [EnodeB Setup](profiles/mcord/enodeb-setup.md)
+ * [Offline POD install](offline-install.md)
* [Helm Reference](charts/helm.md)
* [XOS-CORE](charts/xos-core.md)
* [ONOS](charts/onos.md)
diff --git a/charts/voltha.md b/charts/voltha.md
index 61efa0b..26641a0 100644
--- a/charts/voltha.md
+++ b/charts/voltha.md
@@ -78,13 +78,11 @@
cd ~/cord/incubator/voltha
REPOSITORY=voltha/ TAG=dev VOLTHA_BUILD=docker make build
cd ~/cord/automation-tools/developer
-bash tag_and_push.sh dev 192.168.99.100:30500
+bash tag_and_push.sh -t dev -r 192.168.99.100:30500
```
_This set of commands will build the VOLTHA containers and push them to a local
-[docker registry](../prereqs/docker-registry.md) using a TAG called `dev`_
-
-> NOTE: Read more about the `tag_and_push` script [here](../prereqs/docker-registry.md)
+[docker registry](../partials/push-images-to-registry.md) using a TAG called *dev*.
Once the images are pushed to a docker registry on the POD,
you can use a values file like the following one:
diff --git a/developer/workflows.md b/developer/workflows.md
index ea8b5da..f57f946 100644
--- a/developer/workflows.md
+++ b/developer/workflows.md
@@ -93,14 +93,14 @@
> the containers, you can just delete the corresponding POD and kubernetes will
> restart it with the new image.
-## Pushing Changes to a Remote Registry
+## Pushing Changes to a docker registry
If you have a remote POD that you want to test your changes on, you
-need to push your docker images to a registry that can be accessed
+need to push your docker images to a docker registry that can be accessed
from the POD.
-The way we recommend doing this is via a private docker-registry.
+The way we recommend doing this is via a private docker registry.
You can find more informations about what a
-docker-registry is [here](../prereqs/docker-registry.md).
+docker registry is in the [offline installation section](../offline-install.md).
{% include "/partials/push-images-to-registry.md" %}
diff --git a/offline-install.md b/offline-install.md
new file mode 100644
index 0000000..b3a0f34
--- /dev/null
+++ b/offline-install.md
@@ -0,0 +1,314 @@
+# Offline installation
+
+Often, CORD PODs' (management networks) don't have access to Internet.
+
+This section of the guide provides guidelines, best-practices and examples to deploy the CORD/SEBA software without Internet connectivity.
+
+> NOTE: The guide assumes that the Operating Systems (both on servers and on network devices) and Kubernetes are already installed and running.
+
+The offline installation is useful
+
+* When the CORD POD has no access to Internet, so artifacts used for the installation (i.e. Docker images) cannot be downloaded directly to the POD.
+
+* While developing, you may want to test your changes pushing artifacts to the POD, before uploading them to the official docker repository.
+
+## Target Infrastructure, requirements overview
+
+Your target infrastructure (where the POD runs) needs
+
+* A **local Docker Registry** where to push your Docker images (previously pulled from the web). If you don't have one, follow the notes below to deploy with helm a local Docker registry on top of your existing Kubernetes cluster.
+
+> More informations about docker registries can be found at <https://docs.docker.com/registry/>.
+
+* A **local webserver** to host the ONOS applications (.oar files), that are instead normally downloaded from Sonatype. If you don't have one, follow the notes below to quickly deploy a webserver on top of your existing Kubernetes cluster.
+
+* **Kubernetes servers default gateway**: In order for kube-dns to work, a default route (even pointing to a non-exisiting/working gateway) needs to be set on the machines hosting Kubernetes. This is something related to Kubernetes, not to CORD.
+
+## Prepare the offline installation
+
+This should be done from a machine that has access to Internet.
+
+* Clone the *helm-charts* repository
+
+```shell
+git clone https://gerrit.opencord.org/helm-charts
+```
+
+Next steps largely depend on the type of profile you want to install.
+
+* Add external helm repositories and pull external dependencies.
+
+* Fetch helm-charts not available locally.
+
+* Modify the CORD helm charts to instruct kubernetes to pull the images from your local registry (where images will be pushed), instead of DockerHub. One option is to modify the *values.yaml* in each chart. A better option consists in extending the charts, rather than directly modifying them. This way, the original configuration can be kept as is, just overriding some values as needed. You can do this by writing your additional configuration yaml file, and parsing it as needed, adding `-f my-additional-config.yml` while using the helm install/upgrade commands. The full CORD helm charts reference documentation is available [here](../charts/helm.md).
+
+* Download the ONOS applications (OAR files). Informations about the oar applications used can be found here: <https://github.com/opencord/onos-service/blob/master/xos/synchronizer/steps/sync_onos_app.py#L125-L130>
+
+* Pull from DockerHub all the Docker images that need to be used on your POD. The *automation-tools* repository has a *images_from_charts.sh* utility inside the *developer* folder that can help you to get all the image names given the helm-chart repository and a list of chart names. More informations in the sections below.
+
+* Override the default helm chart values for the ONOS applications download links, pointing them to the HTTP addresses of the files, once they'll be on your local web server.
+
+* Optionally download the OpenOLT driver deb files from your vendor website (i.e. EdgeCore).
+
+* Optionally, save as tar files the Docker images downloaded. This can be useful if you'll use a different machine to upload the images on the local registry running in your infrastructure. To do that, for each image use the Docker command.
+
+```shell
+docker save IMAGE_NAME:TAG > FILE_NAME.tar
+```
+
+* If the artifacts need to be deployed to the target infrastructure from a different machine, save the helm-charts directory, the ONOS applications, the docker images downloaded and the additional helm charts variable extension file.
+
+## Deploy the artifacts to your infrastructure and install CORD/SEBA
+
+This should not require any Internet connectivity. To deploy the artifacts to your POD, do the following from machine that has access to your Kubernetes cluster:
+
+* Optionally, if at the previous step you saved the Docker images on an external hard drive as .tar files, restore them in the deployment machine Docker registry. For each image (file), use the docker command
+
+```shell
+docker load < FILE_NAME.tar
+```
+
+* Tag and push your Docker images to the local Docker registry running in your infrastructure. More info on this can be found in the paragraph below.
+
+* Copy the ONOS applications to your local web server. The procedure largely varies from the web server you run, its configuration, and what ONOS applications you need.
+
+* Deploy CORD/SEBA using the helm charts. Remember to load with the *-f* option the additional configuration file to extend the helm charts, if any.
+
+{% include "/partials/push-images-to-registry.md" %}
+
+## Optional packages
+
+### Install a Docker Registry using helm
+
+If you don't have a local Docker registry deployed in your infrastructure, you can install an **insecure** one using the official Kubernetes helm-chart.
+
+> **Note:** *Insecure* registries can be used for development, POCs or lab trials. **You should not use this in production.** There are planty of documents online that guide you through secure registries setup.
+
+The following command deploys the registry and exposes the port *30500*. (You may want to change it with any value that fit your deployment needs).
+
+```shell
+helm install stable/docker-registry --set service.nodePort=30500,service.type=NodePort -n docker-registry
+```
+
+The registry can be queried at any time, for example:
+
+```shell
+curl -X GET http://KUBERNETES_IP:30500/v2/_catalog
+```
+
+### Install a local web server using helm (optional)
+
+If you don't have a local web server that can be accessed from the POD, you can easily install one on top of your existing Kubernetes cluster.
+
+```shell
+# From the helm-charts directory, while preparing the offline install
+helm repo add bitnami https://charts.bitnami.com/bitnami
+helm fetch bitnami/nginx --untar
+
+# Then, while deploying offline
+helm install -n maven-repo bitnami/nginx
+```
+
+The webserver will be up in few seconds and you'll be able to reach the root web page using the IP of one of your Kubernetes nodes, port *30278*. For example, you can do:
+
+```shell
+wget KUBERNETES_IP:30278
+```
+
+OAR images can be copied to the document root of the web server using the *kubectl cp* command. For example:
+
+```shell
+kubectl cp my-onos-app.oar `kubectl get pods | grep maven-repo | awk '{print $1;}'`:/opt/bitnami/nginx/html
+```
+
+## Example: offline SEBA install
+
+The following section provides an exemplary list of commands to perform an offline SEBA POD installation. Please, note that some command details (i.e. chart names, image names, tools) may have changed over time.
+
+### Assumptions
+
+* The Operating Systems (both on servers and network devices) is already installed and running.
+
+* The EdgeCore asfvolt16 OLT is used as access device. The device has an IP address 192.168.0.200.
+
+* For convenience, the example makes also use of some automation tools, also described in paragraphs above.
+
+* The same machine used to prepare the installation files (while still connected to Internet) is also used to perform the offline install.
+
+* The infrastructure doesn't have a local Docker registry. As such, a dedicated local Docker registry is deployed on top of the existing Kubernetes cluster, while the Kubernetes cluster can still access Internet.
+
+* The infrastructure doesn't have a local web server. As such, an additional web server (to host ONOS applications) is deployed on top of the existing Kubernetes cluster.
+
+* Configuration files for SEBA have been already created. They live in the root directory and they are called fabric.yaml, olt.yaml, subscriber.yaml.
+
+* The IP address of the machine hosting Kubernetes is 192.168.0.100.
+
+### Prepare the installation
+
+```shell
+# Clone repos
+git clone https://gerrit.opencord.org/automation-tools
+git clone https://gerrit.opencord.org/helm-charts
+
+# Copy automation scripts in the right place
+cp automation-tools/developer/images_from_charts.sh helm-charts
+cp automation-tools/developer/pull_images.sh helm-charts
+cp automation-tools/developer/tag_and_push.sh helm-charts
+
+cd helm-charts
+
+# Add online helm repositories and update dependencies
+helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
+helm repo add rook-beta https://charts.rook.io/beta
+helm repo add bitnami https://charts.bitnami.com/bitnami
+helm dep update voltha
+helm dep update xos-core
+helm dep update xos-profiles/att-workflow
+xos-profiles/base-kubernetes
+helm dep update nem-monitoring
+helm dep update logging
+helm dep update storage/rook-operator
+
+# Fetch helm-charts not available locally
+helm fetch stable/docker-registry --untar
+helm fetch --version 0.8.0 stable/etcd-operator --untar
+helm fetch --version 0.8.8 incubator/kafka --untar
+helm fetch bitnami/nginx --untar
+
+# Update Kafka dependencies
+helm dep update kafka
+
+# For demo, install the local, helm-based Docker Registry on the remote POD (this will require POD connectivity to download the docker registry image)
+helm install stable/docker-registry --set service.nodePort=30500,service.type=NodePort -n docker-registry
+
+# For demo, install the local web-server to host ONOS images
+helm install -n maven-repo bitnami/nginx
+
+# Identify images form the official helm charts and pull images from DockerHub. If you see some "skipped value for filters" warning that's fine
+bash images_from_charts.sh etcd-cluster etcd-operator voltha onos xos-core xos-profiles/att-workflow xos-profiles/base-kubernetes nem-monitoring logging storage/rook-operator | bash pull_images.sh > images
+
+# Download ONOS apps
+wget https://oss.sonatype.org/service/local/repositories/snapshots/content/org/opencord/olt-app/2.1.0-SNAPSHOT/olt-app-2.1.0-20181030.071543-35.oar
+wget https://oss.sonatype.org/service/local/repositories/snapshots/content/org/opencord/sadis-app/2.2.0-SNAPSHOT/sadis-app-2.2.0-20181030.071559-154.oar
+wget https://oss.sonatype.org/service/local/repositories/snapshots/content/org/opencord/dhcpl2relay/1.5.0-SNAPSHOT/dhcpl2relay-1.5.0-20181030.071500-154.oar
+wget https://oss.sonatype.org/service/local/repositories/snapshots/content/org/opencord/aaa/1.8.0-SNAPSHOT/aaa-1.8.0-20181113.081456-110.oar
+wget https://oss.sonatype.org/service/local/repositories/snapshots/content/org/opencord/kafka/1.0.0-SNAPSHOT/kafka-1.0.0-20181030.071524-104.oar
+
+# Create file to extend the helm charts (call it extend.yaml)
+global:
+ registry: 192.168.0.100:30500/
+
+image: 192.168.0.100:30500/confluentinc/cp-kafka
+imageTag: 4.1.2-2
+
+zookeeper:
+ image:
+ repository: 192.168.0.100:30500/gcr.io/google_samples/k8szk
+
+etcd-cluster:
+ spec:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd
+ pod:
+ busyboxImage: 192.168.0.100:30500/busybox:1.28.1-glibc
+
+etcdOperator:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
+backupOperator:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
+restoreOperator:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/etcd-operator
+
+grafana:
+ image:
+ repository: 192.168.0.100:30500/grafana/grafana
+ sidecar:
+ image: 192.168.0.100:30500/kiwigrid/k8s-sidecar:0.0.3
+
+prometheus:
+ server:
+ image:
+ repository: 192.168.0.100:30500/prom/prometheus
+ alertmanager:
+ image:
+ repository: 192.168.0.100:30500/prom/alertmanager
+ configmapReload:
+ image:
+ repository: 192.168.0.100:30500/jimmidyson/configmap-reload
+ kubeStateMetrics:
+ image:
+ repository: 192.168.0.100:30500/quay.io/coreos/kube-state-metrics
+ nodeExporter:
+ image:
+ repository: 192.168.0.100:30500/prom/node-exporter
+ pushgateway:
+ image:
+ repository: 192.168.0.100:30500/prom/pushgateway
+ initChownData:
+ image:
+ repository: 192.168.0.100:30500/busybox
+
+elasticsearch:
+ image:
+ repository: 192.168.0.100:30500/docker.elastic.co/elasticsearch/elasticsearch-oss
+ initImage:
+ repository: 192.168.0.100:30500/busybox
+
+kibana:
+ image:
+ repository: 192.168.0.100:30500/docker.elastic.co/kibana/kibana-oss
+
+logstash:
+ image:
+ repository: 192.168.0.100:30500/docker.elastic.co/logstash/logstash-oss
+
+oltAppUrl: http://192.168.0.100:30278/olt-app-2.1.0-20181030.071543-35.oar
+sadisAppUrl: http://192.168.0.100:30278/sadis-app-2.2.0-20181030.071559-154.oar
+dhcpL2RelayAppUrl: http://192.168.0.100:30278/dhcpl2relay-1.5.0-20181030.071500-154.oar
+aaaAppUrl: http://192.168.0.100:30278/aaa-1.8.0-20181113.081456-110.oar
+kafkaAppUrl: http://192.168.0.100:30278/kafka-1.0.0-20181030.071524-104.oar
+
+# Download the openolt.deb driver installation file from the vendor website (command varies)
+scp/wget... openolt.deb
+```
+
+### Offline deployment
+
+```shell
+cd helm-charts
+
+# Tag and push the images to the local Docker registry
+cat images | bash tag_and_push.sh -r 192.168.0.100:30500
+
+# Copy the ONOS applications to the local web server
+MAVEN_REPO=$(kubectl get pods | grep maven-repo | awk '{print $1;}')
+kubectl cp olt-app-2.1.0-20181030.071543-35.oar $MAVEN_REPO:/opt/bitnami/nginx/html
+kubectl cp sadis-app-2.2.0-20181030.071559-154.oar $MAVEN_REPO:/opt/bitnami/nginx/html
+kubectl cp dhcpl2relay-1.5.0-20181030.071500-154.oar $MAVEN_REPO:/opt/bitnami/nginx/html
+kubectl cp aaa-1.8.0-20181113.081456-110.oar $MAVEN_REPO:/opt/bitnami/nginx/html
+kubectl cp kafka-1.0.0-20181030.071524-104.oar $MAVEN_REPO:/opt/bitnami/nginx/html
+
+# Install SEBA
+helm install -n etcd-operator -f extend.yaml --version 0.8.0 etcd-operator
+helm install -n cord-kafka -f examples/kafka-single.yaml -f extend.yaml --version 0.8.8 kafka
+helm install -n voltha -f extend.yaml voltha
+helm install -n onos -f configs/onos.yaml -f extend.yaml onos
+helm install -n xos-core -f extend.yaml xos-core
+helm install -n att-workflow -f extend.yaml xos-profiles/att-workflow
+helm install -n base-kubernetes -f extend.yaml xos-profiles/base-kubernetes
+helm install -n nem-monitoring -f extend.yaml nem-monitoring
+helm install -n logging -f extend.yaml logging
+
+# On the OLT, copy, install and run openolt.deb
+scp openolt.deb root@192.168.0.200:
+ssh root@192.168.0.200 'dpkg -i openolt.deb'
+ssh root@192.168.0.200 'service bal_core_dist start'
+ssh root@192.168.0.200 'service openolt start'
+
+# Configure SEBA
+curl -H "xos-username: admin@opencord.org" -H "xos-password: letmein" -X POST --data-binary @fabric.yaml http://192.168.0.100:30007/run
+curl -H "xos-username: admin@opencord.org" -H "xos-password: letmein" -X POST --data-binary @olt.yaml http://192.168.0.100:30007/run
+curl -H "xos-username: admin@opencord.org" -H "xos-password: letmein" -X POST --data-binary @subscriber.yaml http://192.168.0.100:30007/run
+```
diff --git a/partials/push-images-to-registry.md b/partials/push-images-to-registry.md
index cac2de6..0a8d8db 100644
--- a/partials/push-images-to-registry.md
+++ b/partials/push-images-to-registry.md
@@ -35,10 +35,28 @@
The image should now be in the local docker registry on your cluster.
-## Use the tag-and-push Script
+## Identify, download, tag and push images
-Sometimes you may need to download, tag and push lots of images.
+Sometimes you may need to identify, download, tag and push lots of images.
This can become a long and error prone operation if done manually.
-For this reason, we provide an optional tool that automates the tag
-and push procedures. The script can be found
-[here](https://github.com/opencord/automation-tools/tree/master/developer).
+For this reason, we provide a set of tool that automate procedure. The script can be found [here](https://github.com/opencord/automation-tools/tree/master/developer).
+
+### image_from_helm.sh: identify images
+
+The *image_from_helm.sh* script prints the list of images used by one or multiple charts. The script needs to be executed from the *helm-charts* directory. More info can be found invoking the *-h* or *--help* option of the command. The output can be piped in other utility scripts.
+
+### pull_images.sh: pull images from DockerHub
+
+The *pull_images.sh* script pulls from DockerHub the list of images provided in input and prints the image name if the image is correctly pulled. More info can be found invoking the *-h* or *--help* option of the command. The output can be piped in other utility scripts.
+
+### tag_and_push.sh: tag and push images to a target Docker registry
+
+The *tag_and_push.sh* script tags and push images to a target Docker registry (including DockerHub itself). It can add a prefix (useful for example when deploying on local registries) to images and change the tag of an image. More info can be found invoking the *-h* or *--help* option of the command. The output can be piped in other utility scripts.
+
+### Examples:
+
+Assume you'd like to prepare an offline SEBA installation. As such, you need to identify all the images used in the charts, download them, tag them and push them to a local Docker registry (in the example, 192.168.0.100, port 30500). From the helm-charts folder, this can be done in one command:
+
+```shell
+bash images_from_charts.sh voltha onos xos-core xos-profiles/att-workflow nem-monitoring logging | bash pull_images.sh | bash tag_and_push.sh -r 192.168.0.100:30500
+```
diff --git a/prereqs/docker-registry.md b/prereqs/docker-registry.md
deleted file mode 100644
index cce78ee..0000000
--- a/prereqs/docker-registry.md
+++ /dev/null
@@ -1,50 +0,0 @@
-# Docker Registry (Optional)
-
-The section describes how to install an **insecure** *docker registry* in Kubernetes, using the standard Kubernetes helm charts.
-
-A local docker registry can be used to push container images directly to the cluster,
-which could be useful for example in the following cases:
-
-* The CORD POD has no Internet access, so container images cannot be downloaded directly from DockerHub to the POD.
-
-* You are developing new CORD components, or modifying existing ones. You may want to test your changes before uploading the image to the official docker repository. In this case, your workflow might be to build your new container and push it to the local registry.
-
-More informations about docker registries can be found at <https://docs.docker.com/registry/>.
-
-> **Note:** *Insecure* registries can be used for development, POCs or lab trials. **You should not use this in production.** There are planty of documents online that guide you through secure registry setup.
-
-## Deploy a Registry Using Helm
-
-Helm provides a default helm chart to deploy an insecure registry on your
-Kubernetes cluster. The following command deploys the registry and exposes
-the port *30500*. (You may want to change it with any value that fit your
-deployment needs.)
-
-```shell
-helm install stable/docker-registry --set service.nodePort=30500,service.type=NodePort -n docker-registry
-```
-
-The registry can be queried at any time, for example:
-
-```shell
-curl -X GET http://KUBERNETES_IP:30500/v2/_catalog
-```
-
-{% include "/partials/push-images-to-registry.md" %}
-
-## Modify the Helm Charts to Use Your Images
-
-Now that your custom images are in the local docker registry on the Kubernetes
-cluster, you can modify the CORD helm charts to instruct the system to consume
-them instead of using the default images from DockerHub.
-
-Image names and tags are specified in the *values.yaml* file of each chart
-(look in the main chart directory), or alternatively, in the configuration
-files in the config directory.
-
-Simply modify the values as needed, uninstall the containers previously deployed,
-and deploy them again.
-
-> **Note**: It is better to extend the existing helm charts, rather than directly modifying them. This way you can keep the original configuration as it is, and just override some values when needed. You can do this by writing your additional configuration yaml file, and parsing it as needed, adding `-f my-additional-config.yml` to your helm commands.
-
-The full CORD helm charts reference documentation is available [here](../charts/helm.md).
diff --git a/prereqs/optional.md b/prereqs/optional.md
index 370041e..e19dbe3 100644
--- a/prereqs/optional.md
+++ b/prereqs/optional.md
@@ -1,13 +1,6 @@
# Optional Packages
-Although not required, you may want to install one or both of the following
-packages:
-
-* **Local Registry:** If your environment does not permit connecting your
- POD to ther public Internet, you may want to take advantage of a local Docker
- registery. The following [registry setup](docker-registry.md) will help.
- (Having a local registry is also useful when doing local development, as outlined
- in the [Developer Guide](../developer/workflows.md).)
+Although not required, you may want to install the following packages:
* **OpenStack:** If you need to include OpenStack in your deployment,
so you can bring up VMs on your POD, you will need to following the