Setting up an Ubuntu Development Environment
============================================

These notes describe the checking out and building from the multiple
gerrit repositories needed to run a VOLTHA 2.x environment with
docker-compose. Starting point is a basic Ubuntu 16.04 or 18.04
installation with internet access.

These notes are intended for iterative development only. The testing
environments and production environments will run a Kubernetes and Helm
based deployment.

Install prerequisites
---------------------

Patch and updated

.. code:: sh

   sudo apt update
   sudo apt dist-upgrade

Add ``docker-ce`` apt repo and install docker and build tools

.. code:: sh

   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
   sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
   sudo apt update
   sudo apt install build-essential docker-ce git

Install current ``docker-compose``. Older versions may cause docker
build problems:
https://github.com/docker/docker-credential-helpers/issues/103

.. code:: sh

   sudo curl -L "https://github.com/docker/compose/releases/download/1.25.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   sudo chmod 755 /usr/local/bin/docker-compose

Install the Golang ppa apt repository and install **golang 1.13**.

.. code:: sh

   sudo add-apt-repository ppa:longsleep/golang-backports
   sudo apt update
   sudo apt install golang-1.13

Setup environment
~~~~~~~~~~~~~~~~~

Setup a local Golang and docker-compose environment, verifying the
golang-1.13 binaries are in your path. Also add your local ``GOPATH``
bin folder to ``PATH`` Add export statements to your ``~/.profile`` to
persist.

.. code:: sh

   mkdir $HOME/source
   mkdir $HOME/go
   export GO111MODULE=on
   export GOPATH=$HOME/go
   export DOCKER_TAG=latest
   export PATH=$PATH:/usr/lib/go-1.13/bin:$GOPATH/bin
   go version

Allow your current non-root user ``$USER`` docker system access

.. code:: sh

   sudo usermod -a -G docker $USER

Logout/Login to assume new group membership needed for running docker as
non-root user and verify any environment variables set in
``~/.profile``.

Checkout source and build images
--------------------------------

VOLTHA 2.x Core Containers
~~~~~~~~~~~~~~~~~~~~~~~~~~

Checkout needed source from gerrit. Build the ``voltha-rw-core`` docker
image.

.. code:: sh

   cd ~/source/
   git clone https://gerrit.opencord.org/voltha-go.git
   cd ~/source/voltha-go
   make build

For more details regarding building and debugging the 2.x core outside
of Docker refer to voltha-go BUILD.md.

https://github.com/opencord/voltha-go/blob/master/BUILD.md

VOLTHA 2.x OpenOLT Container
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Checkout needed source from gerrit. Build the ``voltha-openolt-adapter``
docker image.

.. code:: sh

   cd ~/source/
   git clone https://gerrit.opencord.org/voltha-openolt-adapter.git
   cd ~/source/voltha-openolt-adapter/
   make build

VOLTHA 2.x OpenONU Container
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Checkout needed source from gerrit. Build the ``voltha-openonu-adapter``
docker image.

.. code:: sh

   cd ~/source/
   git clone https://gerrit.opencord.org/voltha-openonu-adapter.git
   cd ~/source/voltha-openonu-adapter/
   make build

VOLTHA 2.x OFAgent
~~~~~~~~~~~~~~~~~~

Checkout needed source from gerrit. Build the ``voltha-ofagent`` docker
image.

.. code:: sh

   cd ~/source/
   git clone https://gerrit.opencord.org/ofagent-go.git
   cd ~/source/ofagent-go/
   make docker-build

ONOS Container with VOLTHA Compatible Apps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By default the standard ONOS docker image does not contain nor start any
apps needed by VOLTHA. If you use the standard image then you need to
use the ONOS restful API to load needed apps separately.

For development convenience there is an ONOS docker image build that
adds in the current compatible VOLTHA apps. Checkout and build the ONOS
image with added ONOS apps (olt, aaa, sadis, dhcpl2relay, and kafka).

.. code:: sh

   cd ~/source/
   git clone https://gerrit.opencord.org/voltha-onos.git
   cd ~/source/voltha-onos
   make build

Install voltctl VOLTHA Command Line Management Tool
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A working Golang build environment is required as ``voltctl`` is build
and run directly from the host. Build the ``voltctl`` executable and
install in ~/go/bin which should already be in your $PATH

.. code:: sh

   cd ~/source/
   git clone https://gerrit.opencord.org/voltctl.git
   cd ~/source/voltctl
   make build
   make install

Configure the ``voltctl`` environment configuration files
``~/.volt/config`` and ``~/.volt/command_options`` to point at the local
``voltha-rw-core`` instance that will be running.

.. code:: sh

   mkdir ~/.volt/

   cat << EOF > ~/.volt/config
   apiVersion: v2
   server: localhost:50057
   tls:
     useTls: false
     caCert: ""
     cert: ""
     key: ""
     verify: ""
   grpc:
     timeout: 10s
   EOF

   cat << EOF > ~/.volt/command_options
   device-list:
     format: table{{.Id}}\t{{.Type}}\t{{.Root}}\t{{.ParentId}}\t{{.SerialNumber}}\t{{.Address}}\t{{.AdminState}}\t{{.OperStatus}}\t{{.ConnectStatus}}\t{{.Reason}}
     order: -Root,SerialNumber

   device-ports:
     order: PortNo

   device-flows:
     order: Priority,EthType

   logical-device-list:
     order: RootDeviceId,DataPathId

   logical-device-ports:
     order: Id

   logical-device-flows:
     order: Priority,EthType

   adapter-list:
     order: Id

   component-list:
     order: Component,Name,Id

   loglevel-get:
     order: ComponentName,PackageName,Level

   loglevel-list:
     order: ComponentName,PackageName,Level
   EOF


Install VOLTHA bbsim olt/onu Simulator (Optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you do not have physical OLT/ONU hardware you can build a simulator.

.. code:: sh

   cd ~/source/
   git clone https://gerrit.opencord.org/bbsim.git
   cd ~/source/bbsim
   make docker-build

Test
----

Startup
~~~~~~~

Run the combined docker-compose yaml configuration file that starts the
core, its dependent systems (etcd, zookeeper, and kafka) and the openonu
and openolt adapters. Export the ``DOCKER_HOST_IP`` environment variable
to your non-localhost IP address needed for inter-container
communication. This can be the IP assigned to ``eth0`` or the
``docker0`` bridge (typically 172.17.0.1)

For convenience you can also export ``DOCKER_TAG`` to signify the docker
images tag you would like to use. Though for typical development you may
have to edit ``compose/system-test.yml`` to override the specific docker
image ``DOCKER_TAG`` needed. The ``DOCKER_REGISTRY`` and
``DOCKER_REPOSITORY`` variables are not needed unless you wish to
override the docker image path. See the ``system-test.yml`` file for
details on the docker image path creation string.

.. code:: sh

   export DOCKER_HOST_IP=172.17.0.1
   export DOCKER_TAG=latest

   cd ~/source/voltha-go
   docker-compose -f compose/system-test.yml up -d


   WARNING: The DOCKER_REGISTRY variable is not set. Defaulting to a blank string.
   WARNING: The DOCKER_REPOSITORY variable is not set. Defaulting to a blank string.
   Creating network "compose_default" with driver "bridge"
   Pulling zookeeper (wurstmeister/zookeeper:latest)...
   latest: Pulling from wurstmeister/zookeeper
   a3ed95caeb02: Pull complete
   ef38b711a50f: Pull complete
   e057c74597c7: Pull complete
   666c214f6385: Pull complete
   c3d6a96f1ffc: Pull complete
   3fe26a83e0ca: Pull complete
   3d3a7dd3a3b1: Pull complete
   f8cc938abe5f: Pull complete
   9978b75f7a58: Pull complete
   4d4dbcc8f8cc: Pull complete
   8b130a9baa49: Pull complete
   6b9611650a73: Pull complete
   5df5aac51927: Pull complete
   76eea4448d9b: Pull complete
   8b66990876c6: Pull complete
   f0dd38204b6f: Pull complete
   Digest: sha256:7a7fd44a72104bfbd24a77844bad5fabc86485b036f988ea927d1780782a6680
   Status: Downloaded newer image for wurstmeister/zookeeper:latest
   Pulling kafka (wurstmeister/kafka:2.11-2.0.1)...
   2.11-2.0.1: Pulling from wurstmeister/kafka
   4fe2ade4980c: Pull complete
   6fc58a8d4ae4: Pull complete
   819f4a45746c: Pull complete
   a3133bc2e3e5: Pull complete
   72f0dc369677: Pull complete
   1e1130fc942d: Pull complete
   Digest: sha256:20d08a6849383b124bccbe58bc9c48ec202eefb373d05e0a11e186459b84f2a0
   Status: Downloaded newer image for wurstmeister/kafka:2.11-2.0.1
   Pulling etcd (quay.io/coreos/etcd:v3.2.9)...
   v3.2.9: Pulling from coreos/etcd
   88286f41530e: Pull complete
   2fa4a2c3ffb5: Pull complete
   539b8e6ccce1: Pull complete
   79e70e608afa: Pull complete
   f1bf8f503bff: Pull complete
   c4abfc27d146: Pull complete
   Digest: sha256:1913dd980d55490fa50640bbef0f4540d124e5c66d6db271b0b4456e9370a272
   Status: Downloaded newer image for quay.io/coreos/etcd:v3.2.9
   Creating compose_kafka_1           ... done
   Creating compose_cli_1             ... done
   Creating compose_adapter_openolt_1 ... done
   Creating compose_rw_core_1         ... done
   Creating compose_adapter_openonu_1 ... done
   Creating compose_etcd_1            ... done
   Creating compose_onos_1            ... done
   Creating compose_ofagent_1         ... done
   Creating compose_zookeeper_1       ... done

Verify containers have continuous uptime and no restarts

.. code:: sh

   $ docker-compose -f compose/system-test.yml ps
   WARNING: The DOCKER_REGISTRY variable is not set. Defaulting to a blank string.
   WARNING: The DOCKER_REPOSITORY variable is not set. Defaulting to a blank string.
             Name                         Command               State                                             Ports
   ---------------------------------------------------------------------------------------------------------------------------------------------------------------
   compose_adapter_openolt_1   /app/openolt --kafka_adapt ...   Up      0.0.0.0:50062->50062/tcp
   compose_adapter_openonu_1   /voltha/adapters/brcm_open ...   Up
   compose_etcd_1              etcd --name=etcd0 --advert ...   Up      0.0.0.0:2379->2379/tcp, 0.0.0.0:32929->2380/tcp, 0.0.0.0:32928->4001/tcp
   compose_kafka_1             start-kafka.sh                   Up      0.0.0.0:9092->9092/tcp
   compose_ofagent_1           /app/ofagent --controller= ...   Up
   compose_onos_1              ./bin/onos-service server        Up      6640/tcp, 0.0.0.0:6653->6653/tcp, 0.0.0.0:8101->8101/tcp, 0.0.0.0:8181->8181/tcp, 9876/tcp
   compose_rw_core_1           /app/rw_core -kv_store_typ ...   Up      0.0.0.0:50057->50057/tcp
   compose_zookeeper_1         /bin/sh -c /usr/sbin/sshd  ...   Up      0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp

.. code:: sh

   $ docker ps
   CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                                                                                        NAMES
   08a0e7a1ee5c        voltha-openolt-adapter:latest   "/app/openolt --kafk…"   31 seconds ago      Up 27 seconds       0.0.0.0:50062->50062/tcp                                                                     compose_adapter_openolt_1
   1f364cf7912d        wurstmeister/zookeeper:latest   "/bin/sh -c '/usr/sb…"   31 seconds ago      Up 27 seconds       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp                                           compose_zookeeper_1
   ab1822baed41        wurstmeister/kafka:2.11-2.0.1   "start-kafka.sh"         31 seconds ago      Up 24 seconds       0.0.0.0:9092->9092/tcp                                                                       compose_kafka_1
   22a4fe4b2eb4        voltha-ofagent-go:latest        "/app/ofagent --cont…"   31 seconds ago      Up 23 seconds                                                                                                    compose_ofagent_1
   d34e1c976db5        voltha-rw-core:latest           "/app/rw_core -kv_st…"   31 seconds ago      Up 26 seconds       0.0.0.0:50057->50057/tcp                                                                     compose_rw_core_1
   f6ef52975dc0        voltha-openonu-adapter:latest   "/voltha/adapters/br…"   31 seconds ago      Up 29 seconds                                                                                                    compose_adapter_openonu_1
   7ce8bcf7436c        voltha-onos:latest              "./bin/onos-service …"   31 seconds ago      Up 25 seconds       0.0.0.0:6653->6653/tcp, 0.0.0.0:8101->8101/tcp, 6640/tcp, 9876/tcp, 0.0.0.0:8181->8181/tcp   compose_onos_1
   60ac172726f5        quay.io/coreos/etcd:v3.4.1      "etcd --name=etcd0 -…"   31 seconds ago      Up 28 seconds       0.0.0.0:2379->2379/tcp, 0.0.0.0:32931->2380/tcp, 0.0.0.0:32930->4001/tcp                     compose_etcd_1

Verify Cluster Communication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Use ``voltctl`` commands to verify core and adapters are running.

.. code:: sh

   voltctl adapter list
   ID                   VENDOR            VERSION      SINCELASTCOMMUNICATION
   brcm_openomci_onu    VOLTHA OpenONU    2.3.2-dev    UNKNOWN
   openolt              VOLTHA OpenOLT    2.3.5-dev    UNKNOWN

List “devices” to verify no devices exist.

.. code:: sh

   voltctl device list
   ID    TYPE    ROOT    PARENTID    SERIALNUMBER    ADDRESS    ADMINSTATE    OPERSTATUS    CONNECTSTATUS    REASON

At this point create/preprovision and enable an olt device and add flows
via onos and ofagent.

Physical OLT/ONU Testing with Passing Traffic
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Start a physical OLT and ONU. Tested with Edgecore OLT, Broadcom based
ONU, and RG capable of EAPoL. Create/Preprovision the OLT and enable.
The create command returns the device ID needed for enable and
subsequent commands.

**Add device to VOLTHA**

.. code:: sh

   $ voltctl device create -t openolt -H 10.64.1.206:9191
   db87c4b48843bb99567d3d94

   $ voltctl device enable db87c4b48843bb99567d3d94
   db87c4b48843bb99567d3d94

**Verify device state**

.. code:: sh

   $ voltctl device list
   ID                          TYPE                 ROOT     PARENTID                    SERIALNUMBER    ADDRESS             ADMINSTATE    OPERSTATUS    CONNECTSTATUS    REASON
   db87c4b48843bb99567d3d94    openolt              true     a82bb53678ae                EC1721000221    10.64.1.206:9191    ENABLED       ACTIVE        REACHABLE
   082d7c2e628325ccc3336275    brcm_openomci_onu    false    db87c4b48843bb99567d3d94    ALPHe3d1cf57    unknown             ENABLED       ACTIVE        REACHABLE        omci-flows-pushed

.. code:: sh

   $ voltctl device port list db87c4b48843bb99567d3d94
   PORTNO       LABEL            TYPE            ADMINSTATE    OPERSTATUS    DEVICEID    PEERS
   1048576      nni-1048576      ETHERNET_NNI    ENABLED       ACTIVE                    []
   536870912    pon-536870912    PON_OLT         ENABLED       ACTIVE                    [{082d7c2e628325ccc3336275 536870912}]
   536870913    pon-536870913    PON_OLT         ENABLED       ACTIVE                    []
   536870914    pon-536870914    PON_OLT         ENABLED       ACTIVE                    []
   536870915    pon-536870915    PON_OLT         ENABLED       ACTIVE                    []
   536870916    pon-536870916    PON_OLT         ENABLED       ACTIVE                    []
   536870917    pon-536870917    PON_OLT         ENABLED       ACTIVE                    []
   536870918    pon-536870918    PON_OLT         ENABLED       ACTIVE                    []
   536870919    pon-536870919    PON_OLT         ENABLED       ACTIVE                    []
   536870920    pon-536870920    PON_OLT         ENABLED       ACTIVE                    []
   536870921    pon-536870921    PON_OLT         ENABLED       ACTIVE                    []
   536870922    pon-536870922    PON_OLT         ENABLED       ACTIVE                    []
   536870923    pon-536870923    PON_OLT         ENABLED       ACTIVE                    []
   536870924    pon-536870924    PON_OLT         ENABLED       ACTIVE                    []
   536870925    pon-536870925    PON_OLT         ENABLED       ACTIVE                    []
   536870926    pon-536870926    PON_OLT         ENABLED       ACTIVE                    []
   536870927    pon-536870927    PON_OLT         ENABLED       ACTIVE                    []

.. code:: sh

   $ voltctl device port list 082d7c2e628325ccc3336275
   PORTNO       LABEL       TYPE            ADMINSTATE    OPERSTATUS    DEVICEID    PEERS
   16           uni-16      ETHERNET_UNI    ENABLED       ACTIVE                    []
   17           uni-17      ETHERNET_UNI    ENABLED       DISCOVERED                []
   18           uni-18      ETHERNET_UNI    ENABLED       DISCOVERED                []
   19           uni-19      ETHERNET_UNI    ENABLED       DISCOVERED                []
   20           uni-20      ETHERNET_UNI    ENABLED       DISCOVERED                []
   536870912    PON port    PON_ONU         ENABLED       ACTIVE                    [{db87c4b48843bb99567d3d94 536870912}]

Verify ONOS device state and eventual EAPoL authentication. ONOS default
Username is ``karaf``, Password is ``karaf``

.. code:: sh

   ssh -p 8101 karaf@localhost

Display the device and ports discovered

.. code:: sh

   karaf@root > ports

   id=of:0000a82bb53678ae, available=true, local-status=connected 4m27s ago, role=MASTER, type=SWITCH, mfr=VOLTHA Project, hw=open_pon, sw=open_pon, serial=EC1721000221, chassis=a82bb53678ae, driver=voltha, channelId=172.27.0.1:59124, managementAddress=172.27.0.1, protocol=OF_13
     port=16, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:10, portName=ALPHe3d1cf57-1
     port=17, state=disabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:11, portName=ALPHe3d1cf57-2
     port=18, state=disabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:12, portName=ALPHe3d1cf57-3
     port=19, state=disabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:13, portName=ALPHe3d1cf57-4
     port=20, state=disabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:14, portName=ALPHe3d1cf57-5
     port=1048576, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=a8:2b:b5:36:78:ae, portName=nni-1048576

EAPoL may take up to 30 seconds to complete.

.. code:: sh

   karaf@root > aaa-users

   of:0000a82bb53678ae/16: AUTHORIZED_STATE, last-changed=4m22s ago, mac=94:CC:B9:DA:AB:D1, subid=PON 1/1/3/1:2.1.1, username=94:CC:B9:DA:AB:D1

**Provision subscriber flows**

.. code:: sh

   karaf@root > volt-add-subscriber-access of:0000a82bb53678ae 16

   karaf@root > volt-programmed-subscribers

   location=of:0000a82bb53678ae/16 tagInformation=UniTagInformation{uniTagMatch=0, ponCTag=20, ponSTag=11, usPonCTagPriority=-1, usPonSTagPriority=-1, dsPonCTagPriority=-1, dsPonSTagPriority=-1, technologyProfileId=64, enableMacLearning=false, upstreamBandwidthProfile='Default', downstreamBandwidthProfile='Default', serviceName='', configuredMacAddress='A4:23:05:00:00:00', isDhcpRequired=true, isIgmpRequired=false}

After about 30 seconds the RG should attempt DHCP which should be
visible in onos. At this point the RG should be able to pass database
traffic via the ONU/OLT.

.. code:: sh

   karaf@root > dhcpl2relay-allocations

   SubscriberId=ALPHe3d1cf57-1,ConnectPoint=of:0000a82bb53678ae/16,State=DHCPREQUEST,MAC=94:CC:B9:DA:AB:D1,CircuitId=PON 1/1/3/1:2.1.1,IP Allocated=29.29.206.20,Allocation Timestamp=2020-02-17T15:34:31.572746Z

BBSIM Simulated OLT/ONU Testing Control Plane Traffic
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you do not have physical OLT/ONU hardware you can start VOLTHA
containers and the bbsim olt/onu hardware simulator using a different
docker-compose yaml file. Verify containers are running as above with
the addition of the bbsim and radius server containers.

.. code:: sh

   export DOCKER_HOST_IP=172.17.0.1
   export DOCKER_TAG=latest

   cd ~/source/voltha-go
   docker-compose -f compose/system-test-bbsim.yml up -d

Create/Preprovision and enable similarly to the physical OLT above,
providing the local bbsim IP and listening port

.. code:: sh

   voltctl device create -t openolt -H 172.17.0.1:50060
   ece94c86e93c6e06dd0a544b

   voltctl device enable ece94c86e93c6e06dd0a544b
   ece94c86e93c6e06dd0a544b

Proceed with the verification and ONOS provisioning commands similar to
the physical OLT described above.