VOL-617 - ability to build without VM
Change-Id: Iec79d66b7ab98a5421eabe7819fa6325ff6690ce
diff --git a/DOCKER_BUILD.md b/DOCKER_BUILD.md
new file mode 100644
index 0000000..c277712
--- /dev/null
+++ b/DOCKER_BUILD.md
@@ -0,0 +1,132 @@
+# How to Build VOLTHA using only Docker
+
+The standard (original) build environment for VOLTHA required the installation
+of many support tools or a custom `Vagrant` VM with those same support tools
+installed on that VM.
+
+This build guide walks through a VOLTHA build on a system that only support
+Docker. If after building VOLTHA, it is desired to run VOLTHA you will need
+a version of Docker that supports Swarm Mode or `docker-compose`. It is
+recommended that you use the latest stable version of `docker`.
+
+## Building
+
+### Prerequisites
+
+* git - in order to clone the VOLTHA source, not required if you already have
+the source or can obtain it via a different mechanism.
+* make - standard build utility
+* docker - version `17.06.0-ce` or later recommended, the latest stable
+version preferred.
+
+### Git Clone source
+
+```bash
+git clone http://gerrit.opencord.org/voltha
+```
+
+### Build VOLTHA
+
+```bash
+cd voltha # if you are not alread in the voltha directory
+VOLTHA_BUILD=docker make build
+```
+
+The build can take a little while, so feel free to get a cup of coffee, go for
+a short walk, or otherwise rest your mind. A build on a clean Ubuntu VM takes
+about 30 minutes. The actual time for a build depends on available network
+bandwidth and CPU speed.
+
+## Running VOLTHA
+
+### Running VOLTHA
+
+VOLTHA runs as a Docker Swarm Stack. Thus, to run VOLTHA you should have
+initialized your Docker Swarm using
+```bash
+docker swarm init
+```
+
+After the swarm has been initialized VOLTHA can be started with
+```bash
+VOLTHA_BUILD=docker make start
+```
+
+Eventually all the VOLTHA service will be started. You can view the service
+list using
+```bash
+docker service ls
+```
+
+which should generate an output similar to
+
+```bash
+ID NAME MODE REPLICAS IMAGE PORTS
+db4sd6qr4ovd voltha_cli replicated 1/1 voltha-cli:latest *:5022->22/tcp
+f4am7jkrfkid voltha_consul global 1/1 consul:0.9.2 *:8300->8300/tcp,*:8400->8400/tcp,*:8500->8500/tcp,*:8600->8600/udp
+b0y0op65zijd voltha_fluentd replicated 1/1 voltha-fluentd:latest *:30011->24224/tcp
+qqqba5wdug8i voltha_fluentdactv replicated 1/1 voltha-fluentd:latest *:30010->24224/tcp
+aaba0xdriixw voltha_fluentdstby replicated 1/1 voltha-fluentd:latest *:30009->24224/tcp
+watbhno8ylf6 voltha_freeradius replicated 0/0 marcelmaatkamp/freeradius:latest *:1812->1812/udp,*:1813->1813/tcp,*:18120->18120/tcp
+q58ptpueojha voltha_kafka global 1/1 wurstmeister/kafka:latest *:9092->9092/tcp
+pqp9o1z0ojpz voltha_netconf global 1/1 voltha-netconf:latest *:830->1830/tcp
+sthhtxdv6trv voltha_ofagent replicated 1/1 voltha-ofagent:latest
+uk8c7f3cutpn voltha_onos replicated 1/1 voltha-onos:latest *:6653->6653/tcp,*:8101->8101/tcp,*:8181->8181/tcp
+jauyicnmzy2m voltha_onos_cluster_manager replicated 1/1 voltha-unum:latest *:5411->5411/tcp
+vnsladm0ar0b voltha_tools replicated 1/1 voltha-tools:latest *:4022->22/tcp
+on4hpyuwiyw2 voltha_vcore replicated 1/1 voltha-voltha:latest *:8880->8880/tcp,*:18880->18880/tcp,*:50556->50556/tcp
+u9g9vaip2nhf voltha_voltha replicated 1/1 voltha-envoy:latest *:8001->8001/tcp,*:8443->8443/tcp,*:8882->8882/tcp,*:50555->50555/tcp
+hyuak4pr8pt3 voltha_zk1 replicated 1/1 wurstmeister/zookeeper:latest
+hdshxxj1sxoj voltha_zk2 replicated 1/1 wurstmeister/zookeeper:latest
+y70234pasn6g voltha_zk3 replicated 1/1 wurstmeister/zookeeper:latest
+```
+
+After all the services are started you can access the VOLTHA CLI using `ssh`
+```bash
+ssh -p 5022 voltha@localhost
+```
+
+VOLTHA can be stopped with
+```bash
+VOLTHA_BUILD=docker make stop
+```
+
+### Running VOLTHA from pre-build docker images
+The VOLTHA docker images are published on `dockerhub.com` as the `voltha`
+repository: `https://hub.docker.com/u/voltha/`.
+
+To run VOLTHA using this containers (and therefore not requiring a build) the
+following command can be used
+
+```bash
+REPOSITORY=voltha/ VOLTHA_BUILD=docker make start
+```
+
+_NOTE: the slash (`/`) at the end of the `REPOSITORY` specification is
+required._
+
+Running VOLTHA in this way should produce the following `docker service ls`
+output
+
+```bash
+ID NAME MODE REPLICAS IMAGE PORTS
+86iemjy8q1e1 voltha_cli replicated 1/1 voltha/voltha-cli:latest *:5022->22/tcp
+sm0zuqcq41go voltha_consul global 1/1 consul:0.9.2 *:8300->8300/tcp,*:8400->8400/tcp,*:8500->8500/tcp,*:8600->8600/udp
+vx5ir7dsciq3 voltha_fluentd replicated 1/1 voltha/voltha-fluentd:latest *:30014->24224/tcp
+x1ptzxq37cjw voltha_fluentdactv replicated 1/1 voltha/voltha-fluentd:latest *:30012->24224/tcp
+wfu6ebh3id6a voltha_fluentdstby replicated 1/1 voltha/voltha-fluentd:latest *:30013->24224/tcp
+h4r0z661t2u9 voltha_freeradius replicated 0/0 marcelmaatkamp/freeradius:latest *:1812->1812/udp,*:1813->1813/tcp,*:18120->18120/tcp
+hzhqj0rvjsh8 voltha_kafka global 1/1 wurstmeister/kafka:latest *:9092->9092/tcp
+vzewlgoxb3j6 voltha_netconf global 1/1 voltha/voltha-netconf:latest *:830->1830/tcp
+v1uj00lyzgj8 voltha_ofagent replicated 1/1 voltha/voltha-ofagent:latest
+bafqv7fvb1qb voltha_onos replicated 1/1 voltha/voltha-onos:latest *:6653->6653/tcp,*:8101->8101/tcp,*:8181->8181/tcp
+umams0s8jq6h voltha_onos_cluster_manager replicated 1/1 voltha/voltha-unum:latest *:5411->5411/tcp
+tnn5ce8x4k89 voltha_tools replicated 1/1 voltha/voltha-tools:latest *:4022->22/tcp
+h4c94dvhx0ig voltha_vcore replicated 1/1 voltha/voltha-voltha:latest *:8880->8880/tcp,*:18880->18880/tcp,*:50556->50556/tcp
+9l5ubtie7lt4 voltha_voltha replicated 1/1 voltha/voltha-envoy:latest *:8001->8001/tcp,*:8443->8443/tcp,*:8882->8882/tcp,*:50555->50555/tcp
+k43f3a1wa0hv voltha_zk1 replicated 1/1 wurstmeister/zookeeper:latest
+kl5lpi0mt35e voltha_zk2 replicated 1/1 wurstmeister/zookeeper:latest
+t9eh5whkivfe voltha_zk3 replicated 1/1 wurstmeister/zookeeper:latest
+```
+
+_Notice the image names in this output are prefixed with `voltha/`_
diff --git a/Jenkinsfile b/Jenkinsfile
index 05f7bb6..b4acd75 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -17,10 +17,7 @@
sh 'vagrant ssh -c "cp /cord/incubator/voltha/voltha/voltha.jenkins.yml /cord/incubator/voltha/voltha/voltha.yml"'
stage 'Build voltha'
- sh 'vagrant ssh -c "cd /cord/incubator/voltha && source env.sh && make fetch-jenkins && make jenkins" voltha'
-
- stage 'Bring up voltha containers'
- sh 'vagrant ssh -c "cd /cord/incubator/voltha && source env.sh && docker-compose -f compose/docker-compose-docutests.yml up -d" voltha'
+ sh 'vagrant ssh -c "cd /cord/incubator/voltha && source env.sh && make fetch && make build" voltha'
stage 'Run Integration Tests'
sh 'vagrant ssh -c "cd /cord/incubator/voltha && source env.sh && make jenkins-test" voltha'
diff --git a/Makefile b/Makefile
index 92367ef..1cba5c3 100644
--- a/Makefile
+++ b/Makefile
@@ -14,9 +14,11 @@
# limitations under the License.
#
+ifneq ($(VOLTHA_BUILD),docker)
ifeq ($(VOLTHA_BASE)_set,_set)
$(error To get started, please source the env.sh file)
endif
+endif
ifeq ($(TAG),)
TAG := latest
@@ -26,6 +28,12 @@
TARGET_TAG := latest
endif
+# If no DOCKER_HOST_IP is specified grab a v4 IP address associated with
+# the default gateway
+ifeq ($(DOCKER_HOST_IP),)
+DOCKER_HOST_IP := $(shell ifconfig $$(netstat -rn | grep -E '^(default|0.0.0.0)' | head -1 | awk '{print $$NF}') | grep inet | awk '{print $$2}' | sed -e 's/addr://g')
+endif
+
include setup.mk
ifneq ($(http_proxy)$(https_proxy),)
@@ -40,7 +48,9 @@
--build-arg FTP_PROXY=$(FTP_PROXY) \
--build-arg NO_PROXY=$(NO_PROXY)
endif
-DOCKER_BUILD_ARGS = --build-arg TAG=$(TAG) \
+
+DOCKER_BUILD_ARGS = \
+ --build-arg TAG=$(TAG) \
--build-arg REGISTRY=$(REGISTRY) \
--build-arg REPOSITORY=$(REPOSITORY) \
$(DOCKER_PROXY_ARGS) $(DOCKER_CACHE_ARG) \
@@ -51,6 +61,8 @@
DOCKER_IMAGE_LIST = \
base \
+ protoc \
+ protos \
voltha \
ofagent \
tools \
@@ -69,9 +81,61 @@
unum \
tester \
config-push \
- j2
+ j2 \
+ test_runner
-.PHONY: $(DIRS) $(DIRS_CLEAN) $(DIRS_FLAKE8) flake8 docker-base voltha ofagent netconf shovel onos dashd cli portainer grafana nginx consul envoy golang envoyd tools opennms logstash unum start stop tag push pull
+# The following list was scavanged from the compose / stack files as well as
+# from the Dockerfiles. If nothing else it highlights that VOLTHA is not
+# using consistent versions for some of the containers.
+
+# grep -i "^FROM" docker/Dockerfile.* | grep -v voltha- | sed -e 's/ as .*$//g' -e 's/\${REGISTRY}//g' | awk '{print $NF}' | grep -v '^scratch' | sed '/:.*$/!s/$/:latest/g' | sort -u | sed -e 's/^/ /g' -e 's/$/ \\/g'
+FETCH_BUILD_IMAGE_LIST = \
+ alpine:3.6 \
+ centos:7 \
+ centurylink/ca-certs:latest \
+ consul:0.9.2 \
+ debian:stretch-slim \
+ docker.elastic.co/logstash/logstash:5.6.0 \
+ fluent/fluentd:v0.12.42 \
+ gliderlabs/registrator:v7 \
+ golang:1.9.2 \
+ grpc/python:latest \
+ kamon/grafana_graphite:3.0 \
+ lyft/envoy:29361deae91575a1d46c7a21e913f19e75622ebe \
+ maven:3-jdk-8-alpine \
+ onosproject/onos:1.10.9 \
+ opennms/horizon-core-web:19.0.1-1 \
+ portainer/portainer:1.15.2 \
+ ubuntu:xenial
+
+# find compose -type f | xargs grep image: | awk '{print $NF}' | grep -v voltha- | sed -e 's/\"//g' -e 's/\${REGISTRY}//g' -e 's/:\${.*:-/:/g' -e 's/\}//g' -e '/:.*$/!s/$/:latest/g' | sort -u | sed -e 's/^/ /g' -e 's/$/ \\/g'
+FETCH_COMPOSE_IMAGE_LIST = \
+ consul:0.9.2 \
+ docker.elastic.co/elasticsearch/elasticsearch:5.6.0 \
+ fluent/fluentd:latest \
+ fluent/fluentd:v0.12.42 \
+ gliderlabs/registrator:latest \
+ kamon/grafana_graphite:latest \
+ marcelmaatkamp/freeradius:latest \
+ postgres:9.6.1 \
+ quay.io/coreos/etcd:v3.2.9 \
+ registry:2 \
+ tianon/true:latest \
+ wurstmeister/kafka:latest \
+ wurstmeister/zookeeper:latest
+
+# find k8s -type f | xargs grep image: | awk '{print $NF}' | sed -e 's/\"//g' | sed '/:.*$/!s/$/:latest/g' | sort -u | sed -e 's/^/ /g' -e 's/$/ \\/g'
+# Manually remove some image from this list as they don't reflect the new
+# naming conventions for the VOLTHA build
+FETCH_K8S_IMAGE_LIST = \
+ consul:0.9.2 \
+ quay.io/coreos/etcd-operator:v0.7.2 \
+ wurstmeister/kafka:1.0.0 \
+ zookeeper:3.4.11
+
+FETCH_IMAGE_LIST = $(shell echo $(FETCH_BUILD_IMAGE_LIST) $(FETCH_COMPOSE_IMAGE_LIST) $(FETCH_K8S_IMAGE_LIST) | tr ' ' '\n' | sort -u)
+
+.PHONY: $(DIRS) $(DIRS_CLEAN) $(DIRS_FLAKE8) flake8 base voltha ofagent netconf shovel onos dashd cli portainer grafana nginx consul envoy go-builder envoyd tools opennms logstash unum start stop tag push pull
# This should to be the first and default target in this Makefile
help:
@@ -86,13 +150,14 @@
@echo "fetch : Pre-fetch artifacts for subsequent local builds"
@echo "flake8 : Run specifically flake8 tests"
@echo "help : Print this help"
+ @echo "protoc : Build a container with protoc installed"
@echo "protos : Compile all grpc/protobuf files"
@echo "rebuild-venv : Rebuild local Python virtualenv from scratch"
@echo "venv : Build local Python virtualenv if did not exist yet"
@echo "utest : Run all unit tests"
@echo "itest : Run all integration tests"
@echo "containers : Build all the docker containers"
- @echo "docker-base : Build the base docker container used by all other dockers"
+ @echo "base : Build the base docker container used by all other dockers"
@echo "voltha : Build the voltha docker container"
@echo "ofagent : Build the ofagent docker container"
@echo "netconf : Build the netconf docker container"
@@ -106,6 +171,7 @@
@echo "consul : Build the consul docker container"
@echo "unum : Build the unum docker container"
@echo "j2 : Build the Jinja2 template container"
+ @echo "test_runner : Build a container from which tests are run"
@echo "start : Start VOLTHA on the current system"
@echo "stop : Stop VOLTHA on the current system"
@echo "tag : Tag a set of images"
@@ -143,29 +209,38 @@
@echo " FLAKE8 $(basename $@)"
-$(Q)$(MAKE) -C $(basename $@) flake8
-build: protos containers
+build: protoc protos go-builder containers
-production: protos prod-containers
+production: protoc protos go-builder prod-containers
-jenkins : protos jenkins-containers
+jenkins: build
-jenkins-containers: docker-base voltha ofagent netconf consul unum j2
+jenkins-containers: base voltha ofagent netconf consul cli envoy fluentd unum j2
-prod-containers: docker-base voltha ofagent netconf shovel dashd cli grafana consul tools golang envoyd envoy fluentd unum j2
+prod-containers: base voltha ofagent netconf shovel onos dashd cli grafana consul tools envoy fluentd unum j2
-containers: docker-base voltha ofagent netconf shovel onos tester config-push dashd cli portainer grafana nginx consul tools golang envoyd envoy fluentd unum j2
+containers: base voltha ofagent netconf shovel onos tester config-push dashd cli portainer grafana nginx consul tools envoy fluentd unum j2 test_runner
-docker-base:
+base:
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-base:${TAG} -f docker/Dockerfile.base .
+ifneq ($(VOLTHA_BUILD),docker)
voltha: voltha-adapters
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-voltha:${TAG} -f docker/Dockerfile.voltha .
+else
+voltha:
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-voltha:${TAG} -f docker/Dockerfile.voltha_d .
+endif
voltha-adapters:
make -C voltha/adapters/asfvolt16_olt
ofagent:
+ifneq ($(VOLTHA_BUILD),docker)
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-ofagent:${TAG} -f docker/Dockerfile.ofagent .
+else
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-ofagent:${TAG} -f docker/Dockerfile.ofagent_d .
+endif
tools:
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-tools:${TAG} -f docker/Dockerfile.tools .
@@ -173,18 +248,28 @@
fluentd:
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-fluentd:${TAG} -f docker/Dockerfile.fluentd .
-envoy:
+envoy: envoyd
+ifneq ($(VOLTHA_BUILD),docker)
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-envoy:${TAG} -f docker/Dockerfile.envoy .
+else
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-envoy:${TAG} -f docker/Dockerfile.envoy_d .
+endif
envoyd:
+ifneq ($(VOLTHA_BUILD),docker)
make -C envoy
make -C envoy/go/envoyd
+endif
-golang:
+go-builder:
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-go-builder:${TAG} -f envoy/go/golang-builder/Dockerfile ./envoy/go/golang-builder
netconf:
+ifneq ($(VOLTHA_BUILD),docker)
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-netconf:${TAG} -f docker/Dockerfile.netconf .
+else
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-netconf:${TAG} -f docker/Dockerfile.netconf_d .
+endif
netopeer:
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-netopeer:${TAG} -f docker/Dockerfile.netopeer .
@@ -196,10 +281,18 @@
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-dashd:${TAG} -f docker/Dockerfile.dashd .
cli:
+ifneq ($(VOLTHA_BUILD),docker)
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-cli:${TAG} -f docker/Dockerfile.cli .
+else
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-cli:${TAG} -f docker/Dockerfile.cli_d .
+endif
portainer:
+ifneq ($(VOLTHA_BUILD),docker)
REGISTRY=${REGISTRY} REPOSITORY=${REPOSITORY} TAG=${TAG} portainer/buildPortainer.sh
+else
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-portainer:${TAG} -f docker/Dockerfile.portainer_d .
+endif
nginx:
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-nginx:${TAG} -f docker/Dockerfile.nginx .
@@ -231,6 +324,11 @@
j2:
docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-j2:${TAG} -f docker/Dockerfile.j2 docker
+test_runner:
+ifeq ($(VOLTHA_BUILD),docker)
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-test_runner:${TAG} -f docker/Dockerfile.test_runner .
+endif
+
start:
bash -c 'echo $$VOLTHA_LOGS && TMP_STACK_FILE=$$(mktemp -u) && \
echo $$TMP_STACK_FILE && \
@@ -257,10 +355,19 @@
%.pull:
docker pull ${REGISTRY}${REPOSITORY}voltha-$(subst .pull,,$@):${TAG}
+protoc:
+ifeq ($(VOLTHA_BUILD),docker)
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-protoc:${TAG} -f docker/Dockerfile.protoc .
+endif
+
protos:
+ifneq ($(VOLTHA_BUILD),docker)
make -C voltha/protos
make -C ofagent/protos
make -C netconf/protos
+else
+ docker build $(DOCKER_BUILD_ARGS) -t ${REGISTRY}${REPOSITORY}voltha-protos:${TAG} -f docker/Dockerfile.protos .
+endif
install-protoc:
make -C voltha/protos install-protoc
@@ -271,30 +378,24 @@
distclean: clean
rm -rf ${VENVDIR}
-
-fetch-jenkins:
- docker pull consul:0.9.2
- docker pull fluent/fluentd:v0.12.42
- docker pull ubuntu:xenial
- docker pull wurstmeister/kafka:1.0.0
- docker pull zookeeper:3.4.11
fetch:
- docker pull consul:0.9.2
- docker pull fluent/fluentd:v0.12.42
- docker pull ubuntu:xenial
- docker pull wurstmeister/kafka:1.0.0
- docker pull zookeeper:3.4.11
- docker pull portainer/portainer:1.15.2
- docker pull lyft/envoy:29361deae91575a1d46c7a21e913f19e75622ebe
- docker pull registry:2
- docker pull kamon/grafana_graphite:3.0
+ @bash -c ' \
+ for i in $(FETCH_IMAGE_LIST); do \
+ docker pull $$i; \
+ done'
+
+fetch-jenkins: fetch
purge-venv:
rm -fr ${VENVDIR}
rebuild-venv: purge-venv venv
+ifneq ($(VOLTHA_BUILD),docker)
venv: ${VENVDIR}/.built
+else
+venv:
+endif
${VENVDIR}/.built:
@ virtualenv ${VENVDIR}
@@ -308,48 +409,172 @@
uname -s > ${VENVDIR}/.built; \
fi
+ifneq ($(VOLTHA_BUILD),docker)
test: venv protos run-as-root-tests
@ echo "Executing all tests"
. ${VENVDIR}/bin/activate && \
nosetests -s tests \
--exclude-dir=./tests/itests/run_as_root/
+else
+test: protos test_runner run-as-root-tests
+ docker run \
+ -e VOLTHA_BUILD=docker \
+ -e REGISTRY=${REGISTRY} \
+ -e REPOSITORY=${REPOSITORY} \
+ -e TAG=${TAG} \
+ -e DOCKER_HOST_IP=${DOCKER_HOST_IP} \
+ --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock \
+ ${REGISTRY}${REPSOITORY}voltha-test_runner:${TAG} \
+ nosetests -s tests --exclude-dir=./tests/itests/run_as_root/
+endif
+ifneq ($(VOLTHA_BUILD),docker)
utest: venv protos
@ echo "Executing all unit tests"
. ${VENVDIR}/bin/activate && \
for d in $$(find ./tests/utests -type d|sort -nr); do echo $$d:; nosetests $$d; done
+else
+utest: protos test_runner
+ docker run \
+ -e VOLTHA_BUILD=docker \
+ -e REGISTRY=${REGISTRY} \
+ -e REPOSITORY=${REPOSITORY} \
+ -e TAG=${TAG} \
+ -e DOCKER_HOST_IP=${DOCKER_HOST_IP} \
+ --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock \
+ ${REGISTRY}${REPSOITORY}voltha-test_runner:${TAG} \
+ bash -c \
+ 'for d in $$(find ./tests/utests -type d|sort -nr); do \
+ echo $$d:; \
+ nosetests $$d; \
+ done'
+endif
+ifneq ($(VOLTHA_BUILD),docker)
utest-with-coverage: venv protos
@ echo "Executing all unit tests and producing coverage results"
. ${VENVDIR}/bin/activate && \
for d in $$(find ./tests/utests -type d|sort -nr); do echo $$d:; \
nosetests --with-xcoverage --with-xunit --cover-package=voltha,common,ofagent $$d; done
+else
+utest-with-coverage: protos test_runner
+ @echo "Executing all unit tests and producing coverage results"
+ docker run \
+ -e VOLTHA_BUILD=docker \
+ -e REGISTRY=${REGISTRY} \
+ -e REPOSITORY=${REPOSITORY} \
+ -e TAG=${TAG} \
+ -e DOCKER_HOST_IP=${DOCKER_HOST_IP} \
+ --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock \
+ ${REGISTRY}${REPSOITORY}voltha-test_runner:${TAG} \
+ bash -c \
+ 'for d in $$(find ./tests/utests -type d|sort -nr); do \
+ echo $$d:; \
+ nosetests --with-xcoverage --with-xunit --cover-package=voltha,common,ofagent $$d; \
+ done'
+endif
+ifneq ($(VOLTHA_BUILD),docker)
itest: venv run-as-root-tests
@ echo "Executing all integration tests"
. ${VENVDIR}/bin/activate && \
+ rm -rf /tmp/fluentd/* && \
+ REGISTRY=${REGISTRY} \
+ REPOSITORY=${REPOSITORY} \
+ TAG=${TAG} \
+ DOCKER_HOST_IP=${DOCKER_HOST_IP} \
nosetests -s \
- tests/itests/docutests/build_md_test.py \
- --exclude-dir=./tests/utests/ \
- --exclude-dir=./tests/itests/run_as_root/
+ tests/itests/docutests/build_md_test.py \
+ --exclude-dir=./tests/utests/ \
+ --exclude-dir=./tests/itests/run_as_root/
+else
+itest: protos test_runner
+ @ echo "Executing all integration tests"
+ docker run \
+ -e VOLTHA_BUILD=docker \
+ -e REGISTRY=${REGISTRY} \
+ -e REPOSITORY=${REPOSITORY} \
+ -e TAG=${TAG} \
+ -e DOCKER_HOST_IP=${DOCKER_HOST_IP} \
+ --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock \
+ ${REGISTRY}${REPSOITORY}voltha-test_runner:${TAG} \
+ nosetests -s \
+ tests/itests/docutests/build_md_test.py \
+ --exclude-dir=./tests/utests/ \
+ --exclude-dir=./tests/itests/run_as_root/
+endif
+ifneq ($(VOLTHA_BUILD),docker)
smoke-test: venv run-as-root-tests
@ echo "Executing smoke tests"
. ${VENVDIR}/bin/activate && \
+ rm -rf /tmp/fluentd/* && \
+ REGISTRY=${REGISTRY} \
+ REPOSITORY=${REPOSITORY} \
+ TAG=${TAG} \
+ DOCKER_HOST_IP=${DOCKER_HOST_IP} \
nosetests -s \
tests/itests/docutests/build_md_test.py:BuildMdTests.test_07_start_all_containers \
--exclude-dir=./tests/itests/run_as_root/
+else
+smoke-test: protos test_runner run-as-root-tests
+ @ echo "Executing smoke tests"
+ docker run \
+ -e VOLTHA_BUILD=docker \
+ -e REGISTRY=${REGISTRY} \
+ -e REPOSITORY=${REPOSITORY} \
+ -e TAG=${TAG} \
+ -e DOCKER_HOST_IP=${DOCKER_HOST_IP} \
+ --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock \
+ ${REGISTRY}${REPSOITORY}voltha-test_runner:${TAG} \
+ nosetests -s \
+ tests/itests/docutests/build_md_test.py:BuildMdTests.test_07_start_all_containers \
+ --exclude-dir=./tests/itests/run_as_root/
+endif
+ifneq ($(VOLTHA_BUILD),docker)
jenkins-test: venv
@ echo "Executing jenkins smoke tests"
. ${VENVDIR}/bin/activate && \
+ rm -rf /tmp/fluentd/* && \
+ REGISTRY=${REGISTRY} \
+ REPOSITORY=${REPOSITORY} \
+ TAG=${TAG} \
+ DOCKER_HOST_IP=${DOCKER_HOST_IP} \
nosetests -s \
- tests/itests/docutests/build_md_test.py:BuildMdTests.test_07_start_all_containers \
- --exclude-dir=./tests/itests/run_as_root/
+ tests/itests/docutests/build_md_test.py:BuildMdTests.test_07_start_all_containers \
+ --exclude-dir=./tests/itests/run_as_root/
+else
+jenkins-test: protos test_runner
+ @ echo "Executing jenkins smoke tests"
+ @ echo "Starting VOLTHA as docker-compose services"
+ docker run \
+ -e REGISTRY=${REGISTRY} \
+ -e REPOSITORY=${REPOSITORY} \
+ -e TAG=${TAG} \
+ -e DOCKER_HOST_IP=${DOCKER_HOST_IP} \
+ --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock \
+ ${REGISTRY}${REPSOITORY}voltha-test_runner:${TAG} \
+ nosetests -s \
+ tests/itests/docutests/build_md_test.py:BuildMdTests.test_07_start_all_containers \
+ --exclude-dir=./tests/itests/run_as_root/
+endif
-
+ifneq ($(VOLTHA_BUILD),docker)
run-as-root-tests:
docker run -i --rm -v /cord/incubator/voltha:/voltha --privileged ${REGISTRY}${REPOSITORY}voltha-base:${TAG} env PYTHONPATH=/voltha python /voltha/tests/itests/run_as_root/test_frameio.py
+else
+run-as-root-tests:
+ docker run \
+ -e VOLTHA_BUILD=docker \
+ -e REGISTRY=${REGISTRY} \
+ -e REPOSITORY=${REPOSITORY} \
+ -e TAG=${TAG} \
+ -e DOCKER_HOST_IP=${DOCKER_HOST_IP} \
+ --rm --privileged \
+ ${REGISTRY}${REPOSITORY}voltha-test_runner:${TAG} \
+ env PYTHONPATH=/work python tests/itests/run_as_root/test_frameio.py
+endif
flake8: $(DIRS_FLAKE8)
diff --git a/Vagrantfile b/Vagrantfile
index ab1104b..f0941fc 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -73,8 +73,12 @@
d.vm.hostname = "#{settings['server_name']}"
d.vm.network "private_network", ip: "10.100.198.220"
d.vm.provision :shell, path: "ansible/scripts/bootstrap_ansible.sh"
- d.vm.provision :shell, inline: "PYTHONUNBUFFERED=1 ansible-playbook /cord/incubator/voltha/ansible/voltha.yml -c local"
- d.vm.provision :shell, inline: "cd /cord/incubator/voltha && source env.sh && make install-protoc && chmod 777 /tmp/fluentd"
+ if "docker" == "#{settings['build_mode']}"
+ d.vm.provision :shell, inline: "PYTHONUNBUFFERED=1 ansible-playbook /vagrant/ansible/voltha-docker.yml -c local"
+ else
+ d.vm.provision :shell, inline: "PYTHONUNBUFFERED=1 ansible-playbook /cord/incubator/voltha/ansible/voltha.yml -c local"
+ d.vm.provision :shell, inline: "cd /cord/incubator/voltha && source env.sh && make install-protoc && chmod 777 /tmp/fluentd"
+ end
d.vm.provider Provider do |v|
v.memory = 6144
v.cpus = 4
diff --git a/ansible/roles/common/defaults/main.yml b/ansible/roles/common/defaults/main.yml
index 7be66d2..be41648 100644
--- a/ansible/roles/common/defaults/main.yml
+++ b/ansible/roles/common/defaults/main.yml
@@ -5,19 +5,7 @@
use_latest_for:
- debian-keyring
- debian-archive-keyring
- - python-dev
- - kafkacat
- - libssl-dev
- - libffi-dev
- - libpcap-dev
- - libxml2-dev
- - libxslt1-dev
- - python-virtualenv
- jq
- - python-nose
- - python-flake8
- - python-scapy
-# - python-libpcap
obsolete_services:
- puppet
diff --git a/ansible/roles/voltha-dev/defaults/main.yml b/ansible/roles/voltha-dev/defaults/main.yml
new file mode 100644
index 0000000..36b247c
--- /dev/null
+++ b/ansible/roles/voltha-dev/defaults/main.yml
@@ -0,0 +1,17 @@
+use_latest_for:
+ - python-dev
+ - kafkacat
+ - libssl-dev
+ - libffi-dev
+ - libpcap-dev
+ - libxml2-dev
+ - libxslt1-dev
+ - python-virtualenv
+ - python-nose
+ - python-flake8
+ - python-scapy
+# - python-libpcap
+
+obsolete_services:
+ - puppet
+ - chef-client
diff --git a/ansible/roles/voltha-dev/files/ssh_config b/ansible/roles/voltha-dev/files/ssh_config
new file mode 100644
index 0000000..990a43d
--- /dev/null
+++ b/ansible/roles/voltha-dev/files/ssh_config
@@ -0,0 +1,3 @@
+Host *
+ StrictHostKeyChecking no
+ UserKnownHostsFile=/dev/null
diff --git a/ansible/roles/voltha-dev/tasks/main.yml b/ansible/roles/voltha-dev/tasks/main.yml
new file mode 100644
index 0000000..de23ddd
--- /dev/null
+++ b/ansible/roles/voltha-dev/tasks/main.yml
@@ -0,0 +1,13 @@
+- name: Latest apt packages
+ apt:
+ name: "{{ item }}"
+ with_items: "{{ use_latest_for }}"
+ tags: [voltha-dev]
+
+- name: Services are not running
+ service:
+ name: "{{ item }}"
+ state: stopped
+ ignore_errors: yes
+ with_items: "{{ obsolete_services }}"
+ tags: [voltha-dev]
diff --git a/ansible/voltha-docker.yml b/ansible/voltha-docker.yml
new file mode 100644
index 0000000..f516bb4
--- /dev/null
+++ b/ansible/voltha-docker.yml
@@ -0,0 +1,7 @@
+- hosts: localhost
+ remote_user: vagrant
+ serial: 1
+ roles:
+ - common
+ - docker
+ - docker-compose
diff --git a/ansible/voltha.yml b/ansible/voltha.yml
index a92fc7c..6b97222 100644
--- a/ansible/voltha.yml
+++ b/ansible/voltha.yml
@@ -3,6 +3,7 @@
serial: 1
roles:
- common
+ - voltha-dev
- docker
- docker-compose
- java
diff --git a/compose/docker-compose-auth-test.yml b/compose/docker-compose-auth-test.yml
index c79babc..7f5b442 100644
--- a/compose/docker-compose-auth-test.yml
+++ b/compose/docker-compose-auth-test.yml
@@ -28,7 +28,7 @@
ipv4_address: 172.25.0.200
# onos_config_push:
-# image: "${REGISTRY}${REPOSITORY}onos-config-push:${TAG:-latest}"
+# image: "${REGISTRY}${REPOSITORY}voltha-config-push:${TAG:-latest}"
# environment:
# ONOS_CONNECTION: 'onos:8181'
# volumes:
diff --git a/docker/Dockerfile.cli_d b/docker/Dockerfile.cli_d
new file mode 100755
index 0000000..80db472
--- /dev/null
+++ b/docker/Dockerfile.cli_d
@@ -0,0 +1,53 @@
+# Copyright 2016 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+ARG TAG=latest
+ARG REGISTRY=
+ARG REPOSITORY=
+
+FROM ${REGISTRY}${REPOSITORY}voltha-protos:${TAG} as protos
+FROM ${REGISTRY}${REPOSITORY}voltha-base:${TAG}
+
+MAINTAINER Sergio Slobodrian <sslobodr@ciena.com>
+
+# Bundle app source
+RUN mkdir /cli && touch /cli/__init__.py
+ENV PYTHONPATH=/cli
+COPY common /cli/common
+COPY cli /cli/cli
+COPY voltha /cli/voltha
+COPY --from=protos /protos/voltha /cli/voltha/protos
+COPY --from=protos /protos/google/api /cli/voltha/protos/third_party/google/api
+COPY --from=protos /protos/asfvolt16_olt /cli/voltha/adapters/asfvolt16_olt/protos
+RUN useradd -b /home -d /home/voltha voltha -s /bin/bash
+RUN mkdir /home/voltha
+RUN chown voltha.voltha /home/voltha
+RUN echo "voltha:admin" | chpasswd
+RUN apt-get update && apt-get install -y openssh-server
+RUN apt-get update && apt-get install -y openssh-server
+RUN mkdir /var/run/sshd
+RUN echo 'root:screencast' | chpasswd
+RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
+
+# SSH login fix. Otherwise user is kicked off after login
+RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
+
+ENV NOTVISIBLE "in users profile"
+RUN echo "export VISIBLE=now" >> /etc/profile
+
+EXPOSE 22
+
+# Exposing process and default entry point
+ENTRYPOINT ["/usr/bin/dumb-init", "--"]
+
+CMD ["/cli/cli/setup.sh"]
diff --git a/docker/Dockerfile.envoy_d b/docker/Dockerfile.envoy_d
new file mode 100644
index 0000000..57843bf
--- /dev/null
+++ b/docker/Dockerfile.envoy_d
@@ -0,0 +1,43 @@
+# Copyright 2018 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+ARG REGISTRY=
+ARG REPOSITORY=
+ARG TAG=latest
+ARG HTTP_PROXY=
+ARG HTTPS_PROXY=
+
+FROM grpc/python as protos
+COPY voltha/protos/*.proto /voltha/protos/
+COPY voltha/protos/third_party/google/api/*.proto /voltha/protos/third_party/google/api/
+#RUN protoc -I/voltha/protos -I/voltha/protos/third_party --include_imports --include_source_info --descriptor_set_out=/proto.pb /voltha/protos/*.proto
+RUN python -m grpc.tools.protoc -I/voltha/protos -I/voltha/protos/third_party --include_imports --include_source_info --descriptor_set_out=/proto.pb /voltha/protos/*.proto
+
+FROM ${REGISTRY}${REPOSITORY}voltha-go-builder:${TAG} as build
+ENV http_proxy ${HTTP_PROXY}
+ENV https_proxy ${HTTPS_PROXY}
+
+COPY envoy/go/envoyd/*.go /src/
+RUN mkdir /output
+RUN OUTPUT=/ /build.sh
+RUN ls /output
+
+FROM lyft/envoy:29361deae91575a1d46c7a21e913f19e75622ebe
+
+RUN apt-get update && apt-get -q install -y curl
+COPY envoy/front-proxy /envoy
+COPY --from=protos /proto.pb /envoy/
+COPY pki /envoy/
+COPY --from=build /envoyd /usr/local/bin/envoyd
+
+CMD /usr/local/bin/envoy -c /envoy/front-proxy/voltha-grpc-proxy.json
diff --git a/docker/Dockerfile.netconf_d b/docker/Dockerfile.netconf_d
new file mode 100644
index 0000000..56e1c7a
--- /dev/null
+++ b/docker/Dockerfile.netconf_d
@@ -0,0 +1,45 @@
+# Copyright 2018 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+ARG TAG=latest
+ARG REGISTRY=
+ARG REPOSITORY=
+FROM ${REGISTRY}${REPOSITORY}voltha-protos:${TAG} as protos
+FROM ${REGISTRY}${REPOSITORY}voltha-base:${TAG}
+
+MAINTAINER Voltha Community <info@opennetworking.org>
+
+# Install protoc version 3.0.0; this is not yet the supported
+# version on xenial, so we need to "backport" it
+RUN apt-get update && \
+ apt-get install -y zlib1g-dev wget && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotoc10_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotobuf-lite10_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotobuf-dev_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotobuf10_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/protobuf-compiler_3.0.0-9_amd64.deb && \
+ dpkg -i *.deb && \
+ protoc --version && \
+ rm -f *.deb
+
+# Bundle app source
+RUN mkdir /netconf && touch /netconf/__init__.py
+ENV PYTHONPATH=/netconf
+COPY common /netconf/common
+COPY netconf /netconf/netconf
+COPY --from=protos /protos/voltha /netconf/netconf/protos
+COPY --from=protos /protos/google/api /netconf/netconf/protos/third_party/google/api
+COPY --from=protos /protos/voltha/yang_options* /netconf/netconf/protoc_plugins/
+
+# Exposing process and default entry point
+CMD ["python", "netconf/netconf/main.py"]
diff --git a/docker/Dockerfile.ofagent_d b/docker/Dockerfile.ofagent_d
new file mode 100644
index 0000000..6c9fa86
--- /dev/null
+++ b/docker/Dockerfile.ofagent_d
@@ -0,0 +1,53 @@
+# Copyright 2016 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+ARG TAG=latest
+ARG REGISTRY=
+ARG REPOSITORY=
+
+FROM ${REGISTRY}${REPOSITORY}voltha-protos:${TAG} as protos
+FROM ${REGISTRY}${REPOSITORY}voltha-base:${TAG}
+
+MAINTAINER Zsolt Haraszti <zharaszt@ciena.com>
+MAINTAINER Ali Al-Shabibi <ali.al-shabibi@onlab.us>
+MAINTAINER Nathan Knuth <nathan.knuth@tibitcom.com>
+
+# Install protoc version 3.0.0; this is not yet the supported
+# version on xenial, so we need to "backport" it
+RUN apt-get update && \
+ apt-get install -y zlib1g-dev wget && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotoc10_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotobuf-lite10_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotobuf-dev_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/libprotobuf10_3.0.0-9_amd64.deb && \
+ wget http://ftp.us.debian.org/debian/pool/main/p/protobuf/protobuf-compiler_3.0.0-9_amd64.deb && \
+ dpkg -i *.deb && \
+ protoc --version && \
+ rm -f *.deb
+
+# Bundle app source
+RUN mkdir -p /ofagent/ofagent/protos/third_party/google/api && \
+ touch /ofagent/__init__.py \
+ /ofagent/ofagent/protos/third_party/google/__init__.py \
+ /ofagent/ofagent/protos/third_party/google/api/__init__.py
+ENV PYTHONPATH=/ofagent
+COPY common /ofagent/common
+COPY ofagent /ofagent/ofagent
+COPY pki /ofagent/pki
+COPY --from=protos /protos/voltha /ofagent/ofagent/protos
+COPY --from=protos /protos/google/api /ofagent/ofagent/protos/third_party/google/api
+
+ENTRYPOINT ["/usr/bin/dumb-init", "--"]
+
+# Exposing process and default entry point
+CMD ["dumb-init", "python", "ofagent/ofagent/main.py"]
diff --git a/docker/Dockerfile.portainer_d b/docker/Dockerfile.portainer_d
new file mode 100644
index 0000000..ec69bfa
--- /dev/null
+++ b/docker/Dockerfile.portainer_d
@@ -0,0 +1,49 @@
+# Copyright 2018 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM ${REGISTRY}portainer/portainer:1.15.2 as base
+
+FROM ${REGISTRY}alpine:3.6 as work
+COPY --from=base / /work
+RUN find /work -print
+
+RUN sed -i \
+-e 's~constant("DOCKER_ENDPOINT","api/docker")~constant("DOCKER_ENDPOINT","docker/api/docker")~' \
+-e 's~constant("CONFIG_ENDPOINT","api/settings")~constant("CONFIG_ENDPOINT","docker/api/settings")~' \
+-e 's~constant("AUTH_ENDPOINT","api/auth")~constant("AUTH_ENDPOINT","docker/api/auth")~' \
+-e 's~constant("USERS_ENDPOINT","api/users")~constant("USERS_ENDPOINT","docker/api/users")~' \
+-e 's~constant("ENDPOINTS_ENDPOINT","api/endpoints")~constant("ENDPOINTS_ENDPOINT","docker/api/endpoints")~' \
+-e 's~constant("TEMPLATES_ENDPOINT","api/templates")~constant("TEMPLATES_ENDPOINT","docker/api/templates")~' \
+/work/public/js/app.*.js
+
+RUN sed -i \
+-e 's~href="~href="docker/~' \
+-e 's~href='\''~href='\''docker/~' \
+-e 's~src="~src="docker/~' \
+-e 's~src='\''~src='\''docker/~' \
+-e 's~"images/logo.png"~"docker/images/logo.png"~' \
+/work/public/index.html
+
+FROM centurylink/ca-certs
+MAINTAINER Voltha Community <info@opennetworking.org>
+
+COPY --from=work /work /
+
+VOLUME /data
+
+WORKDIR /
+
+EXPOSE 9000
+
+ENTRYPOINT ["/portainer"]
diff --git a/docker/Dockerfile.protoc b/docker/Dockerfile.protoc
new file mode 100644
index 0000000..f9cd10f
--- /dev/null
+++ b/docker/Dockerfile.protoc
@@ -0,0 +1,39 @@
+# Copyright 2018 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG REGISTRY=
+ARG PROTOC_PREFIX=/usr/local
+ARG ROTOC_LIBDIR=${PROTOC_PREFIX}/lib
+ARG PROTOC=${PROTOC_PREFIX}/bin/protoc
+ARG PROTOC_VERSION=3.3.0
+
+FROM ${REGISTRY}debian:stretch-slim
+MAINTAINER Voltha Community <info@opennetworking.org>
+
+ENV PROTOC_PREFIX=/usr/local
+ENV ROTOC_LIBDIR=${PROTOC_PREFIX}/lib
+ENV PROTOC=${PROTOC_PREFIX}/bin/protoc
+ENV PROTOC_VERSION=3.3.0
+ENV PROTOC_DOWNLOAD_PREFIX=https://github.com/google/protobuf/releases/download
+ENV PROTOC_DIR=protobuf-${PROTOC_VERSION}
+ENV PROTOC_TARBALL=protobuf-python-${PROTOC_VERSION}.tar.gz
+ENV PROTOC_DOWNLOAD_URI=${PROTOC_DOWNLOAD_PREFIX}/v${PROTOC_VERSION}/${PROTOC_TARBALL}
+
+RUN apt-get update -y && apt-get install -y wget build-essential python-dev python-pip
+RUN pip install --upgrade pip && pip install grpcio-tools==1.3.5
+WORKDIR /build
+RUN wget -q --no-check-certificate ${PROTOC_DOWNLOAD_URI}
+RUN tar --strip-components=1 -zxf ${PROTOC_TARBALL}
+RUN ./configure --prefix=${PROTOC_PREFIX}
+RUN make install
diff --git a/docker/Dockerfile.protos b/docker/Dockerfile.protos
new file mode 100644
index 0000000..745e166
--- /dev/null
+++ b/docker/Dockerfile.protos
@@ -0,0 +1,40 @@
+# Copyright 2018 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG REGISTRY=
+ARG REPOSITORY=
+ARG TAG=latest
+
+FROM ${REGISTRY}${REPOSITORY}voltha-protoc:${TAG} as builder
+MAINTAINER Voltha Community <info@opennetworking.org>
+
+COPY voltha/protos/third_party/google/api/*.proto /protos/google/api/
+COPY docker/config/Makefile.protos /protos/google/api/Makefile.protos
+WORKDIR /protos
+RUN make -f google/api/Makefile.protos google_api
+RUN touch /protos/google/__init__.py /protos/google/api/__init__.py
+
+COPY voltha/protos/*.proto /protos/voltha/
+COPY docker/config/Makefile.protos /protos/voltha/Makefile.protos
+WORKDIR /protos/voltha
+RUN make -f Makefile.protos build
+
+COPY voltha/adapters/asfvolt16_olt/protos/*.proto /protos/asfvolt16_olt/
+COPY docker/config/Makefile.protos /protos/asfvolt16_olt/Makefile.protos
+WORKDIR /protos/asfvolt16_olt
+RUN make -f Makefile.protos build
+
+# Copy the files to a scrach based container to minimize its size
+FROM ${REGISTRY}scratch
+COPY --from=builder /protos/ /protos/
diff --git a/docker/Dockerfile.test_runner b/docker/Dockerfile.test_runner
new file mode 100644
index 0000000..72ea3d2
--- /dev/null
+++ b/docker/Dockerfile.test_runner
@@ -0,0 +1,37 @@
+# Copyright 2018 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+ARG TAG=latest
+ARG REGISTRY=
+ARG REPOSITORY=
+
+FROM ${REGISTRY}${REPOSITORY}voltha-protos:${TAG} as protos
+FROM ${REGISTRY}${REPOSITORY}voltha-base:${TAG}
+MAINTAINER Voltha Community <info@opennetworking.org>
+
+RUN apt-get update -y && apt-get install -y curl jq kafkacat make
+RUN curl -sSL get.docker.io | CHANNEL=stable bash
+RUN apt-get install -y docker-compose
+
+COPY ./ /work/
+WORKDIR /work
+
+# Copy in the generated GRPC proto code
+COPY --from=protos /protos/voltha /work/voltha/protos
+COPY --from=protos /protos/google/api /work/voltha/protos/third_party/google/api
+
+COPY --from=protos /protos/asfvolt16_olt /work/voltha/adapters/asfvolt16_olt/protos
+
+COPY --from=protos /protos/voltha /work/ofagent/protos
+COPY --from=protos /protos/google/api /work/ofagent/protos/third_party/google/api
diff --git a/docker/Dockerfile.voltha_d b/docker/Dockerfile.voltha_d
new file mode 100644
index 0000000..0032232
--- /dev/null
+++ b/docker/Dockerfile.voltha_d
@@ -0,0 +1,37 @@
+# Copyright 2018 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+ARG TAG=latest
+ARG REGISTRY=
+ARG REPOSITORY=
+
+FROM ${REGISTRY}${REPOSITORY}voltha-protos:${TAG} as protos
+FROM ${REGISTRY}${REPOSITORY}voltha-base:${TAG}
+MAINTAINER Voltha Community <info@opennetworking.org>
+
+# Bundle app source
+RUN mkdir /voltha && touch /voltha/__init__.py
+ENV PYTHONPATH=/voltha
+COPY common /voltha/common
+COPY voltha /voltha/voltha
+COPY pki /voltha/pki
+
+# Copy in the generated GRPC proto code
+COPY --from=protos /protos/voltha /voltha/voltha/protos
+COPY --from=protos /protos/google/api /voltha/voltha/protos/third_party/google/api
+COPY --from=protos /protos/asfvolt16_olt /voltha/voltha/adapters/asfvolt16_olt/protos
+
+# Exposing process and default entry point
+# EXPOSE 8000
+CMD ["python", "voltha/voltha/main.py"]
diff --git a/docker/config/Makefile.protos b/docker/config/Makefile.protos
new file mode 100644
index 0000000..12ff9e3
--- /dev/null
+++ b/docker/config/Makefile.protos
@@ -0,0 +1,59 @@
+#
+# Copyright 2017 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Makefile to build all protobuf and gRPC related artifacts
+
+default: build
+
+PROTO_FILES := $(wildcard *.proto)
+PROTO_ALL_FILES := $(PROTO_FILES) $(PROTO_GOOGLE_API)
+PROTO_PB2_FILES := $(foreach f,$(PROTO_FILES),$(subst .proto,_pb2.py,$(f)))
+PROTO_PB2_GOOGLE_API := $(foreach f,$(PROTO_GOOGLE_API),$(subst .proto,_pb2.py,$(f)))
+PROTO_All_PB2_C_FILES := $(foreach f,$(PROTO_ALL_FILES),$(subst .proto,_pb2.pyc,$(f)))
+PROTO_ALL_PB2_GPRC_FILES := $(foreach f,$(PROTO_ALL_FILES),$(subst .proto,_pb2_grpc.py,$(f)))
+PROTO_ALL_DESC_FILES := $(foreach f,$(PROTO_ALL_FILES),$(subst .proto,.desc,$(f)))
+
+# Google API needs to be built from within the third party directory
+#
+google_api:
+ python -m grpc.tools.protoc \
+ -I. \
+ --python_out=. \
+ --grpc_python_out=. \
+ --descriptor_set_out=google/api/annotations.desc \
+ --include_imports \
+ --include_source_info \
+ google/api/annotations.proto google/api/http.proto
+
+build: $(PROTO_PB2_FILES)
+
+%_pb2.py: %.proto
+ python -m grpc.tools.protoc \
+ -I. \
+ -I/protos \
+ --python_out=. \
+ --grpc_python_out=. \
+ --descriptor_set_out=$(basename $<).desc \
+ --include_imports \
+ --include_source_info \
+ $<
+
+clean:
+ rm -f $(PROTO_PB2_FILES) \
+ $(PROTO_ALL_DESC_FILES) \
+ $(PROTO_ALL_PB2_GPRC_FILES) \
+ $(PROTO_All_PB2_C_FILES) \
+ $(PROTO_PB2_GOOGLE_API)
diff --git a/envoy/go/envoyd/build_binary.sh b/envoy/go/envoyd/build_binary.sh
index ff537a9..a774e21 100755
--- a/envoy/go/envoyd/build_binary.sh
+++ b/envoy/go/envoyd/build_binary.sh
@@ -5,8 +5,6 @@
rm -fr buildreport
rm -f envoyd
docker run -e "http_proxy=$http_proxy" -e "https_proxy=$https_proxy" -v $(pwd):/src ${REGISTRY}${REPOSITORY}voltha-go-builder:${TAG}
-#/build.sh
uid=`id -u`
-gid=`id -g`
-sudo chown -R ${uid}:${gid} buildreport
-sudo chown ${uid}:${gid} envoyd
+sudo chown -R ${uid} buildreport
+sudo chown ${uid} envoyd
diff --git a/envoy/go/golang-builder/build.sh b/envoy/go/golang-builder/build.sh
index 0137347..cdca3b5 100755
--- a/envoy/go/golang-builder/build.sh
+++ b/envoy/go/golang-builder/build.sh
@@ -58,7 +58,7 @@
do
# Grab the last segment from the package name
name=${pkg##*/}
- echo "* Building Go binary: $pkg"
+ echo "* Building Go binary: $pkg to goPath/src/$pkg/$name"
flags=(-a -installsuffix cgo)
ldflags=('-s -X main.version='$BUILD_VERSION)
@@ -76,6 +76,12 @@
goupx $name
fi
+ if [ "$OUTPUT X" != " X" ]; then
+ echo "* Copy Go Binary to \"$OUTPUT/$name\""
+ mkdir -p $OUTPUT
+ cp "$goPath/src/$pkg/$name" "$OUTPUT/$name"
+ fi
+
if [ -e "/var/run/docker.sock" ] && [ -e "$goPath/src/$pkg/Dockerfile" ];
then
@@ -87,4 +93,4 @@
# Build the image from the Dockerfile in the package directory
docker build --pull -t $tagName .
fi
-done
\ No newline at end of file
+done
diff --git a/settings.vagrant.nightly-docker.yaml b/settings.vagrant.nightly-docker.yaml
new file mode 100644
index 0000000..1e10923
--- /dev/null
+++ b/settings.vagrant.nightly-docker.yaml
@@ -0,0 +1,13 @@
+---
+# The name to use for the server
+server_name: "voltha"
+# Use virtualbox for development
+vProvider: "virtualbox"
+build_mode: "docker"
+#vagrant_box: "ubuntu/xenial64"
+# This determines if test mode is active
+testMode: "false"
+# This determines if installer mode is active
+installMode: "false"
+# Use KVM for production
+#vProvider: "KVM"
diff --git a/settings.vagrant.yaml b/settings.vagrant.yaml
index 7df64d2..54b2639 100644
--- a/settings.vagrant.yaml
+++ b/settings.vagrant.yaml
@@ -4,6 +4,9 @@
# Use virtualbox for development
vProvider: "virtualbox"
#vagrant_box: "ubuntu/xenial64"
+# This setting determines if VOLTHA is built with only docker or with host
+# install tools
+#build_mode: "docker"
# This determines if test mode is active
testMode: "false"
# This determines if installer mode is active
diff --git a/tests/itests/docutests/build_md_test.py b/tests/itests/docutests/build_md_test.py
index 57c273a..a6f5450 100644
--- a/tests/itests/docutests/build_md_test.py
+++ b/tests/itests/docutests/build_md_test.py
@@ -40,12 +40,14 @@
LOCAL_CONSUL = "localhost:8500"
LOCAL_CONSUL_URL = "http://%s" % LOCAL_CONSUL
LOCAL_CONSUL_DNS = "@localhost -p 8600"
+DOCKER_COMPOSE_PROJECT = "compose"
DOCKER_COMPOSE_FILE = "compose/docker-compose-docutests.yml"
DOCKER_COMPOSE_FILE_SERVICES_COUNT = 7
command_defs = dict(
makefile_fetch_images="grep \"docker pull\" Makefile",
make="make",
+ make_clean_build="make -e DOCKER_CACHE_ARG=--no-cache build",
make_fetch="make fetch",
remove_env_directory="rm -rf venv-linux",
make_clean="make clean",
@@ -53,9 +55,10 @@
docker_stop="docker stop",
docker_rm="docker rm",
fluentd_logs="less /tmp/fluentd/data.log",
- docker_voltha_logs="docker logs -f compose_voltha_1",
- docker_compose_logs="docker-compose -f {} logs".format(
- DOCKER_COMPOSE_FILE),
+ docker_voltha_logs="docker-compose -p {} -f {} logs voltha"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_logs="docker-compose -p {} -f {} logs"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
docker_stop_and_remove_all_containers="docker stop `docker ps -q` ; "
"docker rm `docker ps -a -q`",
docker_start_voltha="docker run -ti --rm voltha/voltha",
@@ -66,29 +69,31 @@
"compose_consul_1 | jq -r "
"'.[0].NetworkSettings.Networks."
"compose_default.IPAddress'",
- docker_compose_start_consul="docker-compose -f {} up -d "
- "consul".format(DOCKER_COMPOSE_FILE),
- docker_compose_start_all="docker-compose -f {} up -d "
- .format(DOCKER_COMPOSE_FILE),
- docker_compose_stop="docker-compose -f {} stop"
- .format(DOCKER_COMPOSE_FILE),
- docker_compose_rm_f="docker-compose -f {} rm -f"
- .format(DOCKER_COMPOSE_FILE),
- docker_compose_ps="docker-compose -f {} ps".format(DOCKER_COMPOSE_FILE),
+ docker_compose_start_consul="docker-compose -p {} -f {} up -d "
+ "consul".format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_start_all="docker-compose -p {} -f {} up -d "
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_stop="docker-compose -p {} -f {} stop"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_rm_f="docker-compose -p {} -f {} rm -f"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_down="docker-compose -p {} -f {} down"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_ps="docker-compose -p {} -f {} ps"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
docker_ps="docker ps",
docker_ps_count="docker ps -q | wc -l",
- docker_compose_is_consul_up="docker-compose -f {} ps | grep consul"
- .format(DOCKER_COMPOSE_FILE),
+ docker_compose_is_consul_up="docker-compose -p {} -f {} ps | grep consul"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
consul_get_leader_ip_port="curl -s {}/v1/status/leader | jq -r ."
.format(LOCAL_CONSUL_URL),
- docker_compose_services_running="docker-compose -f {} ps -q"
- .format(DOCKER_COMPOSE_FILE),
- docker_compose_services_running_count="docker-compose -f {} ps -q | "
- "grep Up "
- "| wc -l".format(
- DOCKER_COMPOSE_FILE),
- docker_compose_services="docker-compose -f {} config --services"
- .format(DOCKER_COMPOSE_FILE),
+ docker_compose_services_running="docker-compose -p {} -f {} ps -q"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_services_running_count="docker-compose -p {} -f {} ps -q | "
+ "grep Up | wc -l"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_services="docker-compose -p {} -f {} config --services"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
consul_get_services="curl -s {}/v1/catalog/services | jq -r ."
.format(LOCAL_CONSUL_URL),
consul_get_srv_voltha_health="curl -s {}/v1/catalog/service/voltha-health "
@@ -103,11 +108,12 @@
"voltha-health.service.consul SRV | "
" awk \'{{print $3}}'"
.format(LOCAL_CONSUL_DNS),
- docker_compose_scale_voltha_to_10="docker-compose -f {} scale "
- "voltha=10".format(DOCKER_COMPOSE_FILE),
- docker_compose_scaled_voltha_ps="docker-compose -f {} ps voltha | "
+ docker_compose_scale_voltha_to_10="docker-compose -p {} -f {} scale "
+ "voltha=10"
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
+ docker_compose_scaled_voltha_ps="docker-compose -p {} -f {} ps voltha | "
"grep Up | wc -l"
- .format(DOCKER_COMPOSE_FILE),
+ .format(DOCKER_COMPOSE_PROJECT, DOCKER_COMPOSE_FILE),
consul_verify_voltha_registration="curl -s {}"
"/v1/kv/service/voltha/members?recurse |"
" jq -r .".format(LOCAL_CONSUL_DNS)
@@ -195,14 +201,14 @@
"secs \n\n".format(time.time() - t0)
def test_03_make(self):
- print "Test_03_make_Start:------------------"
+ print "Test_03_make_build_Start:------------------"
t0 = time.time()
try:
- cmd = command_defs['make']
+ cmd = command_defs['make_clean_build']
out, err, rc = run_command_to_completion_with_raw_stdout(cmd)
self.assertEqual(rc, 0)
finally:
- print "Test_03_make_Start:------------------ took {} secs \n\n" \
+ print "Test_03_make_build_Start:------------------ took {} secs \n\n" \
.format(time.time() - t0)
def test_04_run_voltha_standalone_without_consul(self):
@@ -368,7 +374,12 @@
try:
# Pre-test - clean up all running docker containers
print "Pre-test: Removing all running containers ..."
- self._stop_and_remove_all_containers()
+ cmd = command_defs['docker_compose_stop']
+ _, err, rc = run_command_to_completion_with_raw_stdout(cmd)
+ self.assertEqual(rc, 0)
+ cmd = command_defs['docker_compose_rm_f']
+ _, err, rc = run_command_to_completion_with_raw_stdout(cmd)
+ self.assertEqual(rc, 0)
# get a list of services in the docker-compose file
print "Getting list of services in docker compose file ..."
@@ -486,8 +497,7 @@
'registrator_1', 'kafka_1', 'zookeeper_1',
'ofagent_1', 'netconf_1']
cmd = command_defs['docker_compose_logs']
- docker_compose_logs = run_long_running_command_with_timeout(cmd, 5,
- 0)
+ docker_compose_logs = run_long_running_command_with_timeout(cmd, 5, 0)
intersected_logs = [l for l in expected_output if
l in docker_compose_logs]
self.assertEqual(len(intersected_logs), len(expected_output))
@@ -499,7 +509,7 @@
expected_output = ['coordinator._renew_session', 'main.heartbeat']
cmd = command_defs['docker_voltha_logs']
docker_voltha_logs = run_long_running_command_with_timeout(cmd,
- 0.5, 3)
+ 0.5, 5)
intersected_logs = [l for l in expected_output if
l in docker_voltha_logs]
self.assertEqual(len(intersected_logs), len(expected_output))
@@ -507,7 +517,9 @@
finally:
print "Stopping all containers ..."
# clean up all created containers for this test
- self._stop_and_remove_all_containers()
+ #self._stop_and_remove_all_containers()
+ cmd = command_defs['docker_compose_down']
+ _, err, rc = run_command_to_completion_with_raw_stdout(cmd)
print "Test_07_start_all_containers_End:------------------ took {}" \
" secs \n\n".format(time.time() - t0)