There are many ways to build and develop Voltha:
brew install jq; on Ubuntu you can do it with sudo apt-get install jq. You will not regret it.To checkout the the voltha source code you will need to install repo. Intructions for this can be found here
If you have not cloned Voltha, it's time to do it now.
repo init -u https://gerrit.opencord.org/manifest -g voltha repo sync
You can build Voltha by:
cd opencord/incubator/voltha vagrant up # when you do this for the first time, this will take considerable time vagrant ssh # the rest to be executed inside the vagrant VM cd /cord/incubator/voltha . env.sh make fetch make
The above has generated a new Docker image '''cord/voltha''' inside the VM. To see it, run:
docker images
The simplest way to run the image (in the foreground):
docker run -ti --rm cord/voltha
Unless you happen to have a consul agent running on your local system, you shall see that voltha is trying to connect to a consul agent, without success.
To bring up a consul agent, you can use docker-compose with the provided compose file:
docker-compose -f compose/docker-compose-system-test.yml up -d consul
You may get a warning from docker-compose about an empty environment variable, which you can ignore for now.
By now you should have consul running, which you can verify:
docker-compose -f compose/docker-compose-system-test.yml ps
To verify that consul is indeed up, you can point your web browser to http://localhost:8500/ui. Alternatively, you can use curl to access consul's REST API. For example:
curl -s http://localhost:8500/v1/status/leader | jq -r .
This should print the IP address (on the docker network) and port number of the internal gossip API for our consul instance (you can ignore the actual data).
Once consul is up, you can extract its IP address programmatically by:
CONSUL_IP=`docker inspect compose_consul_1 | \
jq -r '.[0].NetworkSettings.Networks.compose_default.IPAddress'`
With the IP address in hand, you can now start Voltha manually as:
docker run -ti --rm --net=compose_default cord/voltha /voltha/voltha/main.py --consul=$CONSUL_IP:8500
This time it should successfully connect to consul and actually register itself. You should see a log line simialr to the following:
<timestamp> INFO coordinator._create_session {session_id: <session_id, event: created-consul-session, instance_id: <instance_id}
To test Voltha's self-registration with consul, run this (in another terminal):
curl -s http://localhost:8500/v1/kv/service/voltha/members?recurse | jq -r .
This should print one key-value entry, something similar to:
[
{
"ModifyIndex": 19,
"CreateIndex": 19,
"Session": "139181ff-d4f8-d11c-c450-c41de4baa7d7",
"Value": "YWxpdmU=",
"Flags": 0,
"Key": "service/voltha/members/23d0f3db5877",
"LockIndex": 1
}
]
The above is an ephemeral record in consul. To see how it is auto-deleted, stop the voltha container (with Ctrl-C) and rerun the above curl query again. This time you should no output form the query.
To clean up, stop consul:
docker-compose -f compose/docker-compose-system-test.yml stop docker-compose -f compose/docker-compose-system-test.yml rm -f
Voltha's intended use is with a consul+fluent+registrator combo. You can bring up this entire ensamble in a developer mode, with just one command. But first, you need to setup your local environment variable. This needs to be done only once in a terminal session:
source env.sh
To start the eight-container ensamble:
docker-compose -f compose/docker-compose-system-test.yml up -d
There are a lot of things going on between these containers:
To verify that they are all up and happy, you can use:
docker-compose -f compose/docker-compose-system-test.yml ps
Naturally, you can always use plain docker commands too:
docker ps -a
The registrator container is auto-registering docker containers and their services with consul. This will be handy when accessing the voltha cluster from the north-bound. To see the auto-registrations via Consul's HTTP API:
curl -s http://localhost:8500/v1/catalog/services | jq -r .
This shall list something like this:
{
"zookeeper": [],
"chameleon-rest": [],
"consul": [],
"consul-8600": [
"udp"
],
"consul-rest": [],
"fluentd-intake": [],
"kafka": [],
"voltha-grpc": [],
"voltha-health": []
}
You don't see registrator itself, and you see multiple entries for consul. More importantly you see voltha as a service called "voltha-health" (referring to the REST health check service of voltha). You can query additional info on this endpoint from consul:
curl -s http://localhost:8500/v1/catalog/service/voltha-health | jq -r .
This will provide the complete service record for the voltha instance:
[
{
"ModifyIndex": 14,
"CreateIndex": 9,
"ServiceEnableTagOverride": false,
"Node": "ff30117fb001",
"Address": "172.18.0.3",
"TaggedAddresses": {
"wan": "172.18.0.3",
"lan": "172.18.0.3"
},
"ServiceID": "ef1b466f2ede:compose_voltha_1:8880",
"ServiceName": "voltha-health",
"ServiceTags": [],
"ServiceAddress": "10.0.2.15",
"ServicePort": 32768
}
]
Voltha is logging to its standard output, which is captured by docker. There are multiple ways to access this log stream:
To see (and tail) logs form all running containers that are part of the voltha:
docker-compose -f compose/docker-compose-system-test.yml logs
Once important thing you can see is that voltha uses structured logging, which will come handy once we utilize machine parsing and filtering of the logs.
Alternatively, you can see the individual docker log stream of voltha by:
docker logs -f compose_voltha_1
Voltha sends a periodic log message (heartbeat) to the log stream, in addition to logging major events. Voltha also sends a periodic heartbeat message to the kafka broker.
To subscribe and display to the kafka messages:
python kafka/kafka-consumer.py
In addition to the docker log stream, Voltha is explicitly hooked up to the fluentd log collector infrastructure. We are not using fluentd to its full potential yet, but establising the connection to fluentd and funelling structured logs to fluentd is already in place. To see the fluentd log stream, you can run:
tail -F /tmp/fluentd/*.log
When regstrator registers voltha with consul, it also initiates a periodic healt-check from consul to the voltha instance. This is done by calling the http://<voltha-instance-ip>:<port>/health REST API of voltha. This does not yet do too much, but helps consul to flag a voltha instance "failing" should it not reply to the REST request in a timely manner.
If you look into the lo stream of voltha, you can see an entry every 3 seconds indicating that voltha received the health check request and responded (you need -v (verbose mode) enabled to see these).
One way to see the health checks are passing is to point your browser to the user interface of consul: http://10.100.198.220:8500/ui. Click on the voltha-health entry and you shall see its passing two health tests, the one with the name volta-rest is our healthcheck.
Consul exposes the service records also as a DNS server. This is how it can be used:
To check the IP address(es) for voltha's REST interface, you can use:
dig @localhost -p 8600 voltha-health.service.consul
Which shall print, among other things an A record:
voltha-health.service.consul. 0 IN A 10.0.2.15
Or if you want the IP adress only:
dig @localhost -p 8600 +short voltha-health.service.consul
Which shall print just an IP address.
If you want the exposed service port as well:
dig @localhost -p 8600 +short voltha-health.service.consul SRV
The 3rd field in the response is the exposed TCP port voltha's REST API is accessible.
Now something really cool: voltha can be scaled out horizontally to multiple containers by:
docker-compose -f compose/docker-compose-system-test.yml scale voltha=10
This will bring up nine (9) additional voltha instances as new docker containers. After this completes we encourage you to re-run steps 5, 6 and 7, and observe the changes.
Finally, you can clean up:
To stop all docker instances started by docker-compose, just run:
docker-compose -f compose/docker-compose-system-test.yml stop docker-compose -f compose/docker-compose-system-test.yml rm -f
Or, you can just wipe out your whole Vagrant instance:
exit # from vagrant box back to your native environmnet vagrant destroy -f
Follow the steps below to run integration testing. Note: All output are directed to the shell:
make itest
For advanced developers this may provide a more comfortable developer environment (e.g., by allowing IDE-assisted debugging), but setting it up can be a bit more challenging.
The steps that may work (see list of workarounds in case it does not):
git clone git@bitbucket.org:corddesign/voltha.git cd voltha make venv
Potential issues and workaround:
Missing virtualenv binary. Resolution: install virtualenv.
brew install python pip virtualenv
'make venv' exits with error 'openssl/opensslv.h': file not found. Resolution: install openssl-dev and add a CFLAGS to make venv:
MAC OS:
brew install openssl
Linux:
sudo apt-get install libssl-dev
Note the version that it installed. For example, '1.0.2h_1'. Rerun make venv as:
env CFLAGS="-I /usr/local/Cellar/openssl/1.0.2h_1/include" make venv
These steps are not different from the Vagrant path:
make
Then you shall be able to see the created image and run the container:
docker run -ti cord/voltha
After this, much or all of the things you can do inside the Vagrant box should also work natively on the Mac.
The dreaded "Need to install scapy for packet parsing" error when running olt-oftest based tests. This is due to a missing dnet package which scapy needs. Workaround:
brew uninstall libdnet
brew install --with-python libdnet
cd $VOLTHA_BASE
echo 'import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")' \
>> venv-darwin/lib/python2.7/site-packages/homebrew.pth
pip install pcapy
pip install scapy
pip install pypcap
Missing mininet.topo module (used by oftest):
Unfortunately I was not yet able to resolve this on the Mac.
cd $VOLTHA_BASE . env.sh mkdir tmp cd tmp git clone https://github.com/dugsong/libdnet.git cd libdnet ./configure make sudo make install cd python python setup.py install cd ../.. rm -fr tmp