Merge branch 'master' of bitbucket.org:corddesign/voltha
diff --git a/BUILD.md b/BUILD.md
index 5279884..c92f238 100644
--- a/BUILD.md
+++ b/BUILD.md
@@ -10,7 +10,7 @@
### Prerequisites
* Git client
-* Working installation of Vagrant -- see https://www.vagrantup.com/downloads.html
+* Working installation of Vagrant -- see [https://www.vagrantup.com/downloads.html](https://www.vagrantup.com/downloads.html)
* jq -- a useful command line too to work with JSON data. On the MAC, you can install jq with ```brew install jq```; on Ubuntu you can do it with ```sudo apt-get install jq```. You will not regret it.
@@ -47,14 +47,14 @@
docker-compose -f compose/docker-compose-system-test.yml up -d consul
```
-This should have started a consul docker container:
+You may get a warning from docker-compose about an empty environment variable, which you can ignore for now.
+
+By now you should have consul running, which you can verify:
```
docker-compose -f compose/docker-compose-system-test.yml ps
```
-The above should list the consul conatiner.
-
To verify that consul is indeed up, you can point your web browser to [http://localhost:8500/ui](http://localhost:8500/ui).
Alternatively, you can use curl to access consul's REST API. For example:
@@ -84,14 +84,200 @@
<timestamp> INFO coordinator.register {name: voltha-1, address: localhost, event: registered-with-consul}
```
-To test Voltha's actual service registration with consul, run this (in another terminal):
+To test Voltha's self-registration with consul, run this (in another terminal):
```
-curl -s http://localhost:8500/v1/catalog/service/voltha-1 | jq -r .
+curl -s http://localhost:8500/v1/kv/?recurse | jq -r .
```
+This should print one key-value entry, something similar to:
-## Building natively on MAC OS X
+```
+[
+ {
+ "ModifyIndex": 19,
+ "CreateIndex": 19,
+ "Session": "139181ff-d4f8-d11c-c450-c41de4baa7d7",
+ "Value": "YWxpdmU=",
+ "Flags": 0,
+ "Key": "service/voltha/members/23d0f3db5877",
+ "LockIndex": 1
+ }
+]
+```
+
+The above is an ephemeral record in consul. To see how it is auto-deleted, stop the voltha container (with Ctrl-C) and rerun the above curl query again. This time you should no output form the query.
+
+To clean up, stop consul:
+
+```
+docker-compose -f compose/docker-compose-system-test.yml stop
+docker-compose -f compose/docker-compose-system-test.yml rm -f
+```
+
+### Run with the "voltha ensamble"
+
+Voltha's intended use is with a consul+fluent+registrator combo. You can bring up this entire ensamble in a developer mode, with just one command. But first, you need to setup your local environment variable. This needs to be done only once in a terminal session:
+
+```
+source env.sh
+```
+
+To start the four-conatiner ensamble:
+
+```
+docker-compose -f compose/docker-compose-system-test.yml run -d
+```
+
+There are a lot of things going on between these containers:
+
+1. To verify that they are all up and happy, you can use:
+
+ ```
+ docker-compose -f compose/docker-compose-system-test.yml ps
+ ```
+
+ Naturally, you can always use plan docker commands too:
+
+ ```
+ docker ps -a
+ ```
+
+2. The registrator container is auto-registering docker containers and their services with consul. This will be handy when accessing the voltha cluster from the north-bound. To see the auto-registrations via Consul's HTTP API:
+
+ ```
+ curl -s http://localhost:8500/v1/catalog/services | jq -r .
+ ```
+
+ This shall list something like this:
+
+ ```
+ {
+ "voltha-rest": [],
+ "fluentd-intake": [],
+ "consul-rest": [],
+ "consul-8600": [
+ "udp"
+ ],
+ "consul": []
+ }
+ ```
+
+ You don't see registrator istelf, and you see multiple entries for consul. More importantly you see voltha as a service called "voltha-rest" (referring to the REST service of voltha). You can query additional info on this endpoint from consul:
+
+ ```
+ curl -s http://localhost:8500/v1/catalog/service/voltha-rest | jq -r .
+ ```
+
+ This will provide the complete service record for the voltha instance:
+
+ ```
+ [
+ {
+ "ModifyIndex": 14,
+ "CreateIndex": 9,
+ "ServiceEnableTagOverride": false,
+ "Node": "ff30117fb001",
+ "Address": "172.18.0.3",
+ "TaggedAddresses": {
+ "wan": "172.18.0.3",
+ "lan": "172.18.0.3"
+ },
+ "ServiceID": "ef1b466f2ede:compose_voltha_1:8880",
+ "ServiceName": "voltha-rest",
+ "ServiceTags": [],
+ "ServiceAddress": "10.0.2.15",
+ "ServicePort": 32768
+ }
+ ]
+ ```
+
+3. Voltha is logging to its standard output, which is captured by docker. There are multiple ways to access this log stream:
+
+ To see (and tail) logs form all running containers that are part of the voltha:
+
+ ```
+ docker-compose -f compose/docker-compose-system-test.yml logs
+ ```
+
+ Once important thing you can see is that voltha uses structured logging, which will come handy once we utilize macihine parsing and filtring of the logs.
+
+ Alternatively, you can see the individual docker log stream of voltha by:
+
+ ```
+ docker logs -f compose_voltha_1
+ ```
+
+ Voltha sends a periodic log message (heartbeat) to the log stream, in addition to logging major events.
+
+
+4. In addition to the docker log stream, Voltha is explicitly hooked up to the fluentd log collector infrastructure. We are not using fluentd to its full potential yet, but establising the connection to fluentd and funelling structured logs to fluentd is already in place. To see the fluentd log stream, you can run:
+
+ ```
+ tail -F /tmp/fluentd/*.log
+ ```
+
+5. When regstrator registers voltha with consul, it also initiates a periodic healt-check from consul to the voltha instance. This is done by calling the ```http://<voltha-instance-ip>:<port>/health``` REST API of voltha. This does not yet do too much, but helps consul to flag a voltha instance "failing" should it not reply to the REST request in a timely manner.
+
+ If you look into the lo stream of voltha, you can see an entry every 3 seconds indicating that voltha received the health check request and responded (you need -v (verbose mode) enabled to see these).
+
+ One way to see the health checks are passing is to point your browser to the user interface of consul: [http://10.100.198.220:8500/ui](http://10.100.198.220:8500/ui). Click on the voltha-rest entry and you shall see its passing two health tests, the one with the name volta-rest is our healthcheck.
+
+6. Consul exposes the service records also as a DNS server. This is how it can be used:
+
+ To check the IP address(es) for voltha's REST interface, you can use:
+
+ ```
+ dig @localhost -p 8600 voltha-rest.service.consul
+ ```
+
+ Which shall print, among other things an A record:
+
+ ```
+ voltha-rest.service.consul. 0 IN A 10.0.2.15
+ ```
+
+ Or if you want the IP adress only:
+
+ ```
+ dig @localhost -p 8600 +short voltha-rest.service.consul
+ ```
+
+ Which shall print just an IP address.
+
+ If you want the exposed service port as well:
+
+ ```
+ dig @localhost -p 8600 +short voltha-rest.service.consul SRV
+ ```
+
+ The 3rd field in the response is the exposed TCP port voltha's REST API is accessible.
+
+7. Now something really cool: voltha can be scaled out horizontally to multiple containers by:
+
+ ```
+ docker-compose -f compose/docker-compose-system-test.yml scale voltha=10
+ ```
+
+ This will bring up nine (9) additional voltha instances as new docker containers. After this completes we encourage you to re-run steps 5, 6 and 7, and observe the changes.
+
+Finally, you can clean up:
+
+To stop all docker instances started by docker-compose, just run:
+
+```
+docker-compose -f compose/docker-compose-system-test.yml stop
+docker-compose -f compose/docker-compose-system-test.yml rm -f
+```
+
+Or, you can just wipe out your whole Vagrant instance:
+
+```
+exit # from vagrant box back to your native environmnet
+vagrant destroy -f
+```
+
+# Building natively on MAC OS X
For advanced developers this may provide a more comfortable developer
environment (e.g., by allowing IDE-assisted debugging), but setting it up
@@ -151,4 +337,4 @@
docker run -ti cord/voltha
```
-
+After this, much or all of the things you can do inside the Vagrant box should also work natively on the Mac.
diff --git a/compose/TODOS.md b/compose/TODOS.md
index a47e6ac..6d26d27 100644
--- a/compose/TODOS.md
+++ b/compose/TODOS.md
@@ -1,6 +1,6 @@
## Next Steps Planned
-- [Z] Adding health-check to voltha (and consul)
+- [DONE] Adding health-check to voltha (and consul)
- [Z] Adding membership tracking
- [Z] Adding leader election
- [Z] Work out a load sharding mechanism