Fix heading names and links. Additions to troubleshooting docs.
Change-Id: If8f9097c22e54233676352890f3c5431af9ba098
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 9249df3..2add8c2 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -30,6 +30,7 @@
* [Developing for CORD](develop.md)
* [Getting the Source Code](getting_the_code.md)
* [Workflow: Mock Configuration](xos/dev/workflow_mock_single.md)
+ * [Workflow: Local](xos/dev/workflow_local.md)
* [Workflow: Virtual POD](xos/dev/workflow_pod.md)
* [Example Service](xos/example_service.md)
* [Configuring XOS](xos/modules/xosconfig.md)
diff --git a/docs/book.json b/docs/book.json
index 262c854..6c39c52 100644
--- a/docs/book.json
+++ b/docs/book.json
@@ -8,6 +8,7 @@
"branch": "master"
},
"plugins": [
+ "anchorjs",
"toggle-chapters",
"versions-select"
],
diff --git a/docs/build_images.md b/docs/build_images.md
index 1fef4df..8ca842b 100644
--- a/docs/build_images.md
+++ b/docs/build_images.md
@@ -181,9 +181,9 @@
modifications).
To do this, add the full name of the image, including a version tag, to the
-`https://github.com/opencord/cord/blob/{{ book.branch }}/docker_images.yml`
-file, and to `docker_image_whitelist` list in the
-`scenarios/<scenario name>/config.yml` file.
+`pull_only_images` list in the `docker_images.yml` file, and to
+`docker_image_whitelist` list in the `scenarios/<scenario name>/config.yml`
+file.
These images will be retagged with a `candidate` tag after being pulled.
diff --git a/docs/getting_the_code.md b/docs/getting_the_code.md
index 8c421e6..c80913a 100644
--- a/docs/getting_the_code.md
+++ b/docs/getting_the_code.md
@@ -54,8 +54,8 @@
Which downloads a patch for the `xos` git repo, patchset number `1234` and
version `3`.
-Also see [Configuring your Development Environment:cord-bootstrap.sh script
-](install.md#cord-bootstrap.sh-script) for instructions on downloading
+Also see [Required Tools:cord-bootstrap.sh script
+](install.html#cord-bootstrapsh-script) for instructions on downloading
patchsets during a build using the `cord-bootstrap.sh` script.
## Contributing code to CORD
diff --git a/docs/install.md b/docs/install.md
index 2a381af..48248d8 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -18,20 +18,23 @@
If you've run into trouble or want to know more about the CORD build process,
please see [Troubleshooting and Build Internals](troubleshooting.md).
-## Configuring your Development Environment
+## Required Tools
-CORD has a unified development and deployment environment which uses the
+CORD has a unified build system for development and deployment which uses the
following tools:
- - [Ansible](https://docs.ansible.com/ansible/intro_installation.html)
- - [Repo](https://source.android.com/source/downloading#installing-repo)
+ - [Ansible](https://docs.ansible.com/ansible/intro_installation.html), *tested
+ with v2.4*
+ - [Repo](https://source.android.com/source/downloading#installing-repo),
+ *tested with v1.23 of repo (launcher)*
And either:
- [Docker](https://www.docker.com/community-edition), for *local* build
- scenarios
- - [Vagrant](https://www.vagrantup.com/downloads.html), for all other
- scenarios
+ scenarios, *tested with Community Edition version 17.06*
+ - [Vagrant](https://www.vagrantup.com/downloads.html), for all other scenarios
+ *tested with version 1.9.3, requires specific plugins and modules if using
+ with libvirt, see `cord-bootstrap.sh` for more details *
You can manually install these on your development system - see [Getting the
Source Code](getting_the_code.md) for a more detailed instructions for checking
@@ -39,10 +42,9 @@
### cord-bootstrap.sh script
-If you're working on an Ubuntu 14.04 system (CloudLab or another test
-environment), you can use the `cord-bootstrap.sh` script to install these tools
-and check out the CORD source tree to `~/cord`. This hasn't been tested on
-other versions or distributions.
+If you're working on an cleanly installed Ubuntu 14.04 system, you can use the
+`cord-bootstrap.sh` script to install these tools and check out the CORD source
+tree to `~/cord`.
<pre><code>
curl -o ~/cord-bootstrap.sh https://raw.githubusercontent.com/opencord/cord/{{ book.branch }}/scripts/cord-bootstrap.sh
@@ -62,7 +64,7 @@
Using the `-v` option is required to install Vagrant for running a [Virtual Pod
(CiaB)](install_virtual.md), whereas `-d` is required to install Docker for a
-[Local Workflow](xos/dev/workflow_local.md).
+[Local Workflow](/xos/dev/workflow_local.md).
The `-p` option downloads a patch from gerrit, and the syntax for this is
`<project path>:<changeset>/<revision>`. It can be used multiple
@@ -80,7 +82,7 @@
You can also run make targets with the `-t` option; `-t build` is the same as
running `cd ~/cord/build ; make -j4 build` after the rest of the installations
-and downloads have completed.
+and downloads have completed.
In some cases, you may see a message like this if you install software that
adds you to a group and you aren't already a member:
@@ -117,19 +119,23 @@
The top level configuration for a build is the *POD config* file, which is a
YAML file stored in
-[build/podconfig](https://github.com/opencord/cord/tree/master/podconfig) that
+[build/podconfig](https://github.com/opencord/cord/tree/{{ book.branch }}/podconfig) that
contains a list of variables that control how the build proceeds, and can
-override the configuration of the rest of the build.
+override the configuration of the rest of the build.
A minimal POD Config file must define two variables:
`cord_scenario` - the name of the *scenario* to use, which is defined in a
-directory under [build/scenarios](https://github.com/opencord/cord/tree/master/scenarios).
+directory under [build/scenarios](https://github.com/opencord/cord/tree/{{
+ book.branch }}/scenarios).
`cord_profile` - the name of a *profile* to use, defined as a YAML file in
-[build/platform-install/profile_manifests](https://github.com/opencord/platform-install/tree/master/profile_manifests).
+[build/platform-install/profile_manifests](https://github.com/opencord/platform-install/tree/{{
+ book.branch }}/profile_manifests).
-The included POD configs are generally named `<profile>-<scenario>.yml`.
+The included POD configs are generally named `<profile>-<scenario>.yml`, except
+for the `physical-example.yml` file which is used for a [Physical
+ POD](install_physical.md) and requires a bit more work to configured.
POD configs are used during a build by passing them with the `PODCONFIG`
variable to `make` - ex: `make PODCONFIG=rcord-virtual.yml config`
@@ -138,7 +144,8 @@
The set of services that XOS on-boards into CORD -- the _Service Graph_, and
other per-profile configuration for a CORD deployment. These are located in
-[build/platform-install/profile_manifests](https://github.com/opencord/platform-install/tree/master/profile_manifests).
+[build/platform-install/profile_manifests](https://github.com/opencord/platform-install/tree/{{
+ book.branch }}/profile_manifests).
### Scenarios
@@ -146,11 +153,11 @@
installed into, a default mapping of ansible groups to nodes, the set of Docker
images that can be built, and software and platform features are installed onto
those nodes. Scenarios are subdirectories of the
-[build/scenarios](https://github.com/opencord/cord/tree/master/scenarios)
-directory, and consist of a `config.yaml` file and possibly VM's specified in a
-`Vagrantfile`.
+[build/scenarios](https://github.com/opencord/cord/tree/{{ book.branch
+}}/scenarios) directory, and consist of a `config.yaml` file and possibly VM's
+specified in a `Vagrantfile`.
-The current set of scenarios:
+The current set of scenarios:
- `local`: Minimal set of containers running locally on the development host
- `mock`: Creates a single Vagrant VM with containers and DNS set up, without
@@ -160,4 +167,5 @@
- `cord`: Physical or virtual multi-node CORD pod, with MaaS and OpenStack
- `opencloud`: Physical or virtual multi-node OpenCloud pod, with OpenStack
-The scenario is specified in the POD config.
+The scenario is specified in the POD config's `cord_scenario` line.
+
diff --git a/docs/install_physical.md b/docs/install_physical.md
index 995b28a..28da37c 100644
--- a/docs/install_physical.md
+++ b/docs/install_physical.md
@@ -252,9 +252,7 @@
### Development Machine
It should run either Ubuntu 16.04 LTS (recommended) or Ubuntu 14.04 LTS. See
-[Configuring your Development
-Environment](install.md#configuring-your-development-environment) for more
-details.
+[Required Tools](install.md#required-tools) for more details.
This documentation assumes that you checked out the CORD source tree into
`~/cord`.
@@ -312,9 +310,10 @@
POD configuration files are YAML files with extension .yml, contained in the
`/cord/build/podconfig` directory in the dev VM. You can either create a new
file with your favorite editor or copy-and-edit an existing file. The
-[rcord-physical-example.yml](https://github.com/opencord/cord/blob/master/podconfig/rcord-physical-example.yml)
-configuration file is there for this purpose. All parameters have a
-description. Optional lines have been commented out, but can be used as needed.
+[physical-example.yml](https://github.com/opencord/cord/blob/{{ book.branch }}/podconfig/physical-example.yml)
+configuration file is there for this purpose, and the most commonly set
+parameters are described. Optional lines have been commented out, but can be
+used as needed.
More information about how the network configuration for the POD can be
customized can be found in [Network Settings](appendix_network_settings.md).
diff --git a/docs/install_virtual.md b/docs/install_virtual.md
index 974395b..4fee751 100644
--- a/docs/install_virtual.md
+++ b/docs/install_virtual.md
@@ -53,9 +53,9 @@
* Bootstrap the server by installing software dependencies and checking out the
CORD code
-* (Optional) Customize the source and configuration if desired
-* Run `make` targets to build and deploy the CORD software
-* (Optional) Run end-to-end tests to verify CiaB functionality
+* (Optional) Customize the build (change source and configuration)
+* Build and deploy the CORD software using `make`
+* (Optional) Run End-to-End tests
> NOTE: If you are connecting to a remote target server, it is highly
> recommended that you run the commands in a `tmux` session on the target
@@ -65,13 +65,13 @@
### Bootstrap the server
-See [Configuring your Development Environment:cord-bootstrap.sh script
-](install.md#cord-bootstrapsh-script) for instructions for running the
+See [Required Tools:cord-bootstrap.sh script
+](install.html#cord-bootstrapsh-script) for instructions for running the
bootstrap script to download the CORD source tree and optionally downloading
patches from Gerrit. You must specify the `-v` option to this script in order
to install Vagrant, which is required to build a CiaB.
-### (Optional) Customize your build
+### (Optional) Customize the build
You can now modify your CORD source tree that has been checked out in `~/cord`,
if required, or [download patches manually from gerrit using
diff --git a/docs/quickstarts.md b/docs/quickstarts.md
index 41406a8..6efb850 100644
--- a/docs/quickstarts.md
+++ b/docs/quickstarts.md
@@ -3,11 +3,19 @@
This section provides a short list of essential commands that can be used to
deploy virtual or physical PODs.
-Before you start, you must obtain the CORD source tree and install Vagrat.
-Instructions for doing this can be found at [Configuring your Development
-Environment](install.md#configuring-your-development-environment) - if you're
-on CloudLab, most likely you will want to use the `cord-bootstrap.sh` script
-with the `-v` option.
+Before you start, you must obtain the CORD source tree and install Vagrant.
+Instructions for doing this can be found at [Required
+Tools](install.md#required-tools).
+
+To prepare a newly installed Ubuntu 14.04 system for either a Physical or
+Virtual pod, you can use the
+[cord-bootstrap.sh](install.html#cord-bootstrapsh-script) script:
+
+<pre><code>
+curl -o ~/cord-bootstrap.sh https://raw.githubusercontent.com/opencord/cord/{{ book.branch }}/scripts/cord-bootstrap.sh
+chmod +x cord-bootstrap.sh
+./cord-bootstrap.sh -v
+</code></pre>
## Virtual POD (CORD-in-a-Box)
@@ -28,8 +36,8 @@
ExampleService instances.
If you'll be running these commands frequently, a shortcut is to use the `-t`
-option on the `cord-bootstrap.sh` script to run all the make targets, for a
-more unattended build process, which can be handy when testing:
+option on the `cord-bootstrap.sh` script to download source, then run all the
+make targets involved in doing a build and test cycle:
```
./cord-bootstrap.sh -v -t "PODCONFIG=rcord-virtual.yml config" -t "build" -t "pod-test"
diff --git a/docs/scripts/descriptions.md b/docs/scripts/descriptions.md
index 7fdd076..47e2aae 100644
--- a/docs/scripts/descriptions.md
+++ b/docs/scripts/descriptions.md
@@ -579,9 +579,15 @@
DPDK setting to specify CPU pinning.
+## vsg_test_ping_target
+
+DNS name of the server to ping when running the vSG portion of the
+[pod-test](install_virtual.html#test-vsg)
+
## vtn_management_host_net_interface
-Network interface to use on the head/compute nodes for the management_host network.
+Network interface to use on the head/compute nodes for the management_host
+network.
## xos_admin_first
@@ -660,5 +666,5 @@
## xos_users
List of additional users to create in XOS, in addition to
-[xos_admin_user](#xos_admin_user).
+[xos_admin_user](#xosadminuser).
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 91b62a4..a85de66 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -1,20 +1,176 @@
# Troubleshooting and Build System Internals
+> NOTE: Most of the debugging below assumes that you are logged into the
+> `config` node, which is the node where `make` is run to start the build, and
+> have checked out the CORD source tree to `~/cord`.
+
## Debugging make target Failures
-`make` targets that are built will create a per-target log file in the `logs`
-directory. These are prefixed with a timestamp which is the same for every
-target in a single run of make - re-running make will result in additional sets
-of logs, even for the same target.
+The CORD build process uses `make` and frequently will have multiple concurrent
+jobs running at the same time to speed up the build. Unfortunately, this has
+two downsides:
-If you have a build failure and want to know which targets completed, running:
+1. The output of all concurrent jobs is shown at the same time, so the combined
+ output can be very confusing to read and it might not be clear what failed.
+2. If a target task fails, `make` will wait until all other jobs have finished,
+ and it may not be apparent which has failed.
+
+To address the first issue most CORD `make` targets will create a per-target
+log file in the `~/cord/build/logs` directory - this output isn't interleaved
+with other make targets and will be much easier to read.
```
-ls -ltr milestones ; ls -ltr logs
+~/cord/build$ make printconfig
+Scenario: 'cord'
+Profile: 'rcord'
+~/cord/build$ cd logs
+~/cord/build/logs$ ls -tr
+README.md 20171003T132550Z_publish-maas-images
+20171003T132547Z_config.mk 20171003T132550Z_docker-images
+20171003T132550Z_ciab-ovs 20171003T132550Z_core-image
+20171003T132550Z_prereqs-check 20171003T132550Z_deploy-maas
+20171003T132550Z_vagrant-up 20171003T132550Z_publish-onos-apps
+20171003T132550Z_vagrant-ssh-install 20171003T132550Z_deploy-mavenrepo
+20171003T132550Z_copy-cord 20171003T132550Z_publish-docker-images
+20171003T132550Z_cord-config 20171003T132550Z_deploy-onos
+20171003T132550Z_prep-buildnode 20171003T132550Z_start-xos
+20171003T132550Z_copy-config 20171003T132550Z_onboard-profile
+20171003T132550Z_build-maas-images 20171003T132550Z_deploy-openstack
+20171003T132550Z_build-onos-apps 20171003T132550Z_setup-automation
+20171003T132550Z_prep-headnode 20171003T132550Z_compute1-up
+20171003T132550Z_prep-computenode 20171003T143046Z_collect-diag
+20171003T132550Z_maas-prime 20171003T143046Z_pod-test
```
-And looking for logfiles without a corresponding milestone will point you to
-the make target(s) that failed.
+The logs are prefixed with a timestamp which is the same for every target in a
+single run of make. Re-running make will result in additional sets of logs,
+even for the same target, so make sure to look at the newest log when
+debugging.
+
+The various `make clean-*` targets will not delete logs between sets of builds,
+so you may want to manually delete these periodically on a heavily used copy of
+the CORD source tree.
+
+There are two solutions to the second issue. If you compare the list of
+milestones to the list of logs, you can determine which logs failed to result
+in a milestone, and thus did not complete:
+
+```
+$ cd ~/cord/build
+$ ls -ltr milestones ; ls -ltr logs
+-rw-r--r-- 1 zdw xos-PG0 181 Oct 3 13:23 README.md
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:25 ciab-ovs
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:26 prereqs-check
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:27 vagrant-up
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:27 vagrant-ssh-install
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:27 copy-cord
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:28 cord-config
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:30 prep-buildnode
+-rw-r--r-- 1 zdw xos-PG0 0 Oct 3 13:30 copy-config
+-rw-r--r-- 1 zdw xos-PG0 76 Oct 3 13:23 README.md
+-rw-r--r-- 1 zdw xos-PG0 8819 Oct 3 13:25 20171003T132547Z_config.mk
+-rw-r--r-- 1 zdw xos-PG0 4655 Oct 3 13:25 20171003T132550Z_ciab-ovs
+-rw-r--r-- 1 zdw xos-PG0 3665 Oct 3 13:26 20171003T132550Z_prereqs-check
+-rw-r--r-- 1 zdw xos-PG0 110477 Oct 3 13:27 20171003T132550Z_vagrant-up
+-rw-r--r-- 1 zdw xos-PG0 1712 Oct 3 13:27 20171003T132550Z_vagrant-ssh-install
+-rw-r--r-- 1 zdw xos-PG0 2204 Oct 3 13:27 20171003T132550Z_copy-cord
+-rw-r--r-- 1 zdw xos-PG0 40515 Oct 3 13:28 20171003T132550Z_cord-config
+-rw-r--r-- 1 zdw xos-PG0 4945 Oct 3 13:29 20171003T132550Z_prep-buildnode
+-rw-r--r-- 1 zdw xos-PG0 8246 Oct 3 13:30 20171003T132550Z_copy-config
+-rw-r--r-- 1 zdw xos-PG0 21229 Oct 3 13:33 20171003T132550Z_build-maas-images
+-rw-r--r-- 1 zdw xos-PG0 1069181 Oct 3 13:33 20171003T132550Z_build-onos-apps
+```
+
+In the case above, `build-maas-images` and `build-onos-apps` failed to reach
+the milestone state and should be inspected for errors.
+
+Additionally, `make` job failures generally have this syntax:
+`make: *** [targetname] Error #`, and can be found in log files with this
+`grep` command:
+
+```
+$ grep -F "make: ***" ~/build.out
+make: *** [milestones/build-maas-images] Error 2
+make: *** Waiting for unfinished jobs....
+make: *** [milestones/build-onos-apps] Error 2
+```
+
+## Collecting POD diagnostics
+
+There is a `collect-diag` make target that will collect diagnostic information
+for a Virtual or Physical POD. It is run automatically when the `pod-test`
+target is run, but can be run manually at any time:
+
+```
+$ cd ~/cord/build
+$ make collect-diag
+```
+
+Once it's finished running, ssh to the head node and look for a directory named
+`~/diag-<timestamp>` - a new directory will be created on repeated runs.
+
+These directories will contain subfolders with docker container logs, various
+diagnostic commands run on the head node, Juju status, ONOS diagnostics, and
+OpenStack status:
+
+```
+$ ssh head1
+vagrant@head1:~$ ls
+diag-20171003T203058
+vagrant@head1:~$ cd diag-20171003T203058/
+vagrant@head1:~/diag-20171003T203058$ ls *
+docker:
+allocator rcord_vrouter-synchronizer_1
+automation rcord_vsg-synchronizer_1
+generator rcord_vtn-synchronizer_1
+harvester rcord_vtr-synchronizer_1
+mavenrepo rcord_xos_chameleon_1
+onoscord_xos-onos_1 rcord_xos_core_1
+onosfabric_xos-onos_1 rcord_xos_db_1
+provisioner rcord_xos_gui_1
+rcord_addressmanager-synchronizer_1 rcord_xos_redis_1
+rcord_consul_1 rcord_xos_tosca_1
+rcord_exampleservice-synchronizer_1 rcord_xos_ui_1
+rcord_fabric-synchronizer_1 rcord_xos_ws_1
+rcord_onos-synchronizer_1 registry
+rcord_openstack-synchronizer_1 registry-mirror
+rcord_registrator_1 storage
+rcord_volt-synchronizer_1 switchq
+
+head:
+arp_-n cat__etc_resolv_conf route_-n sudo_virsh_list
+brctl_show date sudo_docker_ps_-a
+cat__etc_lsb-release ifconfig_-a sudo_lxc_list
+
+juju:
+juju_status_--format_json juju_status_--format_tabular
+juju_status_--format_summary
+
+onos:
+apps_-s_-a cordvtn-nodes flows nodes
+bundle_list cordvtn-ports hosts ports
+cordvtn-networks devices log_display rest_hosts
+cordvtn-node-check_compute1 dhcp-list netcfg summary
+
+openstack:
+glance_image-list neutron_net-list nova_host-list
+keystone_tenant-list neutron_port-list nova_hypervisor-list
+keystone_user-list neutron_subnet-list nova_list_--all-tenants
+```
+
+## Raising ONOS log levels
+
+If you're working on an ONOS app and want to raise the logging level of that
+specific ONOS app when debugging, there is an `onos-debug` target which will
+set the level of the any services in the
+[onos_debug_appnames](build_glossary.html#onosdebugappnames) list to
+[onos_debug_level](build_glossary.html#onosdebuglevel).
+
+Add the names of the ONOS apps to `onos_debug_appnames` and add `onos-debug`
+to the `build_targets` list in your POD config file in `~/cord/build/podconfig/`.
+
+This must be done before running the `make config` target - it won't affect an
+already-running POD.
## Config Generation Overview
@@ -102,7 +258,7 @@
not actually delete any images, just causes imagebuilder to be run again.
- `clean-genconfig`: Deletes the `make config` generated config files in
- `genconfig`, useful when switching between podconfigs
+ `genconfig`, useful when switching between POD configs
- `clean-onos`: Stops the ONOS containers on the head node
@@ -124,7 +280,7 @@
### Development workflow
-#### Updating XOS Container Images on a running pod
+#### Updating XOS Container Images on a running POD
To rebuild and update XOS container images, run: