Added Vagrant-based test VM's and setup script for testing Kubespray
Change-Id: Ief467ab579d28fd8ee675c1c061abb37867c02b6
diff --git a/kubespray-installer/README.md b/kubespray-installer/README.md
new file mode 100644
index 0000000..dfbb8d0
--- /dev/null
+++ b/kubespray-installer/README.md
@@ -0,0 +1,81 @@
+# Kubespray Installer
+
+This script will clone
+[kubespray](https://github.com/kubernetes-sigs/kubespray) automatically, and
+deploy production ready Kubernetes cluster by kubespray.
+
+## Install public key to target nodes
+
+```bash
+# ./copy-ssh-keys.sh <set_of_target_nodes>
+./copy-ssh-keys.sh 192.168.0.1 192.168.0.2 192.168.0.3
+
+# Assign customized username if username isn't cord
+REMOTE_SSH_USER=ubuntu ./copy-ssh-keys.sh 192.168.0.1 192.168.0.2 192.168.0.3
+
+# Select the desired public key (default: id_rsa.pub)
+SSH_PUBKEY_PATH=~/.ssh/onoscorddev.pub ./copy-ssh-keys.sh 192.168.0.1 192.168.0.2 192.168.0.3
+```
+
+Then you are able to ssh into the target nodes without password, this is required by Kuberspray script.
+
+## Run the installation script
+
+```bash
+# ./setup -i <inventory_name> <set_of_target_nodes>
+./setup -i pod1 192.168.0.1 192.168.0.2 192.168.0.3
+
+# You can also pipe the output to stdout & file
+./setup -i pod1 192.168.0.1 192.168.0.2 192.168.0.3 | tee /tmp/kubespray-installer.log
+
+# Assign customized username
+REMOTE_SSH_USER=ubuntu ./setup -i pod1 192.168.0.1 192.168.0.2 192.168.0.3
+```
+
+Now you should be able to use your newly installed kubernetes cluster.
+
+## Testing kubespray-installer in VM's
+
+These scripts allow you to test a multi-node kubespray installer on a single
+machine, with additional disks attached for testing persistent storage.
+
+### Prepare test system
+
+Run the `kubespray-vagrant.sh` script which will install all the necesary
+prerequisites and tools.
+#
+Only tested on an Ubuntu 16.04 machine, with a user that has passwordless sudo
+enabled. Works/tested on CloudLab.
+
+Requires a considerable amount of CPU, RAM, and disk space - 3 VM's each with 8
+cores, 16GB RAM, and multiple 40GB drives (sparsely allocated).
+
+## Bring up VM's
+
+Check out `automation_tools` on the target system, then run the following to
+bring up VM's with `vagrant`:
+
+```shell
+vagrant up
+vagrant ssh-config >> ~/.ssh/config
+vagrant ssh-config | sed -e 's/Host k8s-0/Host 10.90.0.10/g' >> ~/.ssh/config
+```
+
+> NOTE: If you tear down and rebuild the VM's in vagrant, you will need to
+> delete the configuration of the previous VM's from ~/.ssh/config before
+> running the `vagrant ssh-config ...` commands.
+
+### Run kubespray-installer/setup.sh script
+
+```shell
+REMOTE_SSH_USER="vagrant" ./setup.sh -i test
+```
+
+### Copy config for k8s tools, bootstrap helm
+
+```shell
+mkdir ~/.kube/
+cp inventories/test/artifacts/admin.conf ~/.kube/config
+
+helm init --upgrade
+```
diff --git a/kubespray-installer/Vagrantfile b/kubespray-installer/Vagrantfile
new file mode 100644
index 0000000..2c22207
--- /dev/null
+++ b/kubespray-installer/Vagrantfile
@@ -0,0 +1,78 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+#
+# kubespray-installer test Vagrantfile
+
+$num_instances = 3
+
+# Script to format/mount vda volume for ceph testing
+# https://www.vagrantup.com/docs/provisioning/shell.html
+$vdisk_setup = <<-EOF
+echo "Formatting and mounting /dev/vda, vdb on /mnt/ceph and /mnt/local"
+parted -s /dev/vda \
+ mklabel gpt \
+ mkpart primary ext4 1MiB 100%
+
+sync
+mkfs.ext4 -q -m 2 -F /dev/vda1
+
+parted -s /dev/vdb \
+ mklabel gpt \
+ mkpart primary ext4 1MiB 20GiB \
+ mkpart primary ext4 20GiB 100%
+
+sync
+mkfs.ext4 -q -m 2 -F /dev/vdb1
+mkfs.ext4 -q -m 2 -F /dev/vdb2
+
+parted -s /dev/vdc \
+ mklabel gpt \
+ mkpart primary ext4 1MiB 20GiB \
+ mkpart primary ext4 20GiB 100%
+
+sync
+mkfs.ext4 -q -m 2 -F /dev/vdc1
+mkfs.ext4 -q -m 2 -F /dev/vdc2
+
+mkdir -p /mnt/ceph
+mkdir -p /mnt/local-storage/hdd/vdb1
+mkdir -p /mnt/local-storage/hdd/vdb2
+mkdir -p /mnt/local-storage/ssd/vdc1
+mkdir -p /mnt/local-storage/ssd/vdc2
+
+echo "/dev/vda1 /mnt/ceph ext4 defaults 0 0" | sudo tee -a /etc/fstab
+echo "/dev/vdb1 /mnt/local-storage/hdd/vdb1 ext4 defaults 0 0" | sudo tee -a /etc/fstab
+echo "/dev/vdb2 /mnt/local-storage/hdd/vdb2 ext4 defaults 0 0" | sudo tee -a /etc/fstab
+echo "/dev/vdc1 /mnt/local-storage/ssd/vdc1 ext4 defaults 0 0" | sudo tee -a /etc/fstab
+echo "/dev/vdc2 /mnt/local-storage/ssd/vdc2 ext4 defaults 0 0" | sudo tee -a /etc/fstab
+mount -a
+EOF
+
+
+Vagrant.configure("2") do |config|
+ config.vm.box = "bento/ubuntu-16.04"
+
+ (1..$num_instances).each do |i|
+ config.vm.define vm_name = "k8s-%02d" % i do |config|
+ config.vm.hostname = vm_name
+
+ config.vm.provider :libvirt do |lv|
+ lv.memory = 16384
+ lv.cpus = 8
+ lv.nested = true
+ # additional disks for testing storage, shows up as `vd[a-c]` in the VM
+ lv.storage :file, :size => '40G'
+ lv.storage :file, :size => '40G'
+ lv.storage :file, :size => '40G'
+ end
+
+ config.vm.provision "shell", inline: "swapoff -a"
+ config.vm.provision "shell", inline: $vdisk_setup
+
+ ip = "10.90.0.#{i+100}"
+ config.vm.network :private_network, ip: ip
+
+ end
+ end
+end
+
diff --git a/kubespray-installer/index.md b/kubespray-installer/index.md
deleted file mode 100644
index cd31e29..0000000
--- a/kubespray-installer/index.md
+++ /dev/null
@@ -1,31 +0,0 @@
-# Kubespray Installer
-
-This script will clone [kubespray](https://github.com/kubernetes-sigs/kubespray) automatically, and deploy production ready Kubernetes cluster by kubespray.
-
-## Install public key to target nodes
-
-```bash
-# ./copy-ssh-keys.sh <set_of_target_nodes>
-./copy-ssh-keys.sh 192.168.0.1 192.168.0.2 192.168.0.3
-
-# Assign customized username if username isn't cord
-REMOTE_SSH_USER=ubuntu ./copy-ssh-keys.sh 192.168.0.1 192.168.0.2 192.168.0.3
-
-# Select the desired public key (default: id_rsa.pub)
-SSH_PUBKEY_PATH=~/.ssh/onoscorddev.pub ./copy-ssh-keys.sh 192.168.0.1 192.168.0.2 192.168.0.3
-```
-
-Then you are able to ssh into the target nodes without password, this is required by Kuberspray script.
-
-## Run the installation script
-
-```bash
-# ./setup -i <inventory_name> <set_of_target_nodes>
-./setup -i pod1 192.168.0.1 192.168.0.2 192.168.0.3
-
-# You can also pipe the output to stdout & file
-./setup -i pod1 192.168.0.1 192.168.0.2 192.168.0.3 | tee /tmp/kubespray-installer.log
-
-# Assign customized username
-REMOTE_SSH_USER=ubuntu ./setup -i pod1 192.168.0.1 192.168.0.2 192.168.0.3
-```
diff --git a/kubespray-installer/kubespray-vagrant.sh b/kubespray-installer/kubespray-vagrant.sh
new file mode 100755
index 0000000..5aa2b23
--- /dev/null
+++ b/kubespray-installer/kubespray-vagrant.sh
@@ -0,0 +1,162 @@
+#!/usr/bin/env bash
+#
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# kubespray-vagrant.sh
+# Bootstraps Vagrant VM's for testing kubernetes-installer scripts
+
+set -eu -o pipefail
+
+# configure git
+if [ ! -e "${HOME}/.gitconfig" ]
+then
+ echo "No ${HOME}/.gitconfig, setting testing defaults..."
+ git config --global user.name 'Test User'
+ git config --global user.email 'test@null.com'
+ git config --global color.ui false
+fi
+
+# install repo if needed
+if [ ! -x "/usr/local/bin/repo" ]
+then
+ echo "Installing repo..."
+
+ REPO_SHA256SUM="394d93ac7261d59db58afa49bb5f88386fea8518792491ee3db8baab49c3ecda"
+ curl -o /tmp/repo 'https://gerrit.opencord.org/gitweb?p=repo.git;a=blob_plain;f=repo;hb=refs/heads/stable'
+ echo "$REPO_SHA256SUM /tmp/repo" | sha256sum -c -
+ sudo cp /tmp/repo /usr/local/bin/repo
+ sudo chmod a+x /usr/local/bin/repo
+fi
+
+# check out cord repo
+if [ ! -d "${HOME}/cord" ]
+then
+ # make sure we can find gerrit.opencord.org as DNS failures will fail the build
+ dig +short gerrit.opencord.org || (echo "ERROR: gerrit.opencord.org can't be looked up in DNS" && exit 1)
+
+ echo "Downloading cord with repo..."
+ pushd "${HOME}"
+ mkdir -p cord
+ cd cord
+ repo init -u https://gerrit.opencord.org/manifest -b master
+ repo sync
+ popd
+fi
+
+# if on cloudlab, format extra space - has to happen before Vagrant install or root will overfill
+"$HOME"/cord/automation-tools/scripts/cloudlab-disksetup.sh
+
+echo "Installing prereqs..."
+sudo apt-get update
+sudo apt-get -y install apt-transport-https build-essential curl git python-dev \
+ python-pip python-virtualenv software-properties-common \
+ sshpass libffi-dev qemu-kvm libvirt-bin libvirt-dev \
+ nfs-kernel-server socat
+
+# Install kubernetes tools (if not installed)
+if [ ! -x "/usr/bin/kubeadm" ]
+then
+
+ cat << EOF | base64 -d > /tmp/k8s-apt-key.gpg
+mQENBFUd6rIBCAD6mhKRHDn3UrCeLDp7U5IE7AhhrOCPpqGF7mfTemZYHf/5JdjxcOxoSFlK
+7zwmFr3lVqJ+tJ9L1wd1K6P7RrtaNwCiZyeNPf/Y86AJ5NJwBe0VD0xHTXzPNTqRSByVYtdN
+94NoltXUYFAAPZYQls0x0nUD1hLMlOlC2HdTPrD1PMCnYq/NuL/Vk8sWrcUt4DIS+0RDQ8tK
+Ke5PSV0+PnmaJvdF5CKawhh0qGTklS2MXTyKFoqjXgYDfY2EodI9ogT/LGr9Lm/+u4OFPvmN
+9VN6UG+s0DgJjWvpbmuHL/ZIRwMEn/tpuneaLTO7h1dCrXC849PiJ8wSkGzBnuJQUbXnABEB
+AAG0QEdvb2dsZSBDbG91ZCBQYWNrYWdlcyBBdXRvbWF0aWMgU2lnbmluZyBLZXkgPGdjLXRl
+YW1AZ29vZ2xlLmNvbT6JAT4EEwECACgFAlUd6rICGy8FCQWjmoAGCwkIBwMCBhUIAgkKCwQW
+AgMBAh4BAheAAAoJEDdGwginMXsPcLcIAKi2yNhJMbu4zWQ2tM/rJFovazcY28MF2rDWGOnc
+9giHXOH0/BoMBcd8rw0lgjmOosBdM2JT0HWZIxC/Gdt7NSRA0WOlJe04u82/o3OHWDgTdm9M
+S42noSP0mvNzNALBbQnlZHU0kvt3sV1YsnrxljoIuvxKWLLwren/GVshFLPwONjw3f9Fan6G
+WxJyn/dkX3OSUGaduzcygw51vksBQiUZLCD2Tlxyr9NvkZYTqiaWW78L6regvATsLc9L/dQU
+iSMQZIK6NglmHE+cuSaoK0H4ruNKeTiQUw/EGFaLecay6Qy/s3Hk7K0QLd+gl0hZ1w1VzIeX
+Lo2BRlqnjOYFX4CwAgADmQENBFrBaNsBCADrF18KCbsZlo4NjAvVecTBCnp6WcBQJ5oSh7+E
+98jX9YznUCrNrgmeCcCMUvTDRDxfTaDJybaHugfba43nqhkbNpJ47YXsIa+YL6eEE9emSmQt
+jrSWIiY+2YJYwsDgsgckF3duqkb02OdBQlh6IbHPoXB6H//b1PgZYsomB+841XW1LSJPYlYb
+IrWfwDfQvtkFQI90r6NknVTQlpqQh5GLNWNYqRNrGQPmsB+NrUYrkl1nUt1LRGu+rCe4bSaS
+mNbwKMQKkROE4kTiB72DPk7zH4Lm0uo0YFFWG4qsMIuqEihJ/9KNX8GYBr+tWgyLooLlsdK3
+l+4dVqd8cjkJM1ExABEBAAG0QEdvb2dsZSBDbG91ZCBQYWNrYWdlcyBBdXRvbWF0aWMgU2ln
+bmluZyBLZXkgPGdjLXRlYW1AZ29vZ2xlLmNvbT6JAT4EEwECACgFAlrBaNsCGy8FCQWjmoAG
+CwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEGoDCyG6B/T78e8H/1WH2LN/nVNhm5TS1VYJ
+G8B+IW8zS4BqyozxC9iJAJqZIVHXl8g8a/Hus8RfXR7cnYHcg8sjSaJfQhqO9RbKnffiuQgG
+rqwQxuC2jBa6M/QKzejTeP0Mgi67pyrLJNWrFI71RhritQZmzTZ2PoWxfv6b+Tv5v0rPaG+u
+t1J47pn+kYgtUaKdsJz1umi6HzK6AacDf0C0CksJdKG7MOWsZcB4xeOxJYuy6NuO6KcdEz8/
+XyEUjIuIOlhYTd0hH8E/SEBbXXft7/VBQC5wNq40izPi+6WFK/e1O42DIpzQ749ogYQ1eode
+xPNhLzekKR3XhGrNXJ95r5KO10VrsLFNd8KwAgAD
+EOF
+
+ sudo apt-key add /tmp/k8s-apt-key.gpg
+
+ echo "deb http://apt.kubernetes.io/ kubernetes-$(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
+
+ sudo apt-get update
+
+ sudo apt-get -y install \
+ "kubeadm=1.14.6-*" \
+ "kubelet=1.14.6-*" \
+ "kubectl=1.14.6-*"
+
+ # enable kubectl bash completion
+ echo "source <(kubectl completion bash)" >> ~/.bashrc
+
+fi
+
+# install helm
+if [ ! -x "/usr/local/bin/helm" ]
+then
+ echo "Installing helm..."
+
+ HELM_VERSION="2.14.3"
+ HELM_SHA256SUM="38614a665859c0f01c9c1d84fa9a5027364f936814d1e47839b05327e400bf55"
+ HELM_PLATFORM="linux-amd64"
+ curl -L -o /tmp/helm.tgz "https://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-${HELM_PLATFORM}.tar.gz"
+ echo "$HELM_SHA256SUM /tmp/helm.tgz" | sha256sum -c -
+ pushd /tmp
+ tar -xzvf helm.tgz
+ sudo cp ${HELM_PLATFORM}/helm /usr/local/bin/helm
+ sudo chmod a+x /usr/local/bin/helm
+ rm -rf helm.tgz ${HELM_PLATFORM}
+ popd
+
+ helm init --client-only
+ helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
+
+ echo "source <(helm completion bash)" >> ~/.bashrc
+fi
+
+# install vagrant
+if [ ! -x "/usr/bin/vagrant" ]
+then
+ echo "Installing vagrant and associated tools..."
+
+ VAGRANT_VERSION="2.2.5"
+ VAGRANT_SHA256SUM="415f50b93235e761db284c761f6a8240a6ef6762ee3ec7ff869d2bccb1a1cdf7"
+ curl -o /tmp/vagrant.deb https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant_${VAGRANT_VERSION}_x86_64.deb
+ echo "$VAGRANT_SHA256SUM /tmp/vagrant.deb" | sha256sum -c -
+ sudo dpkg -i /tmp/vagrant.deb
+fi
+
+echo "Installing vagrant plugins if needed..."
+VAGRANT_LIBVIRT_VERSION="0.0.45"
+vagrant plugin list | grep -q vagrant-libvirt || vagrant plugin install vagrant-libvirt --plugin-version ${VAGRANT_LIBVIRT_VERSION}
+vagrant plugin list | grep -q vagrant-mutate || vagrant plugin install vagrant-mutate
+
+echo "Obtaining libvirt image of Ubuntu"
+UBUNTU_VERSION=${UBUNTU_VERSION:-bento/ubuntu-16.04}
+vagrant box list | grep "${UBUNTU_VERSION}" | grep virtualbox || vagrant box add --provider virtualbox "${UBUNTU_VERSION}"
+vagrant box list | grep "${UBUNTU_VERSION}" | grep libvirt || vagrant mutate "${UBUNTU_VERSION}" libvirt --input-provider virtualbox
+
+echo "DONE! 'cd ~/cord/automation-tools/kubespray-installer && vagrant up' to start the VMs"
+
diff --git a/kubespray-installer/setup.sh b/kubespray-installer/setup.sh
index 4f5be0a..66328fc 100755
--- a/kubespray-installer/setup.sh
+++ b/kubespray-installer/setup.sh
@@ -20,7 +20,7 @@
set -e -o pipefail
-KS_COMMIT="${KS_COMMIT:-master}"
+KS_COMMIT="${KS_COMMIT:-v2.10.4}"
install_kubespray () {
# Cleanup Old Kubespray Installations
@@ -41,7 +41,7 @@
# shellcheck disable=SC1091
source ks_venv/bin/activate
- pip3 install ansible==2.7
+ pip3 install ansible~=2.7.12 # update to newest 2.7.x version of ansible
pip3 install -r kubespray/requirements.txt
else
# shellcheck disable=SC1091
diff --git a/scripts/cloudlab-disksetup.sh b/scripts/cloudlab-disksetup.sh
index b849b1c..518b3f6 100755
--- a/scripts/cloudlab-disksetup.sh
+++ b/scripts/cloudlab-disksetup.sh
@@ -21,6 +21,13 @@
# Don't do anything if not a CloudLab node
[ ! -d /usr/local/etc/emulab ] && exit 0
+# The watchdog will sometimes reset groups, turn it off
+if [ -e /usr/local/etc/emulab/watchdog ]
+then
+ sudo /usr/bin/perl -w /usr/local/etc/emulab/watchdog stop
+ sudo mv /usr/local/etc/emulab/watchdog /usr/local/etc/emulab/watchdog-disabled
+fi
+
# Mount extra space, if haven't already
if [ ! -d /mnt/extra ]
then
@@ -57,7 +64,12 @@
fi
fi
-for DIR in docker kubelet openstack-helm nova
+# prepare for creating the libvirt/images symlink
+sudo mkdir -p /var/lib/libvirt
+sudo rm -rf /var/lib/libvirt/images
+
+# create symlinks to /mnt/extra partition
+for DIR in docker kubelet openstack-helm nova libvirt/images
do
sudo mkdir -p "/mnt/extra/$DIR"
sudo chmod -R a+rwx "/mnt/extra/$DIR"