Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 1 | # Copyright 2017-present Radisys Corporation |
| 2 | # |
| 3 | # Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | # you may not use this file except in compliance with the License. |
| 5 | # You may obtain a copy of the License at |
| 6 | # |
| 7 | # http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | # |
| 9 | # Unless required by applicable law or agreed to in writing, software |
| 10 | # distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | # See the License for the specific language governing permissions and |
| 13 | # limitations under the License. |
| 14 | |
| 15 | |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 16 | *** Settings *** |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 17 | Documentation Test suite for checking default maas,xos and onos containers and fabric switch default services and maas cli commands |
| 18 | Library OperatingSystem |
| 19 | Library ../cord-api/Framework/utils/onosUtils.py |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 20 | Library ../cord-api/Framework/utils/utils.py |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 21 | Resource ../cord-api/Framework/utils/utils.robot |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 22 | |
| 23 | *** Variables *** |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 24 | @{MAAS_SERVICE_STATUS} start/running is running |
| 25 | @{JUJU_SERVICE_STATUS} active is ready unknown |
| 26 | @{LXD_CONTAINER_STATUS} RUNNING |
| 27 | @{BOOT_RESOURCES_OUTPUT} ubuntu/trusty |
| 28 | ${FABRIC_SWITCH_PROMPT} \# |
| 29 | ${FABRIC_SWITCH_USER} root |
| 30 | ${FABRIC_SWITCH_PASSWD} onl |
| 31 | @{FABRIC_SERVICE_STATUS} is running |
| 32 | ${IP_PATTERN} (\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+) |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 33 | ${PUBLIC_IFACE} eth2 |
| 34 | ${NUM_OF_SWITCHES} 4 |
| 35 | ${CORD_PROFILE} rcord |
You Wang | 046fa86 | 2017-11-21 13:13:26 -0800 | [diff] [blame] | 36 | ${FABRIC} on |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 37 | ${DOCKER_CONTAINERS_FILE} ${CURDIR}/../diag/dockerContainers.json |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 38 | |
| 39 | *** Test Cases *** |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 40 | Verify Headnode Interfaces |
You Wang | be30ffe | 2017-12-14 16:16:50 -0800 | [diff] [blame] | 41 | [Tags] fabric notready |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 42 | [Documentation] Verifies the headnode interface is up and has external connectivity |
| 43 | Verify HeadNode Interfaces Detected |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 44 | Test Ping ${PUBLIC_IFACE} www.opennetworking.org |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 45 | |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 46 | Get Compute Node and Fabric Info |
| 47 | [Documentation] Get all information pretaining to the compute nodes and fabric |
| 48 | ${nodes}= Create List |
| 49 | ${hostnames}= Create List |
| 50 | ${hostname_prefixes}= Create List |
| 51 | ${node_ips}= Create List |
| 52 | ${node_data_ips}= Create List |
| 53 | ${node_count} Run cord prov list | grep node | wc -l |
| 54 | ${node_count}= Convert To Integer ${node_count} |
| 55 | Log ${node_count} |
| 56 | ##Get hostname |
| 57 | : FOR ${INDEX} IN RANGE 1 ${node_count}+1 |
| 58 | \ ${hostname}= Run cord prov list | grep node | awk '{print $2}' | sed -n ${INDEX}p |
| 59 | \ Append To List ${hostnames} ${hostname} |
| 60 | ##Get hostname prefixes |
| 61 | : FOR ${INDEX} IN RANGE 0 ${node_count} |
| 62 | \ ${hostname_prefix}= Remove String ${hostnames[${INDEX}]} .cord.lab |
| 63 | \ Append To List ${hostname_prefixes} ${hostname_prefix} |
| 64 | ##Get compute node data ips |
| 65 | ${cordvtnnodes}= ONOS Command Execute onos-cord 8102 cordvtn-nodes | grep fabric |
| 66 | ${nds}= Split To Lines ${cordvtnnodes} |
| 67 | : FOR ${i} IN @{nds} |
| 68 | \ ${data_ip}= Get Compute Node IP ${i} |
| 69 | \ Append To List ${node_data_ips} ${data_ip} |
| 70 | ##Get compute node ips |
| 71 | : FOR ${i} IN @{hostname_prefixes} |
| 72 | \ ${node_ip}= Run cord harvest list | grep ${i} | awk '{print $4}' |
| 73 | \ Append To List ${node_ips} ${node_ip} |
| 74 | @{switch_ips}= Discover FABRIC IPs |
| 75 | Set Suite Variable ${switch_ips} |
| 76 | Set Suite Variable ${hostnames} |
| 77 | Set Suite Variable ${hostname_prefixes} |
| 78 | Set Suite Variable ${node_ips} |
| 79 | Set Suite Variable ${node_data_ips} |
| 80 | |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 81 | Verify Compute Nodes Pingability Through Fabric |
| 82 | [Documentation] Verifies that the two compute nodes can ping each other through the fabric |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 83 | [Tags] fabric |
| 84 | ##Verify pingablilty across compute nodes |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 85 | : FOR ${src} IN @{hostname_prefixes} |
| 86 | \ Ping All Compute Nodes Through Fabric ${src} |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 87 | |
| 88 | Verify Compute Nodes to Fabric Pingability |
| 89 | [Documentation] Verifies that the two compute nodes can ping the switches |
| 90 | [Tags] fabric |
| 91 | ##Verify pingability from compute nodes to fabric |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 92 | : FOR ${src} IN @{hostname_prefixes} |
| 93 | \ Ping All Fabric Switches ${src} |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 94 | |
| 95 | Verify CordVTN Nodes |
| 96 | [Documentation] Verifies that the cordvtn app running in onos identifies the nodes and devices (fabric) |
| 97 | ${nodes}= Execute ONOS Command onos-cord 8102 cordvtn-nodes |
| 98 | : FOR ${i} IN @{node_ips} |
| 99 | \ ${node_1}= Get Lines Containing String ${nodes} ${i} |
You Wang | 046fa86 | 2017-11-21 13:13:26 -0800 | [diff] [blame] | 100 | \ Run Keyword If "${FABRIC}" == "on" Verify CordVTN Node ${node_1} COMPLETE ${i} |
| 101 | \ Run Keyword If "${FABRIC}" == "off" Verify CordVTN Node ${node_1} DEVICE_CREATED ${i} |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 102 | ${ports}= Execute ONOS Command onos-cord 8102 cordvtn-ports |
| 103 | ${devices}= Execute ONOS Command onos-fabric 8101 devices |
| 104 | @{switch_ips}= Discover FABRIC IPs |
| 105 | : FOR ${i} IN @{switch_ips} |
| 106 | \ Should Contain ${devices} ${i} |
| 107 | |
| 108 | Verify MAAS Service State |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 109 | [Template] Verify MAAS Service |
| 110 | maas-dhcpd |
| 111 | maas-regiond |
| 112 | maas-clusterd |
| 113 | maas-proxy |
| 114 | bind9 |
| 115 | |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 116 | Verify Docker Containers State |
| 117 | ${dockerContainers} utils.jsonToList ${DOCKER_CONTAINERS_FILE} docker-containers-${CORD_PROFILE} |
| 118 | : FOR ${container} IN @{dockerContainers} |
Kailash Khalasi | 055904b | 2018-02-20 09:48:21 -0800 | [diff] [blame] | 119 | \ Run Keyword And Continue On Failure Verify Containers ${container} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 120 | |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 121 | Verify Juju Services State |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 122 | [Template] Verify JUJU Service |
| 123 | ceilometer |
| 124 | ceilometer-agent |
| 125 | glance |
| 126 | keystone |
| 127 | mongodb |
| 128 | nagios |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 129 | neutron-api |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 130 | nova-cloud-controller |
| 131 | nova-compute |
| 132 | openstack-dashboard |
| 133 | percona-cluster |
| 134 | rabbitmq-server |
| 135 | |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 136 | Verify Openstack LXD Containers State |
| 137 | [Template] Verify Openstack LXD Containers |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 138 | ceilometer |
| 139 | glance |
| 140 | keystone |
| 141 | mongodb |
| 142 | nagios |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 143 | neutron-api |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 144 | nova-cloud-controller |
| 145 | openstack-dashboard |
| 146 | percona-cluster |
| 147 | rabbitmq-server |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 148 | |
| 149 | Verify MAAS CLI commands |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 150 | [Tags] notready |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 151 | Login MAAS Server |
| 152 | Verify MAAS CLI Commands boot-resources read | jq 'map(select(.type == "Synced"))' ubuntu/trusty |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 153 | Verify MAAS CLI Commands devices list | jq '.' | jq '.[]'.hostname | wc -l ${NUM_OF_SWITCHES} |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 154 | #Verify MAAS CLI Commands events query | jq '.' | jq .events[].id | wc -l 100 |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 155 | Verify MAAS CLI Commands fabrics read | jq '.' | jq .[].name | wc -l 4 |
| 156 | Verify MAAS CLI Commands networks read | jq '.' | jq .[].name | wc -l 4 |
| 157 | Verify MAAS CLI Commands node-groups list | jq '.' | jq .[].status | wc -l 1 |
| 158 | Verify MAAS CLI Commands subnets read | jq '.' | jq .[].name | wc -l 4 |
| 159 | Verify MAAS CLI Commands nodes list | jq '.' | jq .[].substatus_name | wc -l 1 |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 160 | Verify MAAS CLI Commands zones read | jq '.' | jq .[].name | wc -l 2 |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 161 | Logout MAAS Server |
| 162 | |
| 163 | Verify Fabric Switch Service |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 164 | [Tags] fabric |
| 165 | @{switch_ips}= Discover FABRIC IPs |
| 166 | : FOR ${i} IN @{switch_ips} |
| 167 | \ Verify Fabric Switch Service ${i} faultd |
| 168 | \ Verify Fabric Switch Service ${i} netplug |
| 169 | \ Verify Fabric Switch Service ${i} onlp-snmpd |
| 170 | \ Verify Fabric Switch Service ${i} onlpd |
| 171 | \ Verify Fabric Switch Service ${i} rsyslog |
| 172 | \ Verify Fabric Switch Service ${i} snmpd |
| 173 | \ Verify Fabric Switch Service ${i} ssh |
| 174 | \ Verify Fabric Switch Service ${i} udev |
| 175 | \ Verify Fabric Switch Service ${i} watchdog |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 176 | |
| 177 | *** Keywords *** |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 178 | Verify HeadNode Interfaces Detected |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 179 | ${cmd}= Catenate SEPARATOR=| sudo ethtool mgmtbr grep 'Link detected:' awk '{ print $3 }' |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 180 | ${output}= Run ${cmd} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 181 | Should Contain ${output} yes msg= mgmtbr is not detected !!!. Reason: |
| 182 | ${cmd}= Catenate SEPARATOR=| sudo ethtool fabric grep 'Link detected:' awk '{ print $3 }' |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 183 | ${output}= Run ${cmd} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 184 | Should Contain ${output} yes msg= fabric interface is not detected !!!. Reason: |
| 185 | |
You Wang | 046fa86 | 2017-11-21 13:13:26 -0800 | [diff] [blame] | 186 | Verify CordVTN Node |
| 187 | [Arguments] ${node} ${status} ${ip} |
| 188 | Should Contain ${node} ${status} |
| 189 | Should Contain ${node} ${ip} |
| 190 | |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 191 | Verify Containers |
| 192 | [Arguments] ${name} |
| 193 | ${container_id}= Get Docker Container ID ${name} |
| 194 | ${output}= Run docker inspect --format="{{ .State.Running }}" ${container_id} |
| 195 | Should Contain ${output} true msg=${name} is not running !!!. Reason: |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 196 | |
| 197 | Verify MAAS Service |
| 198 | [Arguments] ${name} |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 199 | ${cmd}= Catenate sudo service ${name} status |
| 200 | ${output}= Run ${cmd} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 201 | Should Contain Any ${output} @{MAAS_SERVICE_STATUS} msg= ${name} is not running !!!. Reason: |
| 202 | |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 203 | Verify JUJU Service |
| 204 | [Arguments] ${name} |
| 205 | ${cmd} Catenate SEPARATOR=| juju status --format=tabular grep -v grep grep ${name}/0 awk '{ print $2,$7,$8,$9,$10}' |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 206 | ${output}= Run ${cmd} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 207 | Should Contain Any ${output} @{JUJU_SERVICE_STATUS} msg= ${name} is not running !!!. Reason: |
| 208 | |
| 209 | Verify Openstack LXD Containers |
| 210 | [Arguments] ${name} |
| 211 | ${cmd} Catenate SEPARATOR=| sudo lxc list grep -v grep grep ${name}-1 awk '{ print $2,$4 }' |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 212 | ${output}= Run ${cmd} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 213 | Should Contain Any ${output} @{LXD_CONTAINER_STATUS} msg= ${name} is not running !!!. Reason: |
| 214 | |
| 215 | Verify MAAS CLI Commands |
| 216 | [Arguments] ${name} ${expected} |
| 217 | ${cmd} Catenate maas cord ${name} |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 218 | ${output}= Run ${cmd} |
| 219 | Should Contain ${output} ${expected} msg=Reason: |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 220 | |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 221 | Login MAAS Server |
| 222 | ${cmd} Catenate maas login cord http://localhost/MAAS/api/1.0 $(sudo maas-region-admin apikey --user=cord) |
| 223 | ${output}= Run ${cmd} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 224 | Should Contain ${output} You are now logged in to the MAAS msg= MAAS login failure !!!. Reason: |
| 225 | |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 226 | Logout MAAS Server |
| 227 | ${cmd} Catenate maas logout cord |
| 228 | ${output}= Run ${cmd} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 229 | |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 230 | Discover FABRIC IPs |
| 231 | ${switches}= Run cord prov list | grep fabric | awk '{print $4}' |
| 232 | @{switch_ips}= Split To Lines ${switches} |
| 233 | [Return] ${switch_ips} |
Gunaseelan | ed0eb1b | 2017-08-08 08:12:12 -0500 | [diff] [blame] | 234 | |
| 235 | Verify Fabric Switch Service |
| 236 | [Arguments] ${ip} ${name} |
Kailash Khalasi | 2f567a4 | 2017-09-27 13:50:05 -0700 | [diff] [blame] | 237 | ${cmd}= Catenate service ${name} status |
| 238 | ${output}= Run Command On Remote System ${ip} ${cmd} ${FABRIC_SWITCH_USER} ${FABRIC_SWITCH_PASSWD} ${FABRIC_SWITCH_PROMPT} 60s False |
You Wang | e920610 | 2017-11-17 12:15:11 -0800 | [diff] [blame] | 239 | Should Contain Any ${output} @{FABRIC_SERVICE_STATUS} msg= ${name} is not running !!!. Reason: |
| 240 | |
| 241 | Ping All Compute Nodes Through Fabric |
| 242 | [Arguments] ${src_ip} |
| 243 | : FOR ${dst_ip} IN @{node_data_ips} |
| 244 | \ Verify Ping ubuntu ${src_ip} ${dst_ip} |
| 245 | |
| 246 | Ping All Fabric Switches |
| 247 | [Arguments] ${src_ip} |
| 248 | : FOR ${dst_ip} IN @{switch_ips} |
| 249 | \ Verify Ping ubuntu ${src_ip} ${dst_ip} |
| 250 | |
| 251 | Verify Ping |
| 252 | [Arguments] ${srcName} ${srcIP} ${dst} |
| 253 | ${result}= Run ssh ${srcName}@${srcIP} "ping -c 3 ${dst}" |
| 254 | Should Contain ${result} 64 bytes |
| 255 | Should Not Contain ${result} Destination Host Unreachable |