blob: 13c55bc7be3c35694dd362eebe255cfa10beeab3 [file] [log] [blame]
hwchiu20f35d92019-11-05 05:22:58 +00001#Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15*** Settings ***
hwchiu85695932019-12-18 08:05:25 +000016Documentation Provide the function to perform system related test
hwchiuab524f02020-02-03 23:24:19 +000017Suite Setup Common Test Suite Setup
18Test Setup Setup
19Test Teardown Teardown
hwchiuab524f02020-02-03 23:24:19 +000020Resource ../../libraries/onos.robot
21Resource ../../libraries/voltctl.robot
22Resource ../../libraries/voltha.robot
23Resource ../../libraries/utils.robot
hwchiu85695932019-12-18 08:05:25 +000024Resource ../../libraries/k8s.robot
hwchiuab524f02020-02-03 23:24:19 +000025Resource ../../variables/variables.robot
hwchiu20f35d92019-11-05 05:22:58 +000026
27*** Variables ***
hwchiu58af72d2020-01-14 00:50:35 +000028${timeout} 120s
Zack Williamsa8fe75a2020-01-10 14:25:27 -070029${desired_ETCD_cluster_size} 3
30${minimal_ETCD_cluster_size} 2
31${namespace} voltha
Andy Bavier9ed58952020-06-12 17:46:53 -070032${ETCD_namespace} default
33${ETCD_resources} statefulsets
34${ETCD_name} etcd
35${ETCD_pod_label_key} app
hwchiu85695932019-12-18 08:05:25 +000036${common_pod_label_key} app
37${rwcore_pod_label_value} rw-core
38${ofagent_pod_label_value} ofagent
39${adapter_openolt_pod_label_value} adapter-open-olt
Matteo Scandolo36ac3a72021-01-05 10:18:58 -080040${teardown_device} false
41${has_dataplane} false
hwchiu20f35d92019-11-05 05:22:58 +000042
Andy Bavier9ed58952020-06-12 17:46:53 -070043# Per-test logging on failure is turned off by default; set this variable to enable
44${container_log_dir} ${None}
45
hwchiu20f35d92019-11-05 05:22:58 +000046*** Test Cases ***
Andy Bavier9ed58952020-06-12 17:46:53 -070047ETCD Scale Test
hwchiuab524f02020-02-03 23:24:19 +000048 [Documentation] Perform the sanity test if some ETCD endpoints crash
49 [Tags] functional bbsim
Andy Bavier9ed58952020-06-12 17:46:53 -070050 [Setup] Run Keywords Start Logging EtcdScaleTest
51 ... AND Setup
hwchiu3d9a0982020-02-06 00:19:19 +000052 [Teardown] Run Keywords Collect Logs
hwchiu3d9a0982020-02-06 00:19:19 +000053 ... AND Teardown Suite
Andy Bavier9ed58952020-06-12 17:46:53 -070054 ... AND Stop Logging EtcdScaleTest
55 ${current_size}= Get ETCD Replica Count ${ETCD_namespace}
hwchiu85695932019-12-18 08:05:25 +000056 Pass Execution If '${current_size}' != '${desired_ETCD_cluster_size}'
hwchiu20f35d92019-11-05 05:22:58 +000057 ... 'Skip the test if the cluster size smaller than minimal size 3'
58 # The minimal cluster size after scale down
hwchiu85695932019-12-18 08:05:25 +000059 # based on https://github.com/ETCD-io/ETCD/blob/master/Documentation/faq.md#what-is-failure-tolerance
Andy Bavier9ed58952020-06-12 17:46:53 -070060 Scale ETCD ${ETCD_namespace} ${minimal_ETCD_cluster_size}
hwchiu58af72d2020-01-14 00:50:35 +000061 Wait Until Keyword Succeeds ${timeout} 2s
Andy Bavier9ed58952020-06-12 17:46:53 -070062 ... Validate ETCD Size ${ETCD_namespace} ${minimal_ETCD_cluster_size}
hwchiuab524f02020-02-03 23:24:19 +000063 # Perform the sanity-test
64 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
hwchiu85695932019-12-18 08:05:25 +000065 # We scale up the size to 3, the recommended size of ETCD cluster.
Andy Bavier9ed58952020-06-12 17:46:53 -070066 Scale ETCD ${ETCD_namespace} ${desired_ETCD_cluster_size}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070067 Wait Until Keyword Succeeds ${timeout} 2s
Andy Bavier9ed58952020-06-12 17:46:53 -070068 ... Validate ETCD Size ${ETCD_namespace} ${desired_ETCD_cluster_size}
Matteo Scandolo36ac3a72021-01-05 10:18:58 -080069 # Perform the sanity-test again
70 Run Keyword If ${has_dataplane} Clean Up Linux
71 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
hwchiu85695932019-12-18 08:05:25 +000072
73ETCD Failure Test
74 [Documentation] Failure Scenario Test: ETCD Crash
75 [Tags] FailureTest
Andy Bavier9ed58952020-06-12 17:46:53 -070076 [Setup] Start Logging EtcdFailureTest
77 [Teardown] Run Keywords Collect Logs
78 ... AND Stop Logging EtcdFailureTest
79 Delete K8s Pods By Label ${ETCD_namespace} ${ETCD_pod_label_key} ${ETCD_name}
80 #Wait Until Keyword Succeeds ${timeout} 2s
81 #... Pods Do Not Exist By Label ${ETCD_namespace} ${ETCD_pod_label_key} ${ETCD_name}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070082 Wait Until Keyword Succeeds ${timeout} 2s
83 ... Pods Are Ready By Label ${namespace} ${common_pod_label_key} ${rwcore_pod_label_value}
84 Wait Until Keyword Succeeds ${timeout} 2s
85 ... Pods Are Ready By Label ${namespace} ${common_pod_label_key} ${ofagent_pod_label_value}
86 Wait Until Keyword Succeeds ${timeout} 2s
87 ... Pods Are Ready By Label ${namespace} ${common_pod_label_key} ${adapter_openolt_pod_label_value}
hwchiu20f35d92019-11-05 05:22:58 +000088
89*** Keywords ***
Andy Bavier9ed58952020-06-12 17:46:53 -070090Get ETCD Replica Count
hwchiu20f35d92019-11-05 05:22:58 +000091 [Arguments] ${namespace}
Andy Bavier9ed58952020-06-12 17:46:53 -070092 [Documentation] Get the number of configured ETCD nodes
hwchiu20f35d92019-11-05 05:22:58 +000093 ${rc} ${size}= Run and Return Rc and Output
Andy Bavier9ed58952020-06-12 17:46:53 -070094 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.replicas}'
hwchiu20f35d92019-11-05 05:22:58 +000095 Should Be Equal As Integers ${rc} 0
96 [Return] ${size}
97
hwchiu85695932019-12-18 08:05:25 +000098Scale ETCD
hwchiu20f35d92019-11-05 05:22:58 +000099 [Arguments] ${namespace} ${size}
hwchiu85695932019-12-18 08:05:25 +0000100 [Documentation] Scale down the number of ETCD pod
hwchiu20f35d92019-11-05 05:22:58 +0000101 ${rc}= Run and Return Rc
Andy Bavier9ed58952020-06-12 17:46:53 -0700102 ... kubectl -n ${namespace} patch ${ETCD_resources} ${ETCD_name} -p '{"spec":{"replicas": ${size}}}'
hwchiu20f35d92019-11-05 05:22:58 +0000103 Should Be Equal As Integers ${rc} 0
104
hwchiu85695932019-12-18 08:05:25 +0000105Validate ETCD Size
106 [Arguments] ${namespace} ${ETCD_cluster_size}
107 [Documentation] Scale down the number of ETCD pod
hwchiu20f35d92019-11-05 05:22:58 +0000108 ${rc} ${size}= Run and Return Rc and Output
Andy Bavier9ed58952020-06-12 17:46:53 -0700109 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.replicas}'
110 Should Be Equal As Integers ${rc} 0
111 Should Be Equal As Integers ${size} ${ETCD_cluster_size} Unexpected number of replicas
112 ${rc} ${size}= Run and Return Rc and Output
113 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.readyReplicas}'
114 Should Be Equal As Integers ${rc} 0
115 Should Be Equal As Integers ${size} ${ETCD_cluster_size} Unexpected number of ready replicas