blob: ba14607912950d815997978b7c92de4384daec52 [file] [log] [blame]
hwchiu20f35d92019-11-05 05:22:58 +00001#Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15*** Settings ***
hwchiu85695932019-12-18 08:05:25 +000016Documentation Provide the function to perform system related test
hwchiuab524f02020-02-03 23:24:19 +000017Suite Setup Common Test Suite Setup
18Test Setup Setup
19Test Teardown Teardown
hwchiuab524f02020-02-03 23:24:19 +000020Resource ../../libraries/onos.robot
21Resource ../../libraries/voltctl.robot
22Resource ../../libraries/voltha.robot
23Resource ../../libraries/utils.robot
hwchiu85695932019-12-18 08:05:25 +000024Resource ../../libraries/k8s.robot
hwchiuab524f02020-02-03 23:24:19 +000025Resource ../../variables/variables.robot
hwchiu20f35d92019-11-05 05:22:58 +000026
27*** Variables ***
hwchiu58af72d2020-01-14 00:50:35 +000028${timeout} 120s
Zack Williamsa8fe75a2020-01-10 14:25:27 -070029${desired_ETCD_cluster_size} 3
30${minimal_ETCD_cluster_size} 2
Hardik Windlass4288c6a2021-09-28 07:22:06 +000031${NAMESPACE} voltha
Matteo Scandolo10acf562021-05-03 15:35:16 -070032${INFRA_NAMESPACE} default
Andy Bavier9ed58952020-06-12 17:46:53 -070033${ETCD_resources} statefulsets
34${ETCD_name} etcd
35${ETCD_pod_label_key} app
hwchiu85695932019-12-18 08:05:25 +000036${common_pod_label_key} app
37${rwcore_pod_label_value} rw-core
38${ofagent_pod_label_value} ofagent
Andrea Campanella3dcce272021-01-15 16:04:47 +010039${adapter_openolt_pod_label_value} ${OLT_ADAPTER_APP_LABEL}
Matteo Scandolo36ac3a72021-01-05 10:18:58 -080040${teardown_device} false
41${has_dataplane} false
hwchiu20f35d92019-11-05 05:22:58 +000042
Matteo Scandolo7e519fb2021-08-13 11:35:16 -070043${suppressaddsubscriber} True
44
Andy Bavier9ed58952020-06-12 17:46:53 -070045# Per-test logging on failure is turned off by default; set this variable to enable
46${container_log_dir} ${None}
47
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070048# logging flag to enable Collect Logs, can be passed via the command line too
49# example: -v logging:False
50${logging} True
51
hwchiu20f35d92019-11-05 05:22:58 +000052*** Test Cases ***
Andy Bavier9ed58952020-06-12 17:46:53 -070053ETCD Scale Test
hwchiuab524f02020-02-03 23:24:19 +000054 [Documentation] Perform the sanity test if some ETCD endpoints crash
55 [Tags] functional bbsim
Andy Bavier9ed58952020-06-12 17:46:53 -070056 [Setup] Run Keywords Start Logging EtcdScaleTest
57 ... AND Setup
hwchiu3d9a0982020-02-06 00:19:19 +000058 [Teardown] Run Keywords Collect Logs
hwchiu3d9a0982020-02-06 00:19:19 +000059 ... AND Teardown Suite
Andy Bavier9ed58952020-06-12 17:46:53 -070060 ... AND Stop Logging EtcdScaleTest
Matteo Scandolo10acf562021-05-03 15:35:16 -070061 ${current_size}= Get ETCD Replica Count ${INFRA_NAMESPACE}
hwchiu85695932019-12-18 08:05:25 +000062 Pass Execution If '${current_size}' != '${desired_ETCD_cluster_size}'
hwchiu20f35d92019-11-05 05:22:58 +000063 ... 'Skip the test if the cluster size smaller than minimal size 3'
64 # The minimal cluster size after scale down
hwchiu85695932019-12-18 08:05:25 +000065 # based on https://github.com/ETCD-io/ETCD/blob/master/Documentation/faq.md#what-is-failure-tolerance
Matteo Scandolo10acf562021-05-03 15:35:16 -070066 Scale ETCD ${INFRA_NAMESPACE} ${minimal_ETCD_cluster_size}
hwchiu58af72d2020-01-14 00:50:35 +000067 Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo10acf562021-05-03 15:35:16 -070068 ... Validate ETCD Size ${INFRA_NAMESPACE} ${minimal_ETCD_cluster_size}
hwchiuab524f02020-02-03 23:24:19 +000069 # Perform the sanity-test
70 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
hwchiu85695932019-12-18 08:05:25 +000071 # We scale up the size to 3, the recommended size of ETCD cluster.
Matteo Scandolo10acf562021-05-03 15:35:16 -070072 Scale ETCD ${INFRA_NAMESPACE} ${desired_ETCD_cluster_size}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070073 Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo10acf562021-05-03 15:35:16 -070074 ... Validate ETCD Size ${INFRA_NAMESPACE} ${desired_ETCD_cluster_size}
Matteo Scandolo36ac3a72021-01-05 10:18:58 -080075 # Perform the sanity-test again
76 Run Keyword If ${has_dataplane} Clean Up Linux
Matteo Scandolo7e519fb2021-08-13 11:35:16 -070077 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test ${suppressaddsubscriber}
hwchiu85695932019-12-18 08:05:25 +000078
79ETCD Failure Test
80 [Documentation] Failure Scenario Test: ETCD Crash
81 [Tags] FailureTest
Andy Bavier9ed58952020-06-12 17:46:53 -070082 [Setup] Start Logging EtcdFailureTest
83 [Teardown] Run Keywords Collect Logs
84 ... AND Stop Logging EtcdFailureTest
Matteo Scandolo10acf562021-05-03 15:35:16 -070085 Delete K8s Pods By Label ${INFRA_NAMESPACE} ${ETCD_pod_label_key} ${ETCD_name}
Andy Bavier9ed58952020-06-12 17:46:53 -070086 #Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo10acf562021-05-03 15:35:16 -070087 #... Pods Do Not Exist By Label ${INFRA_NAMESPACE} ${ETCD_pod_label_key} ${ETCD_name}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070088 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass4288c6a2021-09-28 07:22:06 +000089 ... Pods Are Ready By Label ${NAMESPACE} ${common_pod_label_key} ${rwcore_pod_label_value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070090 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass4288c6a2021-09-28 07:22:06 +000091 ... Pods Are Ready By Label ${NAMESPACE} ${common_pod_label_key} ${ofagent_pod_label_value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070092 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass4288c6a2021-09-28 07:22:06 +000093 ... Pods Are Ready By Label ${NAMESPACE} ${common_pod_label_key} ${adapter_openolt_pod_label_value}
hwchiu20f35d92019-11-05 05:22:58 +000094
95*** Keywords ***
Andy Bavier9ed58952020-06-12 17:46:53 -070096Get ETCD Replica Count
hwchiu20f35d92019-11-05 05:22:58 +000097 [Arguments] ${namespace}
Andy Bavier9ed58952020-06-12 17:46:53 -070098 [Documentation] Get the number of configured ETCD nodes
hwchiu20f35d92019-11-05 05:22:58 +000099 ${rc} ${size}= Run and Return Rc and Output
Andy Bavier9ed58952020-06-12 17:46:53 -0700100 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.replicas}'
hwchiu20f35d92019-11-05 05:22:58 +0000101 Should Be Equal As Integers ${rc} 0
102 [Return] ${size}
103
hwchiu85695932019-12-18 08:05:25 +0000104Scale ETCD
hwchiu20f35d92019-11-05 05:22:58 +0000105 [Arguments] ${namespace} ${size}
hwchiu85695932019-12-18 08:05:25 +0000106 [Documentation] Scale down the number of ETCD pod
hwchiu20f35d92019-11-05 05:22:58 +0000107 ${rc}= Run and Return Rc
Andy Bavier9ed58952020-06-12 17:46:53 -0700108 ... kubectl -n ${namespace} patch ${ETCD_resources} ${ETCD_name} -p '{"spec":{"replicas": ${size}}}'
hwchiu20f35d92019-11-05 05:22:58 +0000109 Should Be Equal As Integers ${rc} 0
110
hwchiu85695932019-12-18 08:05:25 +0000111Validate ETCD Size
112 [Arguments] ${namespace} ${ETCD_cluster_size}
113 [Documentation] Scale down the number of ETCD pod
hwchiu20f35d92019-11-05 05:22:58 +0000114 ${rc} ${size}= Run and Return Rc and Output
Andy Bavier9ed58952020-06-12 17:46:53 -0700115 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.replicas}'
116 Should Be Equal As Integers ${rc} 0
117 Should Be Equal As Integers ${size} ${ETCD_cluster_size} Unexpected number of replicas
118 ${rc} ${size}= Run and Return Rc and Output
119 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.readyReplicas}'
120 Should Be Equal As Integers ${rc} 0
121 Should Be Equal As Integers ${size} ${ETCD_cluster_size} Unexpected number of ready replicas