blob: dcf47c5cd1664dcaab259b5bc4901682aa2ee561 [file] [log] [blame]
hwchiu20f35d92019-11-05 05:22:58 +00001#Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15*** Settings ***
hwchiu85695932019-12-18 08:05:25 +000016Documentation Provide the function to perform system related test
hwchiu20f35d92019-11-05 05:22:58 +000017Library OperatingSystem
hwchiu85695932019-12-18 08:05:25 +000018Resource ../../libraries/k8s.robot
hwchiu20f35d92019-11-05 05:22:58 +000019
20*** Variables ***
hwchiu58af72d2020-01-14 00:50:35 +000021${timeout} 120s
hwchiu85695932019-12-18 08:05:25 +000022${desired_ETCD_cluster_size} 3
23${minimal_ETCD_cluster_size} 2
hwchiu20f35d92019-11-05 05:22:58 +000024${namespace} voltha
hwchiu74380812020-01-08 18:05:18 +080025${ETCD_resources} etcdclusters.etcd.database.coreos.com
26${ETCD_name} voltha-etcd-cluster
27${ETCD_pod_label_key} etcd_cluster
hwchiu85695932019-12-18 08:05:25 +000028${common_pod_label_key} app
29${rwcore_pod_label_value} rw-core
30${ofagent_pod_label_value} ofagent
31${adapter_openolt_pod_label_value} adapter-open-olt
hwchiu20f35d92019-11-05 05:22:58 +000032
33*** Test Cases ***
hwchiu85695932019-12-18 08:05:25 +000034Scale Down ETCD Cluster
35 [Documentation] Scale Down the ETCD cluster to minimal size, skip test if current cluster size < 3
36 [Tags] scaledown ETCDdown
37 ${current_size}= Get ETCD Running Size voltha
38 Pass Execution If '${current_size}' != '${desired_ETCD_cluster_size}'
hwchiu20f35d92019-11-05 05:22:58 +000039 ... 'Skip the test if the cluster size smaller than minimal size 3'
40 # The minimal cluster size after scale down
hwchiu85695932019-12-18 08:05:25 +000041 # based on https://github.com/ETCD-io/ETCD/blob/master/Documentation/faq.md#what-is-failure-tolerance
42 Scale ETCD ${namespace} ${minimal_ETCD_cluster_size}
43 Wait Until Keyword Succeeds ${timeout} 2s Validate ETCD Size ${namespace} ${minimal_ETCD_cluster_size}
hwchiu58af72d2020-01-14 00:50:35 +000044 Wait Until Keyword Succeeds ${timeout} 2s
45 ... Check Expected Running Pods Number By Label ${namespace} ${ETCD_pod_label_key} ${ETCD_name} 2
hwchiu20f35d92019-11-05 05:22:58 +000046
hwchiu85695932019-12-18 08:05:25 +000047Scale Up ETCD Cluster
48 [Documentation] Recover the ETCD cluster by scaling up its size
49 [Tags] scaleup ETCDup
50 ${current_size}= Get ETCD Running Size voltha
51 Pass Execution If '${current_size}' != '${minimal_ETCD_cluster_size}'
hwchiu20f35d92019-11-05 05:22:58 +000052 ... 'Skip the test if the cluster size smaller than minimal size 3'
hwchiu85695932019-12-18 08:05:25 +000053 Scale ETCD ${namespace} ${desired_ETCD_cluster_size}
54 # We scale up the size to 3, the recommended size of ETCD cluster.
55 Wait Until Keyword Succeeds ${timeout} 2s Validate ETCD Size ${namespace} ${desired_ETCD_cluster_size}
56
57ETCD Failure Test
58 [Documentation] Failure Scenario Test: ETCD Crash
59 [Tags] FailureTest
60 Delete K8s Pods By Label ${namespace} ${ETCD_pod_label_key} ${ETCD_name}
61 Wait Until Keyword Succeeds ${timeout} 2s
62 ... Pods Does Not Exist By Label ${namespace} ${ETCD_pod_label_key} ${ETCD_name}
63 Wait Until Keyword Succeeds ${timeout} 2s
64 ... Pods Does Not Ready By Label ${namespace} ${common_pod_label_key} ${rwcore_pod_label_value}
65 Wait Until Keyword Succeeds ${timeout} 2s
66 ... Pods Does Not Ready By Label ${namespace} ${common_pod_label_key} ${ofagent_pod_label_value}
67 Wait Until Keyword Succeeds ${timeout} 2s
68 ... Pods Does Not Ready By Label ${namespace} ${common_pod_label_key} ${adapter_openolt_pod_label_value}
hwchiu20f35d92019-11-05 05:22:58 +000069
70*** Keywords ***
hwchiu85695932019-12-18 08:05:25 +000071Get ETCD Running Size
hwchiu20f35d92019-11-05 05:22:58 +000072 [Arguments] ${namespace}
hwchiu85695932019-12-18 08:05:25 +000073 [Documentation] Get the number of running ETCD nodes
hwchiu20f35d92019-11-05 05:22:58 +000074 ${rc} ${size}= Run and Return Rc and Output
hwchiu85695932019-12-18 08:05:25 +000075 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.size}'
hwchiu20f35d92019-11-05 05:22:58 +000076 Should Be Equal As Integers ${rc} 0
77 [Return] ${size}
78
hwchiu85695932019-12-18 08:05:25 +000079Scale ETCD
hwchiu20f35d92019-11-05 05:22:58 +000080 [Arguments] ${namespace} ${size}
hwchiu85695932019-12-18 08:05:25 +000081 [Documentation] Scale down the number of ETCD pod
hwchiu20f35d92019-11-05 05:22:58 +000082 ${rc}= Run and Return Rc
hwchiu85695932019-12-18 08:05:25 +000083 ... kubectl -n ${namespace} patch ${ETCD_resources} ${ETCD_name} --type='merge' -p '{"spec":{"size":${size}}}'
hwchiu20f35d92019-11-05 05:22:58 +000084 Should Be Equal As Integers ${rc} 0
85
hwchiu85695932019-12-18 08:05:25 +000086Validate ETCD Size
87 [Arguments] ${namespace} ${ETCD_cluster_size}
88 [Documentation] Scale down the number of ETCD pod
hwchiu20f35d92019-11-05 05:22:58 +000089 ${rc} ${size}= Run and Return Rc and Output
hwchiu85695932019-12-18 08:05:25 +000090 ... kubectl -n ${namespace} get ${ETCD_resources} ${ETCD_name} -o jsonpath='{.status.size}'
91 Should Be Equal As Integers ${size} ${ETCD_cluster_size}