blob: c3b67feca162cfdc7d835b5f89edb22c411b97db [file] [log] [blame]
David Bainbridge117d23e2019-09-30 20:37:51 +00001# Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
David Bainbridge117d23e2019-09-30 20:37:51 +000014# voltctl common functions
15
16*** Settings ***
17Documentation Library for various utilities
18Library SSHLibrary
David Bainbridge117d23e2019-09-30 20:37:51 +000019Library String
20Library DateTime
21Library Process
22Library Collections
23Library RequestsLibrary
24Library OperatingSystem
25
26*** Keywords ***
27Lookup Service IP
28 [Arguments] ${namespace} ${name}
David Bainbridgef81cd642019-11-20 00:14:47 +000029 [Documentation] Uses kubectl to resolve a service name to an IP
Gilles Depatie675a2062019-10-22 12:44:42 -040030 ${rc} ${ip}= Run and Return Rc and Output
31 ... kubectl get svc -n ${namespace} ${name} -o jsonpath={.spec.clusterIP}
David Bainbridge117d23e2019-09-30 20:37:51 +000032 Should Be Equal as Integers ${rc} 0
33 [Return] ${ip}
34
35Lookup Service PORT
36 [Arguments] ${namespace} ${name}
David Bainbridgef81cd642019-11-20 00:14:47 +000037 [Documentation] Uses kubectl to resolve a service name to an PORT
Gilles Depatie675a2062019-10-22 12:44:42 -040038 ${rc} ${port}= Run and Return Rc and Output
39 ... kubectl get svc -n ${namespace} ${name} -o jsonpath={.spec.ports[0].port}
David Bainbridge117d23e2019-09-30 20:37:51 +000040 Should Be Equal as Integers ${rc} 0
41 [Return] ${port}
suraj gourd64356b2019-11-07 13:26:20 +000042
43Restart Pod
44 [Arguments] ${namespace} ${name}
45 [Documentation] Uses kubectl to force delete pod
suraj gour067451d2019-11-13 11:20:13 +000046 ${rc} ${restart_pod_name}= Run and Return Rc and Output
47 ... kubectl get pods -n ${namespace} | grep ${name} | awk 'NR==1{print $1}'
suraj gourd64356b2019-11-07 13:26:20 +000048 Log ${restart_pod_name}
suraj gour067451d2019-11-13 11:20:13 +000049 Should Not Be Empty ${restart_pod_name} Unable to parse pod name
Zack Williamsa8fe75a2020-01-10 14:25:27 -070050 ${rc} ${output}= Run and Return Rc and Output
51 ... kubectl delete pod ${restart_pod_name} -n ${namespace} --grace-period=0 --force
suraj gourd64356b2019-11-07 13:26:20 +000052 Log ${output}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +000053
suraj gour1ecfae92019-12-20 15:11:40 +000054Validate Pod Status
55 [Arguments] ${pod_name} ${namespace} ${expectedStatus}
56 [Documentation] To run the kubectl command and check the status of the given pod matches the expected status
Zack Williamsa8fe75a2020-01-10 14:25:27 -070057 ${length}= Run kubectl get pod -n ${namespace} | wc -l
suraj gour1ecfae92019-12-20 15:11:40 +000058 FOR ${index} IN RANGE ${length}-1
Zack Williamsa8fe75a2020-01-10 14:25:27 -070059 ${currentPodName}= Run
60 ... kubectl get pod -n ${namespace} -o=jsonpath="{.items[${index}].status.containerStatuses[0].name}"
suraj gour1ecfae92019-12-20 15:11:40 +000061 Log Required Pod : ${pod_name}
62 Log Current Pod: ${currentPodName}
63 Run Keyword and Ignore Error Run Keyword If '${currentPodName}'=='${pod_name}' Exit For Loop
64 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -070065 ${currentStatusofPod}= Run
66 ... kubectl get pod -n ${namespace} -o=jsonpath="{.items[${index}].status.phase}"
suraj gour1ecfae92019-12-20 15:11:40 +000067 Log ${currentStatusofPod}
68 Should Contain ${currentStatusofPod} ${expectedStatus}
69
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +000070Verify All Voltha Pods For Any Error Logs
71 [Arguments] ${datetime}
72 [Documentation] This keyword checks for the error occurence in the voltha pods
73 &{errorPodDict} Create Dictionary
74 &{containerDict} Get Container Dictionary
75 FOR ${podName} IN @{PODLIST1}
76 ${containerName} Get From Dictionary ${containerDict} ${podName}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070077 ${rc} ${logOutput} Run And Return Rc And Output
78 ... kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
79 Run Keyword And Ignore Error
80 ... Run Keyword If '${logOutput}'=='${EMPTY}'
81 ... Run Keywords Log No Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +000082 ... AND Continue For Loop
83 ${errorDict} Check For Error Logs in Pod Type1 Given the Log Output ${logOutput}
84 ${returnStatusFlagList} Get Dictionary Keys ${errorDict}
85 ${returnStatusFlag} Get From List ${returnStatusFlagList} 0
Zack Williamsa8fe75a2020-01-10 14:25:27 -070086 Run Keyword And Ignore Error
87 ... Run Keyword If '${returnStatusFlag}'=='Nologfound'
88 ... Run Keywords Log No Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +000089 ... AND Continue For Loop
Zack Williamsa8fe75a2020-01-10 14:25:27 -070090 Run Keyword And Ignore Error
91 ... Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound'
92 ... Run Keywords Log Unexpected Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +000093 ... AND Set to Dictionary ${errorPodDict} ${podName} ${errorDict}
94 END
95 FOR ${podName} IN @{PODLIST2}
96 ${containerName} Get From Dictionary ${containerDict} ${podName}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070097 ${rc} ${logOutput} Run And Return Rc And Output
98 ... kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
99 Run Keyword And Ignore Error
100 ... Run Keyword If '${logOutput}'=='${EMPTY}'
101 ... Run Keywords Log No Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000102 ... AND Continue For Loop
103 ${errorDict} Check For Error Logs in Pod Type2 Given the Log Output ${logOutput}
104 ${returnStatusFlagList} Get Dictionary Keys ${errorDict}
105 ${returnStatusFlag} Get From List ${returnStatusFlagList} 0
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700106 Run Keyword And Ignore Error
107 ... Run Keyword If '${returnStatusFlag}'=='Nologfound'
108 ... Run Keywords Log No Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000109 ... AND Continue For Loop
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700110 Run Keyword And Ignore Error
111 ... Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound'
112 ... Run Keywords Log Unexpected Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000113 ... AND Set to Dictionary ${errorPodDict} ${podName} ${errorDict}
114 END
115 Print to Console Error Statement logged in the following pods : ${errorPodDict}
116 [Return] ${errorPodDict}
117
118Check For Error Logs in Pod Type1 Given the Log Output
119 [Arguments] ${logOutput} ${logLevel}=error ${errorMessage}=${EMPTY}
120 [Documentation] Checks for error message in the particular list of pods
121 Log ${logOutput}
122 ${linesContainingLog} = Get Lines Matching Regexp ${logOutput} .*\s\${logLevel}.* partial_match=true
123 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingLog}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700124 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
125 ... Nologfound '${is_exec_status}'=='FAIL' Errorlogfound
126 ${linesContainingError} = Get Lines Matching Regexp
127 ... ${logOutput} .*\s\${logLevel}.*${errorMessage} partial_match=true
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000128 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingError}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700129 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
130 ... UnexpectedErrorfound '${is_exec_status}'=='FAIL' MatchingErrorlogfound
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000131 Log {linesContainingError}
132 &{errorDict} Create Dictionary ${returnStatusFlag} ${linesContainingLog}
133 [Return] ${errorDict}
134
135Check For Error Logs in Pod Type2 Given the Log Output
136 [Arguments] ${logOutput} ${logLevel}=warn ${errorMessage}=${EMPTY}
137 [Documentation] Checks for error message in the particular set of pods
138 Log ${logOutput}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700139 ${linesContainingLog} = Get Lines Matching Regexp
140 ... ${logOutput} .*?\s.*level.*${logLevel}.* partial_match=true
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000141 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingLog}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700142 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
143 ... Nologfound '${is_exec_status}'=='FAIL' Errorlogfound
144 ${linesContainingError} = Get Lines Matching Regexp
145 ... ${logOutput} .*?\s.*level.*${logLevel}.*msg.*${errorMessage} partial_match=true
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000146 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingError}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700147 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
148 ... UnexpectedErrorfound '${is_exec_status}'=='FAIL' MatchingErrorlogfound
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000149 Log {linesContainingError}
150 &{errorDict} Create Dictionary ${returnStatusFlag} ${linesContainingLog}
151 [Return] ${errorDict}
152
153Get Container Dictionary
154 [Documentation] Creates a mapping for pod name and container name and returns the same
155 &{containerDict} Create Dictionary
156 ${containerName} Set Variable ${EMPTY}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700157 ${podName} Run kubectl get deployment -n voltha | awk 'NR>1 {print $1}'
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000158 @{podNameList}= Split To Lines ${podName}
159 Append To List ${podNameList} voltha-etcd-cluster voltha-kafka voltha-ro-core voltha-zookeeper
160 Log ${podNameList}
161 #Creatiing dictionary to correspond pod name and container name
162 FOR ${pod} IN @{podNameList}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700163 ${containerName} Run kubectl get pod -n voltha | grep ${pod} | awk '{print $1}'
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000164 &{containerDict} Set To Dictionary ${containerDict} ${pod} ${containerName}
165 END
166 Log ${containerDict}
167 [Return] ${containerDict}
168
169Validate Error For Given Pods
170 [Arguments] ${datetime} ${podDict}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700171 [Documentation]
172 ... This keyword is used to get the list of pods if there is any unexpected error
173 ... in a particular pod(s) given the time-${datetime} from which the log needs to
174 ... be analysed and the dictionary of pods and the error in the dictionary format
175 ... ${podDict] .
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000176 ...
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700177 ... Usage: ${returnStatusFlag} Validate Error For Given Pods ${datetime} ${podDict}
178 ...
179 ... Arguments:
180 ...
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000181 ... ${datetime} = time from which the log needs to be taken
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700182 ... ${podDict} = Key-value pair of the pod name and the error msg
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000183 ...
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700184 ... Example: ${podDict} = Set Dictionary ${podDict} radius sample error message.
185 ...
186 ... In case the radius pod log has any other error than the expected
187 ... error, then the podname will be returned
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000188 ${podList} = Get Dictionary Keys ${podDict}
189 FOR ${podName} IN @{podList}
190 ${containerName} Get From Dictionary ${containerDict} ${podName}
191 ${expectedError} Get From Dictionary ${podDict} ${podName}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700192 ${rc} ${logOutput} Run And Return Rc And Output
193 ... kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
194 Run Keyword And Ignore Error
195 ... Run Keyword If '${logOutput}'=='${EMPTY}'
196 ... Run Keywords Log No Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000197 ... AND Continue For Loop
198 ${returnStatusFlag} Check For Error Logs in Pod Type1 Given the Log Output ${logOutput}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700199 Run Keyword And Ignore Error
200 ... Run Keyword If '${returnStatusFlag}'=='Nologfound'
201 ... Run Keywords Log No Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000202 ... AND Continue For Loop
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700203 Run Keyword And Ignore Error
204 ... Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound'
205 ... Run Keywords Log Unexpected Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000206 ... AND Append To List ${errorPodList} ${podName}
207 END
208 [Return] ${errorPodList}
209
David Bainbridgef81cd642019-11-20 00:14:47 +0000210Delete K8s Pod
211 [Arguments] ${namespace} ${name}
212 [Documentation] Uses kubectl to delete a named POD
213 ${rc} Run and Return Rc
214 ... kubectl delete -n ${namespace} pod/${name}
215 Should Be Equal as Integers ${rc} 0
216
hwchiu85695932019-12-18 08:05:25 +0000217Delete K8s Pods By Label
218 [Arguments] ${namespace} ${key} ${value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700219 [Documentation] Uses kubectl to delete a PODs, filtering by label
hwchiu85695932019-12-18 08:05:25 +0000220 ${rc}= Run and Return Rc
221 ... kubectl -n ${namespace} delete pods -l${key}=${value}
222 Should Be Equal as Integers ${rc} 0
223
David Bainbridgef81cd642019-11-20 00:14:47 +0000224Scale K8s Deployment
225 [Arguments] ${namespace} ${name} ${count}
226 [Documentation] Uses kubectl to scale a named deployment
227 ${rc} Run and Return Rc
228 ... kubectl scale --replicas=${count} -n ${namespace} deploy/${name}
229 Should Be Equal as Integers ${rc} 0
230
231Pod Exists
232 [Arguments] ${namespace} ${name}
233 [Documentation] Succeeds it the named POD exists
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700234 ${rc} ${count} Run and Return Rc
235 ... kubectl get -n ${namespace} pod -o json | jq -r ".items[].metadata.name" | grep ${name}
David Bainbridgef81cd642019-11-20 00:14:47 +0000236 Should Be True ${count}>0
237
238Pod Does Not Exist
239 [Arguments] ${namespace} ${name}
240 [Documentation] Succeeds if the named POD does not exist
241 ${rc} ${count} Run and Return Rc And Output
242 ... kubectl get -n ${namespace} pod -o json | jq -r ".items[].metadata.name" | grep -c ${name}
243 Should Be Equal As Integers ${count} 0
244 Should Be True ${count}==0
245
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700246Pods Do Not Exist By Label
hwchiu85695932019-12-18 08:05:25 +0000247 [Arguments] ${namespace} ${key} ${value}
248 [Documentation] Succeeds if the named POD does not exist
249 ${rc} ${count} Run and Return Rc And Output
250 ... kubectl get -n ${namespace} pod -l${key}=${value} -o json | jq -r ".items[].metadata.name" | wc -l
251 Should Be Equal As Integers ${count} 0
252 Should Be True ${count}==0
253
David Bainbridgef81cd642019-11-20 00:14:47 +0000254Get Available Deployment Replicas
255 [Arguments] ${namespace} ${name}
256 [Documentation] Succeeds if the named POD exists and has a ready count > 0
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700257 ${rc} ${count} Run and Return Rc and Output
258 ... kubectl get -n ${namespace} deploy/${name} -o jsonpath='{.status.availableReplicas}'
David Bainbridgef81cd642019-11-20 00:14:47 +0000259 ${result}= Run Keyword If '${count}' == '' Set Variable 0
260 ... ELSE Set Variable ${count}
261 [Return] ${result}
262
263Check Expected Available Deployment Replicas
264 [Arguments] ${namespace} ${name} ${expected}
265 [Documentation] Succeeds if the named deployment has the expected number of available replicas
266 ${count}= Get Available Deployment Replicas ${namespace} ${name}
267 Should Be Equal As Integers ${expected} ${count}
268
269Get Deployment Replica Count
270 [Arguments] ${namespace} ${name}
271 [Documentation] Uses kubectl to fetch the number of configured replicas on a deployment
272 ${rc} ${value} Run and Return Rc and Output
273 ... kubectl -n ${namespace} get deploy/${name} -o 'jsonpath={.status.replicas}'
274 Should Be Equal as Integers ${rc} 0
275 ${replicas}= Run Keyword If '${value}' == '' Set Variable 0
276 ... ELSE Set Variable ${value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700277 [Return] ${replicas}
David Bainbridgef81cd642019-11-20 00:14:47 +0000278
279Does Deployment Have Replicas
280 [Arguments] ${namespace} ${name} ${expected_count}
281 [Documentation] Uses kubectl to fetch the number of configured replicas on a deployment
282 ${rc} ${value} Run and Return Rc and Output
283 ... kubectl -n ${namespace} get deploy/${name} -o 'jsonpath={.status.replicas}'
284 Should Be Equal as Integers ${rc} 0
285 ${replicas}= Run Keyword If '${value}' == '' Set Variable 0
286 ... ELSE Set Variable ${value}
287 Should be Equal as Integers ${replicas} ${expected_count}
hwchiu85695932019-12-18 08:05:25 +0000288
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700289Pods Are Ready By Label
hwchiu85695932019-12-18 08:05:25 +0000290 [Arguments] ${namespace} ${key} ${value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700291 [Documentation] Check that all pods with a label are ready
292 ${output}= Run
293 ... kubectl -n ${namespace} get pods -l ${key}=${value} -o=jsonpath="{.items[].status.containerStatuses[].ready}"
294 Should Not Contain ${output} "false"
Gayathri.Selvan49398962020-01-13 07:19:12 +0000295
hwchiu58af72d2020-01-14 00:50:35 +0000296Check Expected Running Pods Number By Label
297 [Arguments] ${namespace} ${key} ${value} ${number}
298 [Documentation] Succeeds if the desired pod has expected number replicas
299 ${rc} ${count} Run and Return Rc and Output
300 ... kubectl -n ${namespace} get pods -l ${key}=${value} -o json | jq -r ".items[].status.phase" | wc -l
301 Should Be Equal as Integers ${count} ${number}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000302
303Reboot ONU
304 [Arguments] ${onu_id} ${src} ${dst}
305 [Documentation] Using voltctl command reboot ONU and verify that ONU comes up to running state
306 ${rc} ${devices}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device reboot ${onu_id}
307 Should Be Equal As Integers ${rc} 0
308 Run Keyword and Ignore Error Wait Until Keyword Succeeds 30 1s Validate Device
309 ... ENABLED ACTIVATING UNREACHABLE ${onu_id} onu=True
310