blob: 9a211902be8f1274b3f4e1807ea9239acc66652e [file] [log] [blame]
David Bainbridge117d23e2019-09-30 20:37:51 +00001# Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
David Bainbridge117d23e2019-09-30 20:37:51 +000014# voltctl common functions
15
16*** Settings ***
17Documentation Library for various utilities
18Library SSHLibrary
David Bainbridge117d23e2019-09-30 20:37:51 +000019Library String
20Library DateTime
21Library Process
22Library Collections
23Library RequestsLibrary
24Library OperatingSystem
25
26*** Keywords ***
27Lookup Service IP
28 [Arguments] ${namespace} ${name}
David Bainbridgef81cd642019-11-20 00:14:47 +000029 [Documentation] Uses kubectl to resolve a service name to an IP
Gilles Depatie675a2062019-10-22 12:44:42 -040030 ${rc} ${ip}= Run and Return Rc and Output
31 ... kubectl get svc -n ${namespace} ${name} -o jsonpath={.spec.clusterIP}
David Bainbridge117d23e2019-09-30 20:37:51 +000032 Should Be Equal as Integers ${rc} 0
33 [Return] ${ip}
34
35Lookup Service PORT
36 [Arguments] ${namespace} ${name}
David Bainbridgef81cd642019-11-20 00:14:47 +000037 [Documentation] Uses kubectl to resolve a service name to an PORT
Gilles Depatie675a2062019-10-22 12:44:42 -040038 ${rc} ${port}= Run and Return Rc and Output
39 ... kubectl get svc -n ${namespace} ${name} -o jsonpath={.spec.ports[0].port}
David Bainbridge117d23e2019-09-30 20:37:51 +000040 Should Be Equal as Integers ${rc} 0
41 [Return] ${port}
suraj gourd64356b2019-11-07 13:26:20 +000042
43Restart Pod
44 [Arguments] ${namespace} ${name}
45 [Documentation] Uses kubectl to force delete pod
suraj gour067451d2019-11-13 11:20:13 +000046 ${rc} ${restart_pod_name}= Run and Return Rc and Output
47 ... kubectl get pods -n ${namespace} | grep ${name} | awk 'NR==1{print $1}'
suraj gourd64356b2019-11-07 13:26:20 +000048 Log ${restart_pod_name}
suraj gour067451d2019-11-13 11:20:13 +000049 Should Not Be Empty ${restart_pod_name} Unable to parse pod name
Zack Williamsa8fe75a2020-01-10 14:25:27 -070050 ${rc} ${output}= Run and Return Rc and Output
51 ... kubectl delete pod ${restart_pod_name} -n ${namespace} --grace-period=0 --force
suraj gourd64356b2019-11-07 13:26:20 +000052 Log ${output}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +000053
Scott Baker60e570d2020-02-02 22:10:13 -080054Exec Pod
55 [Arguments] ${namespace} ${name} ${command}
56 [Documentation] Uses kubectl to execute a command in the pod and return the output
57 ${rc} ${exec_pod_name}= Run and Return Rc and Output
58 ... kubectl get pods -n ${namespace} | grep ${name} | awk 'NR==1{print $1}'
59 Log ${exec_pod_name}
60 Should Not Be Empty ${exec_pod_name} Unable to parse pod name
61 ${rc} ${output}= Run and Return Rc and Output
62 ... kubectl exec -i ${exec_pod_name} -n ${namespace} -- ${command}
63 Log ${output}
64 [return] ${output}
65
66Exec Pod Separate Stderr
67 [Arguments] ${namespace} ${name} ${command}
68 [Documentation] Uses kubectl to execute a command in the pod and return the stderr and stdout
69 ${rc} ${exec_pod_name}= Run and Return Rc and Output
70 ... kubectl get pods -n ${namespace} | grep ${name} | awk 'NR==1{print $1}'
71 Log ${exec_pod_name}
72 Should Not Be Empty ${exec_pod_name} Unable to parse pod name
73 @{args}= Split String ${command}
74 ${result}= Run Process
75 ... kubectl exec -i ${exec_pod_name} -n ${namespace} -- @{args}
76 ${stdout}= Set Variable ${result.stdout}
77 ${stderr}= Set Variable ${result.stderr}
78 Log ${stdout}
79 Log ${stderr}
80 [return] ${stdout} ${stderr}
81
82Apply Kubernetes Resources
83 [Arguments] ${resource_yaml} ${namespace}
84 [Documentation] Use kubectl to create resources given a yaml file
85 ${rc} Run and Return Rc
86 ... kubectl apply -n ${namespace} -f ${resource_yaml}
87 Should Be Equal as Integers ${rc} 0
88
suraj gour1ecfae92019-12-20 15:11:40 +000089Validate Pod Status
90 [Arguments] ${pod_name} ${namespace} ${expectedStatus}
91 [Documentation] To run the kubectl command and check the status of the given pod matches the expected status
Zack Williamsa8fe75a2020-01-10 14:25:27 -070092 ${length}= Run kubectl get pod -n ${namespace} | wc -l
suraj gour1ecfae92019-12-20 15:11:40 +000093 FOR ${index} IN RANGE ${length}-1
Zack Williamsa8fe75a2020-01-10 14:25:27 -070094 ${currentPodName}= Run
95 ... kubectl get pod -n ${namespace} -o=jsonpath="{.items[${index}].status.containerStatuses[0].name}"
suraj gour1ecfae92019-12-20 15:11:40 +000096 Log Required Pod : ${pod_name}
97 Log Current Pod: ${currentPodName}
98 Run Keyword and Ignore Error Run Keyword If '${currentPodName}'=='${pod_name}' Exit For Loop
99 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700100 ${currentStatusofPod}= Run
101 ... kubectl get pod -n ${namespace} -o=jsonpath="{.items[${index}].status.phase}"
suraj gour1ecfae92019-12-20 15:11:40 +0000102 Log ${currentStatusofPod}
103 Should Contain ${currentStatusofPod} ${expectedStatus}
104
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000105Verify All Voltha Pods For Any Error Logs
106 [Arguments] ${datetime}
107 [Documentation] This keyword checks for the error occurence in the voltha pods
108 &{errorPodDict} Create Dictionary
109 &{containerDict} Get Container Dictionary
110 FOR ${podName} IN @{PODLIST1}
111 ${containerName} Get From Dictionary ${containerDict} ${podName}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700112 ${rc} ${logOutput} Run And Return Rc And Output
113 ... kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
114 Run Keyword And Ignore Error
115 ... Run Keyword If '${logOutput}'=='${EMPTY}'
116 ... Run Keywords Log No Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000117 ... AND Continue For Loop
118 ${errorDict} Check For Error Logs in Pod Type1 Given the Log Output ${logOutput}
119 ${returnStatusFlagList} Get Dictionary Keys ${errorDict}
120 ${returnStatusFlag} Get From List ${returnStatusFlagList} 0
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700121 Run Keyword And Ignore Error
122 ... Run Keyword If '${returnStatusFlag}'=='Nologfound'
123 ... Run Keywords Log No Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000124 ... AND Continue For Loop
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700125 Run Keyword And Ignore Error
126 ... Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound'
127 ... Run Keywords Log Unexpected Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000128 ... AND Set to Dictionary ${errorPodDict} ${podName} ${errorDict}
129 END
130 FOR ${podName} IN @{PODLIST2}
131 ${containerName} Get From Dictionary ${containerDict} ${podName}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700132 ${rc} ${logOutput} Run And Return Rc And Output
133 ... kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
134 Run Keyword And Ignore Error
135 ... Run Keyword If '${logOutput}'=='${EMPTY}'
136 ... Run Keywords Log No Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000137 ... AND Continue For Loop
138 ${errorDict} Check For Error Logs in Pod Type2 Given the Log Output ${logOutput}
139 ${returnStatusFlagList} Get Dictionary Keys ${errorDict}
140 ${returnStatusFlag} Get From List ${returnStatusFlagList} 0
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700141 Run Keyword And Ignore Error
142 ... Run Keyword If '${returnStatusFlag}'=='Nologfound'
143 ... Run Keywords Log No Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000144 ... AND Continue For Loop
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700145 Run Keyword And Ignore Error
146 ... Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound'
147 ... Run Keywords Log Unexpected Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000148 ... AND Set to Dictionary ${errorPodDict} ${podName} ${errorDict}
149 END
150 Print to Console Error Statement logged in the following pods : ${errorPodDict}
151 [Return] ${errorPodDict}
152
153Check For Error Logs in Pod Type1 Given the Log Output
154 [Arguments] ${logOutput} ${logLevel}=error ${errorMessage}=${EMPTY}
155 [Documentation] Checks for error message in the particular list of pods
156 Log ${logOutput}
157 ${linesContainingLog} = Get Lines Matching Regexp ${logOutput} .*\s\${logLevel}.* partial_match=true
158 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingLog}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700159 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
160 ... Nologfound '${is_exec_status}'=='FAIL' Errorlogfound
161 ${linesContainingError} = Get Lines Matching Regexp
162 ... ${logOutput} .*\s\${logLevel}.*${errorMessage} partial_match=true
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000163 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingError}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700164 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
165 ... UnexpectedErrorfound '${is_exec_status}'=='FAIL' MatchingErrorlogfound
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000166 Log {linesContainingError}
167 &{errorDict} Create Dictionary ${returnStatusFlag} ${linesContainingLog}
168 [Return] ${errorDict}
169
170Check For Error Logs in Pod Type2 Given the Log Output
171 [Arguments] ${logOutput} ${logLevel}=warn ${errorMessage}=${EMPTY}
172 [Documentation] Checks for error message in the particular set of pods
173 Log ${logOutput}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700174 ${linesContainingLog} = Get Lines Matching Regexp
175 ... ${logOutput} .*?\s.*level.*${logLevel}.* partial_match=true
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000176 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingLog}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700177 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
178 ... Nologfound '${is_exec_status}'=='FAIL' Errorlogfound
179 ${linesContainingError} = Get Lines Matching Regexp
180 ... ${logOutput} .*?\s.*level.*${logLevel}.*msg.*${errorMessage} partial_match=true
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000181 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingError}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700182 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS'
183 ... UnexpectedErrorfound '${is_exec_status}'=='FAIL' MatchingErrorlogfound
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000184 Log {linesContainingError}
185 &{errorDict} Create Dictionary ${returnStatusFlag} ${linesContainingLog}
186 [Return] ${errorDict}
187
188Get Container Dictionary
189 [Documentation] Creates a mapping for pod name and container name and returns the same
190 &{containerDict} Create Dictionary
191 ${containerName} Set Variable ${EMPTY}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700192 ${podName} Run kubectl get deployment -n voltha | awk 'NR>1 {print $1}'
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000193 @{podNameList}= Split To Lines ${podName}
194 Append To List ${podNameList} voltha-etcd-cluster voltha-kafka voltha-ro-core voltha-zookeeper
195 Log ${podNameList}
196 #Creatiing dictionary to correspond pod name and container name
197 FOR ${pod} IN @{podNameList}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700198 ${containerName} Run kubectl get pod -n voltha | grep ${pod} | awk '{print $1}'
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000199 &{containerDict} Set To Dictionary ${containerDict} ${pod} ${containerName}
200 END
201 Log ${containerDict}
202 [Return] ${containerDict}
203
204Validate Error For Given Pods
205 [Arguments] ${datetime} ${podDict}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700206 [Documentation]
207 ... This keyword is used to get the list of pods if there is any unexpected error
208 ... in a particular pod(s) given the time-${datetime} from which the log needs to
209 ... be analysed and the dictionary of pods and the error in the dictionary format
210 ... ${podDict] .
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000211 ...
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700212 ... Usage: ${returnStatusFlag} Validate Error For Given Pods ${datetime} ${podDict}
213 ...
214 ... Arguments:
215 ...
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000216 ... ${datetime} = time from which the log needs to be taken
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700217 ... ${podDict} = Key-value pair of the pod name and the error msg
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000218 ...
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700219 ... Example: ${podDict} = Set Dictionary ${podDict} radius sample error message.
220 ...
221 ... In case the radius pod log has any other error than the expected
222 ... error, then the podname will be returned
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000223 ${podList} = Get Dictionary Keys ${podDict}
224 FOR ${podName} IN @{podList}
225 ${containerName} Get From Dictionary ${containerDict} ${podName}
226 ${expectedError} Get From Dictionary ${podDict} ${podName}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700227 ${rc} ${logOutput} Run And Return Rc And Output
228 ... kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
229 Run Keyword And Ignore Error
230 ... Run Keyword If '${logOutput}'=='${EMPTY}'
231 ... Run Keywords Log No Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000232 ... AND Continue For Loop
233 ${returnStatusFlag} Check For Error Logs in Pod Type1 Given the Log Output ${logOutput}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700234 Run Keyword And Ignore Error
235 ... Run Keyword If '${returnStatusFlag}'=='Nologfound'
236 ... Run Keywords Log No Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000237 ... AND Continue For Loop
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700238 Run Keyword And Ignore Error
239 ... Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound'
240 ... Run Keywords Log Unexpected Error Log found in pod ${podName}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +0000241 ... AND Append To List ${errorPodList} ${podName}
242 END
243 [Return] ${errorPodList}
244
David Bainbridgef81cd642019-11-20 00:14:47 +0000245Delete K8s Pod
246 [Arguments] ${namespace} ${name}
247 [Documentation] Uses kubectl to delete a named POD
248 ${rc} Run and Return Rc
249 ... kubectl delete -n ${namespace} pod/${name}
250 Should Be Equal as Integers ${rc} 0
251
hwchiu85695932019-12-18 08:05:25 +0000252Delete K8s Pods By Label
253 [Arguments] ${namespace} ${key} ${value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700254 [Documentation] Uses kubectl to delete a PODs, filtering by label
hwchiu85695932019-12-18 08:05:25 +0000255 ${rc}= Run and Return Rc
256 ... kubectl -n ${namespace} delete pods -l${key}=${value}
257 Should Be Equal as Integers ${rc} 0
258
David Bainbridgef81cd642019-11-20 00:14:47 +0000259Scale K8s Deployment
260 [Arguments] ${namespace} ${name} ${count}
261 [Documentation] Uses kubectl to scale a named deployment
262 ${rc} Run and Return Rc
263 ... kubectl scale --replicas=${count} -n ${namespace} deploy/${name}
264 Should Be Equal as Integers ${rc} 0
265
266Pod Exists
267 [Arguments] ${namespace} ${name}
268 [Documentation] Succeeds it the named POD exists
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700269 ${rc} ${count} Run and Return Rc
270 ... kubectl get -n ${namespace} pod -o json | jq -r ".items[].metadata.name" | grep ${name}
David Bainbridgef81cd642019-11-20 00:14:47 +0000271 Should Be True ${count}>0
272
273Pod Does Not Exist
274 [Arguments] ${namespace} ${name}
275 [Documentation] Succeeds if the named POD does not exist
276 ${rc} ${count} Run and Return Rc And Output
277 ... kubectl get -n ${namespace} pod -o json | jq -r ".items[].metadata.name" | grep -c ${name}
278 Should Be Equal As Integers ${count} 0
279 Should Be True ${count}==0
280
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700281Pods Do Not Exist By Label
hwchiu85695932019-12-18 08:05:25 +0000282 [Arguments] ${namespace} ${key} ${value}
283 [Documentation] Succeeds if the named POD does not exist
284 ${rc} ${count} Run and Return Rc And Output
285 ... kubectl get -n ${namespace} pod -l${key}=${value} -o json | jq -r ".items[].metadata.name" | wc -l
286 Should Be Equal As Integers ${count} 0
287 Should Be True ${count}==0
288
David Bainbridgef81cd642019-11-20 00:14:47 +0000289Get Available Deployment Replicas
290 [Arguments] ${namespace} ${name}
291 [Documentation] Succeeds if the named POD exists and has a ready count > 0
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700292 ${rc} ${count} Run and Return Rc and Output
293 ... kubectl get -n ${namespace} deploy/${name} -o jsonpath='{.status.availableReplicas}'
David Bainbridgef81cd642019-11-20 00:14:47 +0000294 ${result}= Run Keyword If '${count}' == '' Set Variable 0
295 ... ELSE Set Variable ${count}
296 [Return] ${result}
297
298Check Expected Available Deployment Replicas
299 [Arguments] ${namespace} ${name} ${expected}
300 [Documentation] Succeeds if the named deployment has the expected number of available replicas
301 ${count}= Get Available Deployment Replicas ${namespace} ${name}
302 Should Be Equal As Integers ${expected} ${count}
303
304Get Deployment Replica Count
305 [Arguments] ${namespace} ${name}
306 [Documentation] Uses kubectl to fetch the number of configured replicas on a deployment
307 ${rc} ${value} Run and Return Rc and Output
308 ... kubectl -n ${namespace} get deploy/${name} -o 'jsonpath={.status.replicas}'
309 Should Be Equal as Integers ${rc} 0
310 ${replicas}= Run Keyword If '${value}' == '' Set Variable 0
311 ... ELSE Set Variable ${value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700312 [Return] ${replicas}
David Bainbridgef81cd642019-11-20 00:14:47 +0000313
314Does Deployment Have Replicas
315 [Arguments] ${namespace} ${name} ${expected_count}
316 [Documentation] Uses kubectl to fetch the number of configured replicas on a deployment
317 ${rc} ${value} Run and Return Rc and Output
318 ... kubectl -n ${namespace} get deploy/${name} -o 'jsonpath={.status.replicas}'
319 Should Be Equal as Integers ${rc} 0
320 ${replicas}= Run Keyword If '${value}' == '' Set Variable 0
321 ... ELSE Set Variable ${value}
322 Should be Equal as Integers ${replicas} ${expected_count}
hwchiu85695932019-12-18 08:05:25 +0000323
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700324Pods Are Ready By Label
hwchiu85695932019-12-18 08:05:25 +0000325 [Arguments] ${namespace} ${key} ${value}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700326 [Documentation] Check that all pods with a label are ready
327 ${output}= Run
328 ... kubectl -n ${namespace} get pods -l ${key}=${value} -o=jsonpath="{.items[].status.containerStatuses[].ready}"
329 Should Not Contain ${output} "false"
Gayathri.Selvan49398962020-01-13 07:19:12 +0000330
hwchiu58af72d2020-01-14 00:50:35 +0000331Check Expected Running Pods Number By Label
332 [Arguments] ${namespace} ${key} ${value} ${number}
333 [Documentation] Succeeds if the desired pod has expected number replicas
334 ${rc} ${count} Run and Return Rc and Output
335 ... kubectl -n ${namespace} get pods -l ${key}=${value} -o json | jq -r ".items[].status.phase" | wc -l
336 Should Be Equal as Integers ${count} ${number}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000337