blob: 71c8c7e03c1fe5b060424f4fc63ce05f8b8d0ad8 [file] [log] [blame]
David Bainbridge117d23e2019-09-30 20:37:51 +00001# Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15# voltctl common functions
16
17*** Settings ***
18Documentation Library for various utilities
19Library SSHLibrary
20Library HttpLibrary.HTTP
21Library String
22Library DateTime
23Library Process
24Library Collections
25Library RequestsLibrary
26Library OperatingSystem
27
28*** Keywords ***
29Lookup Service IP
30 [Arguments] ${namespace} ${name}
31 [Documentation] Uses kubeclt to resolve a service name to an IP
Gilles Depatie675a2062019-10-22 12:44:42 -040032 ${rc} ${ip}= Run and Return Rc and Output
33 ... kubectl get svc -n ${namespace} ${name} -o jsonpath={.spec.clusterIP}
David Bainbridge117d23e2019-09-30 20:37:51 +000034 Should Be Equal as Integers ${rc} 0
35 [Return] ${ip}
36
37Lookup Service PORT
38 [Arguments] ${namespace} ${name}
39 [Documentation] Uses kubeclt to resolve a service name to an PORT
Gilles Depatie675a2062019-10-22 12:44:42 -040040 ${rc} ${port}= Run and Return Rc and Output
41 ... kubectl get svc -n ${namespace} ${name} -o jsonpath={.spec.ports[0].port}
David Bainbridge117d23e2019-09-30 20:37:51 +000042 Should Be Equal as Integers ${rc} 0
43 [Return] ${port}
suraj gourd64356b2019-11-07 13:26:20 +000044
45Restart Pod
46 [Arguments] ${namespace} ${name}
47 [Documentation] Uses kubectl to force delete pod
48 ${rc} ${restart_pod_name}= Run and Return Rc and Output kubectl get pods --template $'{{range .items}}{{.metadata.name}}{{"\\n"}}{{end}}', -n ${namespace} | grep ${name}
49 Log ${restart_pod_name}
50 ${rc} ${output}= Run and Return Rc and Output kubectl delete pod ${restart_pod_name} -n ${namespace} --grace-period=0 --force
51 Log ${output}
Gayathri.Selvan61fbcdb2019-11-14 05:08:19 +000052
53Verify All Voltha Pods For Any Error Logs
54 [Arguments] ${datetime}
55 [Documentation] This keyword checks for the error occurence in the voltha pods
56 &{errorPodDict} Create Dictionary
57 &{containerDict} Get Container Dictionary
58 FOR ${podName} IN @{PODLIST1}
59 ${containerName} Get From Dictionary ${containerDict} ${podName}
60 ${rc} ${logOutput} Run And Return Rc And Output ${KUBECTL_CONFIG};kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
61 Run Keyword And Ignore Error Run Keyword If '${logOutput}'=='${EMPTY}' Run Keywords Log No Log found in pod ${podName}
62 ... AND Continue For Loop
63 ${errorDict} Check For Error Logs in Pod Type1 Given the Log Output ${logOutput}
64 ${returnStatusFlagList} Get Dictionary Keys ${errorDict}
65 ${returnStatusFlag} Get From List ${returnStatusFlagList} 0
66 Run Keyword And Ignore Error Run Keyword If '${returnStatusFlag}'=='Nologfound' Run Keywords Log No Error Log found in pod ${podName}
67 ... AND Continue For Loop
68 Run Keyword And Ignore Error Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound' Run Keywords Log Unexpected Error Log found in pod ${podName}
69 ... AND Set to Dictionary ${errorPodDict} ${podName} ${errorDict}
70 END
71 FOR ${podName} IN @{PODLIST2}
72 ${containerName} Get From Dictionary ${containerDict} ${podName}
73 ${rc} ${logOutput} Run And Return Rc And Output ${KUBECTL_CONFIG};kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
74 Run Keyword And Ignore Error Run Keyword If '${logOutput}'=='${EMPTY}' Run Keywords Log No Log found in pod ${podName}
75 ... AND Continue For Loop
76 ${errorDict} Check For Error Logs in Pod Type2 Given the Log Output ${logOutput}
77 ${returnStatusFlagList} Get Dictionary Keys ${errorDict}
78 ${returnStatusFlag} Get From List ${returnStatusFlagList} 0
79 Run Keyword And Ignore Error Run Keyword If '${returnStatusFlag}'=='Nologfound' Run Keywords Log No Error Log found in pod ${podName}
80 ... AND Continue For Loop
81 Run Keyword And Ignore Error Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound' Run Keywords Log Unexpected Error Log found in pod ${podName}
82 ... AND Set to Dictionary ${errorPodDict} ${podName} ${errorDict}
83 END
84 Print to Console Error Statement logged in the following pods : ${errorPodDict}
85 [Return] ${errorPodDict}
86
87Check For Error Logs in Pod Type1 Given the Log Output
88 [Arguments] ${logOutput} ${logLevel}=error ${errorMessage}=${EMPTY}
89 [Documentation] Checks for error message in the particular list of pods
90 Log ${logOutput}
91 ${linesContainingLog} = Get Lines Matching Regexp ${logOutput} .*\s\${logLevel}.* partial_match=true
92 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingLog}
93 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS' Nologfound '${is_exec_status}'=='FAIL' Errorlogfound
94 ${linesContainingError} = Get Lines Matching Regexp ${logOutput} .*\s\${logLevel}.*${errorMessage} partial_match=true
95 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingError}
96 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS' UnexpectedErrorfound '${is_exec_status}'=='FAIL' MatchingErrorlogfound
97 Log {linesContainingError}
98 &{errorDict} Create Dictionary ${returnStatusFlag} ${linesContainingLog}
99 [Return] ${errorDict}
100
101Check For Error Logs in Pod Type2 Given the Log Output
102 [Arguments] ${logOutput} ${logLevel}=warn ${errorMessage}=${EMPTY}
103 [Documentation] Checks for error message in the particular set of pods
104 Log ${logOutput}
105 ${linesContainingLog} = Get Lines Matching Regexp ${logOutput} .*?\s.*level.*${logLevel}.* partial_match=true
106 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingLog}
107 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS' Nologfound '${is_exec_status}'=='FAIL' Errorlogfound
108 ${linesContainingError} = Get Lines Matching Regexp ${logOutput} .*?\s.*level.*${logLevel}.*msg.*${errorMessage} partial_match=true
109 ${is_exec_status} ${output} Run Keyword And Ignore Error Should Be Empty ${linesContainingError}
110 ${returnStatusFlag} Set Variable If '${is_exec_status}'=='PASS' UnexpectedErrorfound '${is_exec_status}'=='FAIL' MatchingErrorlogfound
111 Log {linesContainingError}
112 &{errorDict} Create Dictionary ${returnStatusFlag} ${linesContainingLog}
113 [Return] ${errorDict}
114
115Get Container Dictionary
116 [Documentation] Creates a mapping for pod name and container name and returns the same
117 &{containerDict} Create Dictionary
118 ${containerName} Set Variable ${EMPTY}
119 ${podName} Run ${KUBECTL_CONFIG};kubectl get deployment -n voltha | awk 'NR>1 {print $1}'
120 @{podNameList}= Split To Lines ${podName}
121 Append To List ${podNameList} voltha-etcd-cluster voltha-kafka voltha-ro-core voltha-zookeeper
122 Log ${podNameList}
123 #Creatiing dictionary to correspond pod name and container name
124 FOR ${pod} IN @{podNameList}
125 ${containerName} Run ${KUBECTL_CONFIG};kubectl get pod -n voltha | grep ${pod} | awk '{print $1}'
126 &{containerDict} Set To Dictionary ${containerDict} ${pod} ${containerName}
127 END
128 Log ${containerDict}
129 [Return] ${containerDict}
130
131Validate Error For Given Pods
132 [Arguments] ${datetime} ${podDict}
133 [Documentation] This keyword is used to get the list of pods if there is any unexpected error in a particular pod(s) given the time-${datetime} from which the log needs to be analysed and the dictionary of pods and the error in the dictionary format ${podDict] .
134 ...
135 ... Usage : ${returnStatusFlag} Validate Error For Given Pods ${datetime} ${podDict}
136 ... where,
137 ... ${datetime} = time from which the log needs to be taken
138 ... ${podDict} = Key-value pair of the pod name and the error msg expected like ${podDict} = Set Dictionary ${podDict} radius sample error message.
139 ...
140 ... In case the radius pod log has any other error than the expected error, then the podname will be returned
141 ${podList} = Get Dictionary Keys ${podDict}
142 FOR ${podName} IN @{podList}
143 ${containerName} Get From Dictionary ${containerDict} ${podName}
144 ${expectedError} Get From Dictionary ${podDict} ${podName}
145 ${rc} ${logOutput} Run And Return Rc And Output ${KUBECTL_CONFIG};kubectl logs --timestamps -n voltha --since-time=${datetime} ${containerName}
146 Run Keyword And Ignore Error Run Keyword If '${logOutput}'=='${EMPTY}' Run Keywords Log No Log found in pod ${podName}
147 ... AND Continue For Loop
148 ${returnStatusFlag} Check For Error Logs in Pod Type1 Given the Log Output ${logOutput}
149 Run Keyword And Ignore Error Run Keyword If '${returnStatusFlag}'=='Nologfound' Run Keywords Log No Error Log found in pod ${podName}
150 ... AND Continue For Loop
151 Run Keyword And Ignore Error Run Keyword If '${returnStatusFlag}'=='UnexpectedErrorfound' Run Keywords Log Unexpected Error Log found in pod ${podName}
152 ... AND Append To List ${errorPodList} ${podName}
153 END
154 [Return] ${errorPodList}
155