blob: c2083b7b928c3d00ab5e183f74b749536a0cb656 [file] [log] [blame]
Kailash92764922019-07-25 08:21:39 -07001# Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Kailash92764922019-07-25 08:21:39 -070014# robot test functions
15
16*** Settings ***
17Documentation Library for various utilities
18Library SSHLibrary
Kailash92764922019-07-25 08:21:39 -070019Library String
20Library DateTime
21Library Process
22Library Collections
23Library RequestsLibrary
24Library OperatingSystem
Andy Bavier63460da2020-02-20 14:35:12 -070025Library CORDRobot
26Library ImportResource resources=CORDRobot
Kailash92764922019-07-25 08:21:39 -070027
28*** Keywords ***
Kailash528f7c02019-07-31 10:55:49 -070029Check CLI Tools Configured
30 [Documentation] Tests that use 'voltctl' and 'kubectl' should execute this keyword in suite setup
31 # check voltctl and kubectl configured
Zack Williamsa8fe75a2020-01-10 14:25:27 -070032 ${voltctl_rc}= Run And Return RC voltctl device list
33 ${kubectl_rc}= Run And Return RC kubectl get pods
Gilles Depatie675a2062019-10-22 12:44:42 -040034 Run Keyword If ${voltctl_rc} != 0 or ${kubectl_rc} != 0 FATAL ERROR
35 ... VOLTCTL and KUBECTL not configured. Please configure before executing tests.
Gilles Depatieb5682f82019-10-31 10:39:45 -040036
37Send File To Onos
Zack Williamsa8fe75a2020-01-10 14:25:27 -070038 [Documentation] Send the content of the file to Onos to selected section of configuration
39 ... using Post Request
40 [Arguments] ${CONFIG_FILE} ${section}
41 ${Headers}= Create Dictionary Content-Type application/json
42 ${File_Data}= OperatingSystem.Get File ${CONFIG_FILE}
43 Log ${Headers}
44 Log ${File_Data}
45 ${resp}= Post Request ONOS
46 ... /onos/v1/network/configuration/${section} headers=${Headers} data=${File_Data}
47 Should Be Equal As Strings ${resp.status_code} 200
Andy Bavier88cd9f62019-11-26 16:22:33 -070048
49Common Test Suite Setup
50 [Documentation] Setup the test suite
Andy Bavier88cd9f62019-11-26 16:22:33 -070051 Set Global Variable ${KUBECTL_CONFIG} export KUBECONFIG=%{KUBECONFIG}
52 Set Global Variable ${VOLTCTL_CONFIG} export VOLTCONFIG=%{VOLTCONFIG}
53 ${k8s_node_ip}= Evaluate ${nodes}[0].get("ip")
ubuntu6b6e7d42020-03-02 12:35:42 -080054 ${ONOS_REST_IP}= Get Environment Variable ONOS_REST_IP ${k8s_node_ip}
55 ${ONOS_SSH_IP}= Get Environment Variable ONOS_SSH_IP ${k8s_node_ip}
56 Set Global Variable ${ONOS_REST_IP}
57 Set Global Variable ${ONOS_SSH_IP}
Andy Bavier88cd9f62019-11-26 16:22:33 -070058 ${k8s_node_user}= Evaluate ${nodes}[0].get("user")
59 ${k8s_node_pass}= Evaluate ${nodes}[0].get("pass")
60 Check CLI Tools Configured
61 ${onos_auth}= Create List karaf karaf
62 ${HEADERS} Create Dictionary Content-Type=application/json
ubuntu6b6e7d42020-03-02 12:35:42 -080063 Create Session ONOS http://${ONOS_REST_IP}:${ONOS_REST_PORT} auth=${ONOS_AUTH}
Andy Bavier88cd9f62019-11-26 16:22:33 -070064 ${olt_ip}= Evaluate ${olts}[0].get("ip")
65 ${olt_user}= Evaluate ${olts}[0].get("user")
66 ${olt_pass}= Evaluate ${olts}[0].get("pass")
67 ${olt_serial_number}= Evaluate ${olts}[0].get("serial")
68 ${num_onus}= Get Length ${hosts.src}
69 ${num_onus}= Convert to String ${num_onus}
70 #send sadis file to onos
Andy Baviera2691752019-11-27 12:18:38 -070071 ${sadis_file}= Get Variable Value ${sadis.file}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070072 Log To Console \nSadis File:${sadis_file}
Andy Bavier88cd9f62019-11-26 16:22:33 -070073 Run Keyword Unless '${sadis_file}' is '${None}' Send File To Onos ${sadis_file} apps/
74 Set Suite Variable ${num_onus}
75 Set Suite Variable ${olt_serial_number}
76 Set Suite Variable ${olt_ip}
77 Set Suite Variable ${olt_user}
78 Set Suite Variable ${olt_pass}
Andy Bavier88cd9f62019-11-26 16:22:33 -070079 @{container_list}= Create List adapter-open-olt adapter-open-onu voltha-api-server
80 ... voltha-ro-core voltha-rw-core-11 voltha-rw-core-12 voltha-ofagent
81 Set Suite Variable ${container_list}
82 ${datetime}= Get Current Date
suraj gour7f6d5fe2019-11-29 10:56:35 +000083 Set Suite Variable ${datetime}
84
85WPA Reassociate
suraj gour7f6d5fe2019-11-29 10:56:35 +000086 [Documentation] Executes a particular wpa_cli reassociate, which performs force reassociation
Zack Williamsa8fe75a2020-01-10 14:25:27 -070087 [Arguments] ${iface} ${ip} ${user} ${pass}=${None}
88 ... ${container_type}=${None} ${container_name}=${None}
suraj gour7f6d5fe2019-11-29 10:56:35 +000089 #Below for loops are used instead of sleep time, to execute reassociate command and check status
Zack Williamsa8fe75a2020-01-10 14:25:27 -070090 FOR ${i} IN RANGE 70
91 ${output}= Login And Run Command On Remote System
92 ... wpa_cli -i ${iface} reassociate ${ip} ${user}
93 ... ${pass} ${container_type} ${container_name}
94 ${passed}= Run Keyword And Return Status Should Contain ${output} OK
Andy Bavierb63f6d22020-03-12 15:34:37 -070095 Exit For Loop If ${passed}
Zack Williamsa8fe75a2020-01-10 14:25:27 -070096 END
Andy Bavierb63f6d22020-03-12 15:34:37 -070097 Should Be True ${passed} Status does not contain 'SUCCESS'
Zack Williamsa8fe75a2020-01-10 14:25:27 -070098 FOR ${i} IN RANGE 70
99 ${output}= Login And Run Command On Remote System
Andy Bavierae4a17e2020-02-25 14:54:36 -0700100 ... wpa_cli -i ${iface} status | grep SUCCESS ${ip} ${user}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700101 ... ${pass} ${container_type} ${container_name}
102 ${passed}= Run Keyword And Return Status Should Contain ${output} SUCCESS
Andy Bavierb63f6d22020-03-12 15:34:37 -0700103 Exit For Loop If ${passed}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700104 END
Andy Bavierb63f6d22020-03-12 15:34:37 -0700105 Should Be True ${passed} Status does not contain 'SUCCESS'
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700106
suraj gour7f6d5fe2019-11-29 10:56:35 +0000107Validate Authentication After Reassociate
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700108 [Arguments] ${auth_pass} ${iface} ${ip} ${user} ${pass}=${None}
109 ... ${container_type}=${None} ${container_name}=${None}
110 [Documentation]
111 ... Executes a particular reassociate request on the RG using wpa_cli.
112 ... auth_pass determines if authentication should pass
Andy Bavier84834d42020-02-25 13:49:50 -0700113 ${wpa_log}= Catenate SEPARATOR=. /tmp/wpa ${iface} log
114 ${output}= Login And Run Command On Remote System truncate -s 0 ${wpa_log}; cat ${wpa_log}
115 ... ${ip} ${user} ${pass} ${container_type} ${container_name}
116 Log ${output}
117 Should Not Contain ${output} authentication completed successfully
suraj gour7f6d5fe2019-11-29 10:56:35 +0000118 WPA Reassociate ${iface} ${ip} ${user} ${pass} ${container_type} ${container_name}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700119 Run Keyword If '${auth_pass}' == 'True' Wait Until Keyword Succeeds ${timeout} 2s
Andy Bavier84834d42020-02-25 13:49:50 -0700120 ... Check Remote File Contents True ${wpa_log} ${iface}.*authentication completed successfully
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700121 ... ${ip} ${user} ${pass} ${container_type} ${container_name}
suraj gour7f6d5fe2019-11-29 10:56:35 +0000122 Run Keyword If '${auth_pass}' == 'False' Sleep 20s
Suchitra Vemuri3cb29f82020-02-25 13:59:58 -0800123 Run Keyword If '${auth_pass}' == 'False' Check Remote File Contents False /tmp/wpa.log
Andy Bavier84834d42020-02-25 13:49:50 -0700124 ... ${iface}.*authentication completed successfully ${ip} ${user} ${pass}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700125 ... ${container_type} ${container_name}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000126
127Send Dhclient Request To Release Assigned IP
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700128 [Arguments] ${iface} ${ip} ${user} ${path_dhcpleasefile} ${pass}=${None}
129 ... ${container_type}=${None} ${container_name}=${None}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000130 [Documentation] Executes a dhclient with option to release ip against a particular interface on the RG (src)
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700131 ${result}= Login And Run Command On Remote System
132 ... dhclient -nw -r ${iface} && rm ${path_dhcpleasefile}/dhclient.* ${ip} ${user}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000133 ... ${pass} ${container_type} ${container_name}
134 Log ${result}
135 #Should Contain ${result} DHCPRELEASE
136 [Return] ${result}
suraj gour1ecfae92019-12-20 15:11:40 +0000137
138Check Remote File Contents For WPA Logs
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700139 [Arguments] ${file_should_exist} ${file} ${pattern} ${ip} ${user} ${pass}=${None}
140 ... ${container_type}=${None} ${container_name}=${None} ${prompt}=~$
suraj gour1ecfae92019-12-20 15:11:40 +0000141 [Documentation] Checks for particular pattern count in a file
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700142 ${result}= Login And Run Command On Remote System
143 ... cat ${file} | grep '${pattern}' | wc -l ${ip} ${user} ${pass}
suraj gour1ecfae92019-12-20 15:11:40 +0000144 ... ${container_type} ${container_name} ${prompt}
145 [Return] ${result}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000146
147Perform Sanity Test
148 [Documentation] This keyword performs Sanity Test Procedure
149 ... Sanity test performs authentication, dhcp and pings for all the ONUs given for the POD
150 ... This keyword can be used to call in any other tests where sanity check is required
151 ... and avoids duplication of code.
152 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
153 Set Global Variable ${of_id}
154 FOR ${I} IN RANGE 0 ${num_onus}
155 ${src}= Set Variable ${hosts.src[${I}]}
156 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri03005912020-02-20 18:59:46 -0800157 Run Keyword and Ignore Error Collect Logs
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000158 ${onu_device_id}= Get Device ID From SN ${src['onu']}
159 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
160 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Suchitra Vemuri8078b972020-02-06 19:07:41 -0800161 # Check ONU port is Enabled in ONOS
162 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800163 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri8078b972020-02-06 19:07:41 -0800164 # Verify EAPOL flows are added for the ONU port
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000165 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800166 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri8078b972020-02-06 19:07:41 -0800167 # Verify ONU state in voltha
168 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
169 ... ENABLED ACTIVE REACHABLE
170 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
171 # Perform Authentication
Andy Bavier84834d42020-02-25 13:49:50 -0700172 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
173 ... /tmp/wpa ${src['dp_iface_name']} log
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000174 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
175 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700176 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000177 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
ubuntu6b6e7d42020-03-02 12:35:42 -0800178 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000179 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
ubuntu6b6e7d42020-03-02 12:35:42 -0800180 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000181 ... volt-add-subscriber-access ${of_id} ${onu_port}
Suchitra Vemuri9da44302020-03-04 14:24:49 -0800182 # Verify that no pending flows exist for the ONU port
183 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
184 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000185 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
186 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
187 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
188 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
189 ... ${dst['container_name']}
190 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800191 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000192 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
193 Run Keyword and Ignore Error Collect Logs
194 END
195
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000196Perform Sanity Test DT
197 [Documentation] This keyword performs Sanity Test Procedure for DT Workflow
198 ... Sanity test performs dhcp and pings (without EAPOL and DHCP flows) for all the ONUs given for the POD
199 ... This keyword can be used to call in any other tests where sanity check is required
200 ... and avoids duplication of code.
201 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
202 Set Global Variable ${of_id}
Hardik Windlass59a759f2020-03-26 14:49:01 +0530203 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
204 ... Get NNI Port in ONOS ${of_id}
205 Set Global Variable ${nni_port}
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000206 FOR ${I} IN RANGE 0 ${num_onus}
207 ${src}= Set Variable ${hosts.src[${I}]}
208 ${dst}= Set Variable ${hosts.dst[${I}]}
209 ${onu_device_id}= Get Device ID From SN ${src['onu']}
210 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
211 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
212 # Check ONU port is Enabled in ONOS
213 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800214 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000215 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
ubuntu6b6e7d42020-03-02 12:35:42 -0800216 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000217 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass59a759f2020-03-26 14:49:01 +0530218 # Verify subscriber access flows are added for the ONU port
219 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlassf187a872020-03-30 20:05:19 +0530220 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
221 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000222 # Verify ONU state in voltha
223 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
224 ... ENABLED ACTIVE REACHABLE
225 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass59a759f2020-03-26 14:49:01 +0530226 # TODO: Yet to Verify on the GPON based Physical POD (VOL-2652)
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000227 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
228 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
229 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
230 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
231 ... ${dst['container_name']}
232 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
233 Run Keyword and Ignore Error Collect Logs
234 END
Hardik Windlass59a759f2020-03-26 14:49:01 +0530235 # Verify ONOS Flows
236 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
237 ${onos_flows_count}= Evaluate 4 * ${num_onus}
238 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlassf187a872020-03-30 20:05:19 +0530239 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
240 ... ${onos_flows_count}
Hardik Windlass59a759f2020-03-26 14:49:01 +0530241 # Verify VOLTHA Flows
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000242 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
243 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
244 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
245 ${onu_flows}= Set Variable 2
246 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
247 ${List_ONU_Serial} Create List
248 Set Suite Variable ${List_ONU_Serial}
249 Build ONU SN List ${List_ONU_Serial}
250 Log ${List_ONU_Serial}
251 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
252 ... ${List_ONU_Serial} ${onu_flows}
253
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000254Setup
255 [Documentation] Pre-test Setup
256 #test for empty device list
257 Test Empty Device List
Suchitra Vemurice8306a2020-05-29 09:30:35 -0700258 # commenting "Openolt is up" keyword check to resolve issues with timeout with BAL3.4
259 #Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s Openolt is Up
260 #... ${olt_ip} ${olt_user} ${olt_pass}
261 Sleep 180s
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000262 #create/preprovision device
263 ${olt_device_id}= Create Device ${olt_ip} ${OLT_PORT}
264 Set Suite Variable ${olt_device_id}
265 #validate olt states
266 Wait Until Keyword Succeeds ${timeout} 5s
ubuntu6b6e7d42020-03-02 12:35:42 -0800267 ... Validate OLT Device PREPROVISIONED UNKNOWN UNKNOWN ${olt_device_id}
Suchitra Vemuria7e8ad22020-02-11 18:02:40 -0800268 Sleep 5s
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000269 Enable Device ${olt_device_id}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800270 Wait Until Keyword Succeeds 180s 5s
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000271 ... Validate OLT Device ENABLED ACTIVE REACHABLE ${olt_serial_number}
272 ${logical_id}= Get Logical Device ID From SN ${olt_serial_number}
273 Set Suite Variable ${logical_id}
274
Hardik Windlassaaea3402020-03-10 19:45:45 +0530275Validate ONUs After OLT Disable
276 [Documentation] Validates the ONUs state in Voltha, ONUs port state in ONOS
277 ... and that pings do not succeed After corresponding OLT is Disabled
278 FOR ${I} IN RANGE 0 ${num_onus}
279 ${src}= Set Variable ${hosts.src[${I}]}
280 ${dst}= Set Variable ${hosts.dst[${I}]}
281 ${onu_device_id}= Get Device ID From SN ${src['onu']}
282 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
283 ... ${of_id}
284 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
285 ... Validate Device ENABLED DISCOVERED
286 ... UNREACHABLE ${src['onu']} onu=True onu_reason=stopping-openomci
287 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
288 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
289 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
290 ... Wait Until Keyword Succeeds 60s 2s
291 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
292 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
293 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
294 Run Keyword and Ignore Error Collect Logs
295 END
296
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000297Delete All Devices and Verify
298 [Documentation] Remove any devices from VOLTHA and ONOS
299 # Clear devices from VOLTHA
300 Disable Devices In Voltha Root=true
Suchitra Vemuri620a9832020-03-30 17:31:34 -0700301 Sleep 5s
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000302 Wait Until Keyword Succeeds ${timeout} 2s Test Devices Disabled In Voltha Root=true
303 Delete Devices In Voltha Root=true
Suchitra Vemuri620a9832020-03-30 17:31:34 -0700304 Sleep 30s
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000305 Wait Until Keyword Succeeds ${timeout} 2s Test Empty Device List
306 # Clear devices from ONOS
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800307 #Remove All Devices From ONOS
ubuntu6b6e7d42020-03-02 12:35:42 -0800308 #... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000309
310Teardown
311 [Documentation] kills processes and cleans up interfaces on src+dst servers
312 Run Keyword If ${has_dataplane} Clean Up Linux
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000313
314Teardown Suite
315 [Documentation] Clean up device if desired
316 Run Keyword If ${teardown_device} Delete All Devices and Verify
317
318Delete Device and Verify
319 [Documentation] Disable -> Delete devices via voltctl and verify its removed
Suchitra Vemuri2fa9bba2020-01-22 17:38:48 -0800320 ${olt_device_id}= Get Device ID From SN ${olt_serial_number}
Gayathri.Selvanee4a91b2020-01-17 06:49:53 +0000321 ${rc} ${output}= Run and Return Rc and Output
322 ... ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
323 Should Be Equal As Integers ${rc} 0
324 Sleep 5s
325 Wait Until Keyword Succeeds ${timeout} 5s
326 ... Validate OLT Device DISABLED UNKNOWN REACHABLE ${olt_serial_number}
327 ${rc} ${output}= Run and Return Rc and Output
328 ... ${VOLTCTL_CONFIG}; voltctl device delete ${olt_device_id}
329 Sleep 50s
330 Should Be Equal As Integers ${rc} 0
331 Wait Until Keyword Succeeds ${timeout} 5s Validate Device Removed ${olt_device_id}
332
Gayathri.Selvan283a63c2020-01-23 04:09:18 +0000333Repeat Sanity Test
334 [Documentation] This keyword performs Sanity Test Procedure
335 ... Sanity test performs authentication, dhcp and pings for all the ONUs given for the POD
336 ... This keyword can be used to call in any other tests where sanity check is required
337 ... with wpa reassociation
338 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
339 Set Global Variable ${of_id}
340 FOR ${I} IN RANGE 0 ${num_onus}
Suchitra Vemuri03005912020-02-20 18:59:46 -0800341 Run Keyword and Ignore Error Collect Logs
Gayathri.Selvan283a63c2020-01-23 04:09:18 +0000342 ${src}= Set Variable ${hosts.src[${I}]}
343 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri8078b972020-02-06 19:07:41 -0800344 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
345 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
346 # Check ONU port is Enabled in ONOD
347 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
David K. Bainbridgec0617c92020-03-05 21:17:19 -0800348 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri8078b972020-02-06 19:07:41 -0800349 # Verify EAPOL flows are added for the ONU port
350 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
David K. Bainbridgec0617c92020-03-05 21:17:19 -0800351 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri8078b972020-02-06 19:07:41 -0800352 # Verify ONU state in voltha
Suchitra Vemurid1e27fb2020-02-01 16:34:37 -0800353 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Suchitra Vemuri81e4f022020-02-01 13:16:02 -0800354 ... ENABLED ACTIVE REACHABLE
355 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Suchitra Vemuri8078b972020-02-06 19:07:41 -0800356 # Perform Authentication
Gayathri.Selvan283a63c2020-01-23 04:09:18 +0000357 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
358 ... Validate Authentication After Reassociate
359 ... True ${src['dp_iface_name']} ${src['ip']} ${src['user']} ${src['pass']}
360 ... ${src['container_type']} ${src['container_name']}
361 Wait Until Keyword Succeeds ${timeout} 2s
David K. Bainbridgec0617c92020-03-05 21:17:19 -0800362 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Gayathri.Selvan283a63c2020-01-23 04:09:18 +0000363 Wait Until Keyword Succeeds ${timeout} 2s
David K. Bainbridgec0617c92020-03-05 21:17:19 -0800364 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Gayathri.Selvan283a63c2020-01-23 04:09:18 +0000365 ... volt-add-subscriber-access ${of_id} ${onu_port}
366 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
367 ... Validate DHCP and Ping True True
368 ... ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
369 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
370 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
371 ... ${dst['container_name']}
372 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
David K. Bainbridgec0617c92020-03-05 21:17:19 -0800373 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Gayathri.Selvan283a63c2020-01-23 04:09:18 +0000374 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
375 Run Keyword and Ignore Error Collect Logs
376 END
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000377
378Collect Logs
379 [Documentation] Collect Logs from voltha and onos cli for various commands
380 Run Keyword and Ignore Error Get Device List from Voltha
381 Run Keyword and Ignore Error Get Device Output from Voltha ${olt_device_id}
382 Run Keyword and Ignore Error Get Logical Device Output from Voltha ${logical_id}
ubuntu6b6e7d42020-03-02 12:35:42 -0800383 Get ONOS Status ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000384
385Verify ping is succesful except for given device
386 [Arguments] ${num_onus} ${exceptional_onu_id}
387 [Documentation] Checks that ping for all the devices are successful except the given ONU.
388 ${pingStatus} Set Variable True
389 FOR ${I} IN RANGE 0 ${num_onus}
390 ${src}= Set Variable ${hosts.src[${I}]}
391 ${dst}= Set Variable ${hosts.dst[${I}]}
392 ${onu_device_id}= Get Device ID From SN ${src['onu']}
393 ${pingStatus} Run Keyword If '${onu_device_id}' == '${exceptional_onu_id}' Set Variable False
394 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 60s 2s
395 ... Check Ping ${pingStatus} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
396 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
397 END
Andy Bavier4a8450e2020-02-04 08:58:37 -0700398
399Announce Message
400 [Arguments] ${message}
401 [Documentation] Announce a message that will be picked up by the log aggregator
Andy Bavier1af971b2020-02-05 14:26:21 -0700402 Run Process kubectl run announcer -ti --rm --restart Never --image ubuntu
Andy Bavier4a8450e2020-02-04 08:58:37 -0700403 ... bash -- -c echo; sleep 1; echo ${message}; sleep 1; date --rfc-3339\=n ; sleep 1; echo; sleep 1
Suchitra Vemuria5fa3dd2020-02-04 14:38:49 -0800404
Andy Bavierabeba262020-02-07 16:22:16 -0700405Start Logging
406 [Arguments] ${label}
407 [Documentation] Start logging for test ${label}
408 ${kail_process}= Run Keyword If "${container_log_dir}" != "${None}" Start Process kail -n default
409 ... -n voltha cwd=${container_log_dir} stdout=${label}-combined.log
410 Set Test Variable ${kail_process}
411
412Stop Logging
413 [Arguments] ${label}
414 [Documentation] End logging for test; remove logfile if test passed
415 Run sync
416 Run Keyword If ${kail_process} Terminate Process ${kail_process}
417 ${test_logfile}= Run Keyword If "${container_log_dir}" != "${None}"
418 ... Join Path ${container_log_dir} ${label}-combined.log
419 Run Keyword If Test Passed Run Keyword If "${test_logfile}" != "${None}" Remove File ${test_logfile}
420
Suchitra Vemuria5fa3dd2020-02-04 14:38:49 -0800421Clean Up Linux
422 [Documentation] Kill processes and clean up interfaces on src+dst servers
423 FOR ${I} IN RANGE 0 ${num_onus}
424 ${src}= Set Variable ${hosts.src[${I}]}
425 ${dst}= Set Variable ${hosts.dst[${I}]}
426 Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src['ip']}
427 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
428 Run Keyword And Ignore Error Kill Linux Process [d]hclient ${src['ip']}
429 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
430 Run Keyword If '${dst['ip']}' != '${None}' Run Keyword And Ignore Error
431 ... Kill Linux Process [d]hcpd ${dst['ip']} ${dst['user']}
432 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
433 Delete IP Addresses from Interface on Remote Host ${src['dp_iface_name']} ${src['ip']}
434 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
435 Run Keyword If '${dst['ip']}' != '${None}' Delete Interface on Remote Host
436 ... ${dst['dp_iface_name']}.${src['s_tag']} ${dst['ip']} ${dst['user']} ${dst['pass']}
437 ... ${dst['container_type']} ${dst['container_name']}
438 END
Scott Baker60e570d2020-02-02 22:10:13 -0800439
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800440Clean WPA Process
441 [Documentation] Kills wpa_supplicant processes only for all RGs
442 FOR ${I} IN RANGE 0 ${num_onus}
443 ${src}= Set Variable ${hosts.src[${I}]}
444 ${dst}= Set Variable ${hosts.dst[${I}]}
445 Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src['ip']}
446 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
447 END
448
Scott Baker60e570d2020-02-02 22:10:13 -0800449Should Be Larger Than
450 [Documentation] Verify that value_1 is > value_2
451 [Arguments] ${value_1} ${value_2}
452 Run Keyword If ${value_1} <= ${value_2}
453 ... Fail The value ${value_1} is not larger than ${value_2}
454
Scott Bakeree675552020-02-11 10:43:34 -0800455Should Be Larger Than Or Equal To
456 [Documentation] Verify that value_1 is >= value_2
457 [Arguments] ${value_1} ${value_2}
458 Run Keyword If ${value_1} < ${value_2}
459 ... Fail The value ${value_1} is not larger than or equal to ${value_2}
460
Scott Baker60e570d2020-02-02 22:10:13 -0800461Should Be Float
462 [Documentation] Verify that value is a floating point number type
463 [Arguments] ${value}
464 ${type} Evaluate type(${value}).__name__
465 Should Be Equal ${type} float
466
Scott Bakeree675552020-02-11 10:43:34 -0800467Should Be Newer Than Or Equal To
468 [Documentation] Compare two RFC3339 dates
469 [Arguments] ${value_1} ${value_2}
470 ${unix_v1} Parse RFC3339 ${value_1}
471 ${unix_v2} Parse RFC3339 ${value_2}
472 Run Keyword If ${unix_v1} < ${unix_v2}
473 ... Fail The value ${value_1} is not newer than or equal to ${value_2}
474
Scott Baker60e570d2020-02-02 22:10:13 -0800475Get Current Time
476 [Documentation] Return the current time in RFC3339 format
477 ${output}= Run date -u +"%FT%T%:z"
478 [return] ${output}
Scott Bakeree675552020-02-11 10:43:34 -0800479
480Parse RFC3339
481 [Documentation] Parse an RFC3339 timestamp
482 [Arguments] ${dateStr}
483 ${rc} ${output}= Run and Return Rc and Output date --date="${dateStr}" "+%s"
484 Should Be Equal As Numbers ${rc} 0
485 [return] ${output}
Hung-Wei Chiu228d7ae2020-05-29 22:16:56 +0000486
487Execute Remote Command
488 [Documentation] SSH into a remote host and execute a command on the bare host or in a container.
489 ... This replaces and simplifies the Login And Run Command On Remote System keyword in CORDRobot.
490 [Arguments] ${cmd} ${ip} ${user} ${pass}=${None}
491 ... ${container_type}=${None} ${container_name}=${None}
492 ${conn_id}= SSHLibrary.Open Connection ${ip}
493 Run Keyword If '${pass}' != '${None}'
494 ... SSHLibrary.Login ${user} ${pass}
495 ... ELSE
496 ... SSHLibrary.Login With Public Key ${user} %{HOME}/.ssh/id_rsa
497 ${namespace}= Run Keyword If '${container_type}' == 'K8S' SSHLibrary.Execute Command
498 ... kubectl get pods --all-namespaces | grep ${container_name} | awk '{print $1}'
499 ${stdout} ${stderr} ${rc}= Run Keyword If '${container_type}' == 'LXC'
500 ... SSHLibrary.Execute Command lxc exec ${container_name} -- ${cmd}
501 ... return_stderr=True return_rc=True
502 ... ELSE IF '${container_type}' == 'K8S'
503 ... SSHLibrary.Execute Command kubectl -n ${namespace} exec ${container_name} -- ${cmd}
504 ... return_stderr=True return_rc=True
505 ... ELSE
506 ... SSHLibrary.Execute Command ${cmd} return_stderr=True return_rc=True
507
508 Log ${stdout}
509 Log ${stderr}
510 Log ${rc}
511 SSHLibrary.Close Connection
512 [Return] ${stdout} ${stderr} ${rc}
513
514RestoreONUs
515 [Documentation] Restore all connected ONUs
516 [Arguments] ${num_onus}
517 FOR ${I} IN RANGE 0 ${num_onus}
518 ${src}= Set Variable ${hosts.src[${I}]}
519 ${container_type}= Get Variable Value ${src['container_type']} "null"
520 ${container_name}= Get Variable Value ${src['container_name']} "null"
521 ${onu_type}= Get Variable Value ${src['onu_type']} "null"
522 #Get ens6f0 from ens6f0.22
523 ${if_name}= Replace String Using Regexp ${src['dp_iface_name']} \\..* \
524 Run Keyword IF '${onu_type}' == 'alpha' AlphaONURestoreDefault ${src['ip']} ${src['user']}
525 ... ${src['pass']} ${if_name} admin admin ${container_type} ${container_name}
526 END
527
528AlphaONURestoreDefault
529 [Documentation] Restore the Alpha ONU to factory setting
530 [Arguments] ${rg_ip} ${rg_user} ${rg_pass} ${onu_ifname}
531 ... ${onu_user} ${onu_pass} ${container_type}=${None} ${container_name}=${None}
532 ${output}= Login And Run Command On Remote System sudo ifconfig ${onu_ifname} 192.168.1.3/24
533 ... ${rg_ip} ${rg_user} ${rg_pass} ${container_type} ${container_name}
534 ${cmd} Catenate
535 ... (echo open "192.168.1.1"; sleep 1;
536 ... echo "${onu_user}"; sleep 1;
537 ... echo "${onu_pass}"; sleep 1;
538 ... echo "restoredefault"; sleep 1) | telnet
539 ${output}= Login And Run Command On Remote System ${cmd}
540 ... ${rg_ip} ${rg_user} ${rg_pass} ${container_type} ${container_name}
541 Log To Console ${output}
542 ${output}= Login And Run Command On Remote System sudo ifconfig ${onu_ifname} 0
543 ... ${rg_ip} ${rg_user} ${rg_pass} ${container_type} ${container_name}