blob: f214e96b55e3ab0bf1c9c7e975079e0c5faffd42 [file] [log] [blame]
Gayathri.Selvan283a63c2020-01-23 04:09:18 +00001# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000014
15*** Settings ***
HungWei Chiu87edb1f2020-01-28 00:10:58 +000016Documentation Test various failure scenarios
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000017Suite Setup Common Test Suite Setup
18Test Setup Setup
19Test Teardown Teardown
20#Suite Teardown Teardown Suite
21Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../libraries/DependencyLibrary.py
27Resource ../../libraries/onos.robot
28Resource ../../libraries/voltctl.robot
29Resource ../../libraries/voltha.robot
30Resource ../../libraries/utils.robot
31Resource ../../libraries/k8s.robot
32Resource ../../variables/variables.robot
HungWei Chiu87edb1f2020-01-28 00:10:58 +000033Resource ../../libraries/power_switch.robot
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000034
35*** Variables ***
36${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
43${NAMESPACE} voltha
44# For below variable value, using deployment name as using grep for
45# parsing radius pod name, we can also use full radius pod name
46${RESTART_POD_NAME} radius
47${timeout} 60s
48${of_id} 0
49${logical_id} 0
Suchitra Vemuri109feb12020-01-23 16:25:02 -080050${has_dataplane} True
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000051${teardown_device} False
52${scripts} ../../scripts
53
Andy Bavierabeba262020-02-07 16:22:16 -070054# Per-test logging on failure is turned off by default; set this variable to enable
55${container_log_dir} ${None}
56
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000057*** Test Cases ***
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080058Verify ONU after rebooting physically
59 [Documentation] Test the ONU functionality by physically turning on/off ONU.
60 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
61 ... Test case runs only on the PODs that are configured with PowerSwitch that
62 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
63 ... VOL-2634
64 [Tags] functional PowerSwitch
Suchitra Vemuri9a6dd6d2020-02-28 17:46:26 -080065 [Setup] Run Keywords Announce Message START TEST ONUreboot_PowerSwitch
66 ... AND Start Logging ONUreboot_PowerSwitch
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080067 [Teardown] Run Keywords Collect Logs
Suchitra Vemuri9a6dd6d2020-02-28 17:46:26 -080068 ... AND Stop Logging ONUreboot_PowerSwitch
69 ... AND Announce Message END TEST ONUreboot_PowerSwitch
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080070 # Add OLT device
71 setup
72 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
73 Run Keyword If ${has_dataplane} Clean Up Linux
74 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
75 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
76 FOR ${I} IN RANGE 0 ${num_onus}
77 ${src}= Set Variable ${hosts.src[${I}]}
78 ${dst}= Set Variable ${hosts.dst[${I}]}
79 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
80 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
81 Disable Switch Outlet ${src['power_switch_port']}
82 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
83 ... Wait Until Keyword Succeeds 60s 2s
84 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
85 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
86
87 Enable Switch Outlet ${src['power_switch_port']}
88 # Check ONU port is Enabled in ONOS
89 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
90 ... Verify ONU Port Is Enabled ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
91 # Verify EAPOL flows are added for the ONU port
92 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
93 ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
94 # Verify ONU state in voltha
95 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
96 ... ENABLED ACTIVE REACHABLE
97 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080098 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
99 ... Wait Until Keyword Succeeds 60s 2s
100 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
101 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
102 Run Keyword And Ignore Error Collect Logs
103 END
Suchitra Vemurid7096132020-02-21 14:50:47 -0800104 # Deleting OLT after tests completes independently (as this test doesn't not run on each POD)
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800105 Run Keyword If ${has_dataplane} Delete Device and Verify
106
107Verify restart openolt-adapter container after VOLTHA is operational
108 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
109 ... Prerequisite : ONUs are authenticated and pingable.
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800110 [Tags] functional VOL-1958 Restart-OpenOlt released
111 [Setup] Run Keywords Announce Message START TEST Restart-OpenOlt
112 ... AND Start Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800113 [Teardown] Run Keywords Collect Logs
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800114 ... AND Stop Logging Restart-OpenOlt
115 ... AND Announce Message END TEST Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800116 # Add OLT device
117 setup
118 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
119 Run Keyword If ${has_dataplane} Clean Up Linux
120 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
121 ${waitforRestart} Set Variable 120s
122 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
123 Log ${podStatusOutput}
124 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
125 ${podName} Set Variable adapter-open-olt
126 Restart Pod ${NAMESPACE} ${podName}
127 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${podName} ${NAMESPACE}
128 ... Running
129 Repeat Sanity Test
Suchitra Vemurid7096132020-02-21 14:50:47 -0800130 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800131 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
132 Log ${podStatusOutput}
133 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
134 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
135 Log to console Pod ${podName} restarted and sanity checks passed successfully
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800136
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800137Check OLT/ONU Authentication After Radius Pod Restart
138 [Documentation] After radius restart, triggers reassociation, checks status and
139 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
140 ... wpa supplicant is running in background hence it is recommended to remove
141 ... teardown from previous test or uncomment 'Teardown None'.
142 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
143 [Tags] functional RadiusRestart released
144 [Setup] Run Keywords Announce Message START TEST RadiusRestart
145 ... AND Start Logging RadiusRestart
146 [Teardown] Run Keywords Collect Logs
147 ... AND Stop Logging RadiusRestart
148 ... AND Announce Message END TEST RadiusRestart
149 ${waitforRestart} Set Variable 120s
150 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
151 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${RESTART_POD_NAME} ${NAMESPACE}
152 ... Running
153 FOR ${I} IN RANGE 0 ${num_onus}
154 ${src}= Set Variable ${hosts.src[${I}]}
155 ${dst}= Set Variable ${hosts.dst[${I}]}
156 ${onu_device_id}= Get Device ID From SN ${src['onu']}
157 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
158 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
159 Wait Until Keyword Succeeds ${timeout} 2s
160 ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
161 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
162 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
163 ... ${src['ip']} ${src['user']} ${src['pass']}
164 ... ${src['container_type']} ${src['container_name']}
165 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
166 ... ${ONOS_SSH_PORT} ${onu_port}
167 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
168 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
169 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
170 ... ${src['ip']} ${src['user']} ${src['pass']}
171 ... ${src['container_type']} ${src['container_name']}
172 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
173 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
174 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
175 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
176 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
177 Run Keyword and Ignore Error Collect Logs
178 END
179
180Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
181 [Documentation] Deploys an device instance and waits for it to authenticate. After
182 ... authentication is successful the rw-core deployment is scaled to 0 instances to
183 ... simulate a POD crash. The test then scales the rw-core back to a single instance
184 ... and configures ONOS for access. The test succeeds if the device is able to
185 ... complete the DHCP sequence.
Suchitra Vemurie9c79bf2020-02-26 23:11:39 -0800186 [Tags] functional rwcore-restart notready
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800187 [Setup] Run Keywords Announce Message START TEST RwCoreFailAndRestart
188 ... AND Start Logging RwCoreFailAndRestart
189 ... AND Clear All Devices Then Create New Device
190 [Teardown] Run Keywords Collect Logs
191 ... AND Stop Logging RwCoreFailAndRestart
192 ... AND Announce Message END TEST RwCoreFailAndRestart
193 ... AND Delete Device and Verify
194 Run Keyword and Ignore Error Collect Logs
195 Run Keyword If ${has_dataplane} Clean Up Linux
196 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
197 Set Global Variable ${of_id}
198 FOR ${I} IN RANGE 0 ${num_onus}
199 ${src}= Set Variable ${hosts.src[${I}]}
200 ${dst}= Set Variable ${hosts.dst[${I}]}
201 ${onu_device_id}= Get Device ID From SN ${src['onu']}
202 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
203 ... ${of_id}
204 # Bring up the device and verify it authenticates
205 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
206 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
207 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
208 ... ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier84834d42020-02-25 13:49:50 -0700209 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
210 ... /tmp/wpa ${src['dp_iface_name']} log
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800211 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
212 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700213 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800214 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
215 ... ${ONOS_SSH_PORT} ${onu_port}
216 END
217
218 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
219 Scale K8s Deployment voltha voltha-rw-core 0
220 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
221 # Ensure the ofagent POD goes "not-ready" as expected
222 Wait Until keyword Succeeds ${timeout} 2s
223 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
224 # Scale up the core deployment and make sure both it and the ofagent deployment are back
225 Scale K8s Deployment voltha voltha-rw-core 1
226 Wait Until Keyword Succeeds ${timeout} 2s
227 ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
228 Wait Until Keyword Succeeds ${timeout} 2s
229 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
230 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
231 # so restart the port forwarding for the API service
232 Restart VOLTHA Port Foward voltha-api-minimal
233 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
234 # represents system connectivity being restored
235 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
236 ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
237
238 FOR ${I} IN RANGE 0 ${num_onus}
239 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
240 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
241 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
242 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
243 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
244 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
245 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
246 ... ${dst['container_name']}
247 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
248 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
249 END
250
251Sanity E2E Test for OLT/ONU on POD With OLT Adapters Fail and Restart
252 [Documentation] Deploys an device instance and waits for it to authenticate. After
253 ... authentication is successful the rw-core deployment is scaled to 0 instances to
254 ... simulate a POD crash. The test then scales the rw-core back to a single instance
255 ... and configures ONOS for access. The test succeeds if the device is able to
256 ... complete the DHCP sequence.
257 [Tags] bbsim olt-adapter-restart
258 [Setup] Run Keywords Announce Message START TEST OltAdapterRestart
259 ... AND Start Logging OltAdapterRestart
260 ... AND Clear All Devices Then Create New Device
261 [Teardown] Run Keywords Collect Logs
262 ... AND Stop Logging OltAdapterRestart
263 ... AND Announce Message END TEST OltAdapterRestart
264 Run Keyword If ${has_dataplane} Clean Up Linux
265 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
266 Set Global Variable ${of_id}
267
268 FOR ${I} IN RANGE 0 ${num_onus}
269 ${src}= Set Variable ${hosts.src[${I}]}
270 ${dst}= Set Variable ${hosts.dst[${I}]}
271 ${onu_device_id}= Get Device ID From SN ${src['onu']}
272 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
273 ... ${of_id}
274
275 # Bring up the device and verify it authenticates
276 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
277 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
278 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
279 ... ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier84834d42020-02-25 13:49:50 -0700280 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
281 ... /tmp/wpa ${src['dp_iface_name']} log
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800282 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
283 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700284 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800285 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
286 ... ${ONOS_SSH_PORT} ${onu_port}
287 END
288 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
289 Scale K8s Deployment voltha adapter-open-olt 0
290 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha adapter-open-olt
291 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
292 Scale K8s Deployment voltha adapter-open-olt 1
293 Wait Until Keyword Succeeds ${timeout} 2s
294 ... Check Expected Available Deployment Replicas voltha adapter-open-olt 1
295
296 # Ensure the device is available in ONOS, this represents system connectivity being restored
297 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
298 ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
299
300 FOR ${I} IN RANGE 0 ${num_onus}
301 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
302 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
303 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
304 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
305 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
306 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
307 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
308 ... ${dst['container_name']}
309 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
310 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
311 END
312
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000313Verify restart ofagent container after VOLTHA is operational
314 [Documentation] Restart ofagent container after VOLTHA is operational.
315 ... Please note this test case should be run before the restart of other containers.
316 ... Prerequisite : ONUs are authenticated and pingable.
Suchitra Vemuriddba81e2020-02-23 22:15:53 -0800317 [Tags] functional VOL-2409 ofagentRestart notready
Andy Bavierabeba262020-02-07 16:22:16 -0700318 [Setup] Run Keywords Announce Message START TEST ofagentRestart
319 ... AND Start Logging ofagentRestart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700320 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700321 ... AND Stop Logging ofagentRestart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700322 ... AND Announce Message END TEST ofagentRestart
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000323 ${waitforRestart} Set Variable 120s
Andy Bavier4a8450e2020-02-04 08:58:37 -0700324 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000325 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700326 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000327 Restart Pod ${NAMESPACE} ofagent
328 Sleep 60s
329 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
330 ... Running
331 Repeat Sanity Test
Andy Bavier4a8450e2020-02-04 08:58:37 -0700332 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000333 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700334 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000335 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000336
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000337Check ONU adapter crash not forcing authentication again
338 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
339 ... message count to make sure auth not started again and validates EAP status and ping.
340 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
341 [Tags] functional ONUAdaptCrash notready
Andy Bavierabeba262020-02-07 16:22:16 -0700342 [Setup] Run Keywords Announce Message START TEST ONUAdaptCrash
343 ... AND Start Logging ONUAdaptCrash
344 [Teardown] Run Keywords Collect Logs
345 ... AND Stop Logging ONUAdaptCrash
346 ... AND Announce Message END TEST ONUAdaptCrash
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000347 @{before_list}= Create List
348 @{after_list}= Create List
349 FOR ${I} IN RANGE 0 ${num_onus}
350 ${src}= Set Variable ${hosts.src[${I}]}
351 ${dst}= Set Variable ${hosts.dst[${I}]}
352 ${onu_device_id}= Get Device ID From SN ${src['onu']}
353 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
354 ... ${of_id}
355 ${before}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
356 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
357 ... ${src['container_type']} ${src['container_name']}
358 Append To List ${before_list} ${before}
359 END
360 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} adapter-open-onu
361 Wait Until Keyword Succeeds ${timeout} 2s Validate Pod Status ${podName} ${NAMESPACE}
362 ... Running
363 FOR ${I} IN RANGE 0 ${num_onus}
364 ${src}= Set Variable ${hosts.src[${I}]}
365 ${dst}= Set Variable ${hosts.dst[${I}]}
366 ${onu_device_id}= Get Device ID From SN ${src['onu']}
367 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
368 ... ${of_id}
369 ${after}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
370 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
371 ... ${src['container_type']} ${src['container_name']}
372 Append To List ${after_list} ${after}
373 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
374 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
375 ... ${src['container_type']} ${src['container_name']}
376 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
377 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
378 ... Wait Until Keyword Succeeds 60s 2s Check Ping
379 ... True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']} ${src['user']}
380 ... ${src['pass']} ${src['container_type']} ${src['container_name']}
381 END
382 Lists Should Be Equal ${after_list} ${before_list}
383 Log ${after_list}
384 Log ${before_list}
385 Run Keyword and Ignore Error Collect Logs
386
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000387ONU Reboot
388 [Documentation] Reboot ONU and verify that ONU comes up properly
389 [Tags] VOL-1957 RebootONU notready
Andy Bavierabeba262020-02-07 16:22:16 -0700390 [Setup] Run Keywords Announce Message START TEST RebootONU
391 ... AND Start Logging RebootONU
Andy Bavier4a8450e2020-02-04 08:58:37 -0700392 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700393 ... AND Stop Logging RebootONU
Andy Bavier4a8450e2020-02-04 08:58:37 -0700394 ... AND Announce Message END TEST RebootONU
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000395 FOR ${I} IN RANGE 0 ${num_onus}
396 ${src}= Set Variable ${hosts.src[${I}]}
397 ${dst}= Set Variable ${hosts.dst[${I}]}
398 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800399 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
400 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000401 Reboot ONU ${onu_device_id} ${src} ${dst}
402 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800403 # Check ONU port is Enabled in ONOS
404 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
405 ... Verify ONU Port Is Enabled ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
406 # Verify EAPOL flows are added for the ONU port
407 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
408 ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
409 # Verify ONU state in voltha
410 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
411 ... ENABLED ACTIVE REACHABLE
412 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
413 # Verify pings are successful after reboot on the current ONU
414 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
415 ... Wait Until Keyword Succeeds 60s 2s
416 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
417 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000418 END
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800419 #Run Keyword If ${has_dataplane} Clean Up Linux
420 #Check after reboot that ONUs are active, authenticated/DHCP/pingable
421 #Perform Sanity Test
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800422
423*** Keywords ***
424Setup Suite
425 [Documentation] Set up the test suite
426 Common Test Suite Setup
427
428Clear All Devices Then Create New Device
429 [Documentation] Remove any devices from VOLTHA and ONOS
430 # Remove all devices from voltha and nos
431 Delete All Devices and Verify
432 # Execute normal test Setup Keyword
433 Setup