blob: 829efb8d32687d0b2569a5dc06a53a5b84740658 [file] [log] [blame]
Gayathri.Selvan283a63c2020-01-23 04:09:18 +00001# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000014
15*** Settings ***
HungWei Chiu87edb1f2020-01-28 00:10:58 +000016Documentation Test various failure scenarios
HungWei Chiu1408fb92020-03-03 19:43:30 -050017Suite Setup Setup Suite
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000018Test Setup Setup
19Test Teardown Teardown
20#Suite Teardown Teardown Suite
21Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../libraries/DependencyLibrary.py
27Resource ../../libraries/onos.robot
28Resource ../../libraries/voltctl.robot
29Resource ../../libraries/voltha.robot
30Resource ../../libraries/utils.robot
31Resource ../../libraries/k8s.robot
32Resource ../../variables/variables.robot
HungWei Chiu87edb1f2020-01-28 00:10:58 +000033Resource ../../libraries/power_switch.robot
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000034
35*** Variables ***
36${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
43${NAMESPACE} voltha
44# For below variable value, using deployment name as using grep for
45# parsing radius pod name, we can also use full radius pod name
46${RESTART_POD_NAME} radius
47${timeout} 60s
48${of_id} 0
49${logical_id} 0
Suchitra Vemuri109feb12020-01-23 16:25:02 -080050${has_dataplane} True
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000051${teardown_device} False
52${scripts} ../../scripts
53
Andy Bavierabeba262020-02-07 16:22:16 -070054# Per-test logging on failure is turned off by default; set this variable to enable
55${container_log_dir} ${None}
56
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000057*** Test Cases ***
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080058Verify ONU after rebooting physically
59 [Documentation] Test the ONU functionality by physically turning on/off ONU.
60 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
61 ... Test case runs only on the PODs that are configured with PowerSwitch that
62 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
63 ... VOL-2634
64 [Tags] functional PowerSwitch
Suchitra Vemuri9a6dd6d2020-02-28 17:46:26 -080065 [Setup] Run Keywords Announce Message START TEST ONUreboot_PowerSwitch
66 ... AND Start Logging ONUreboot_PowerSwitch
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080067 [Teardown] Run Keywords Collect Logs
Suchitra Vemuri9a6dd6d2020-02-28 17:46:26 -080068 ... AND Stop Logging ONUreboot_PowerSwitch
69 ... AND Announce Message END TEST ONUreboot_PowerSwitch
Suchitra Vemuri0656bc22020-03-21 19:07:49 -070070 ... AND Delete Device and Verify
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080071 # Add OLT device
72 setup
73 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
74 Run Keyword If ${has_dataplane} Clean Up Linux
75 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
76 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
77 FOR ${I} IN RANGE 0 ${num_onus}
78 ${src}= Set Variable ${hosts.src[${I}]}
79 ${dst}= Set Variable ${hosts.dst[${I}]}
80 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
81 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
82 Disable Switch Outlet ${src['power_switch_port']}
83 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
84 ... Wait Until Keyword Succeeds 60s 2s
85 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
86 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
87
88 Enable Switch Outlet ${src['power_switch_port']}
89 # Check ONU port is Enabled in ONOS
90 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
ubuntu6b6e7d42020-03-02 12:35:42 -080091 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080092 # Verify EAPOL flows are added for the ONU port
93 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -080094 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080095 # Verify ONU state in voltha
96 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
97 ... ENABLED ACTIVE REACHABLE
98 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080099 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
100 ... Wait Until Keyword Succeeds 60s 2s
101 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
102 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
103 Run Keyword And Ignore Error Collect Logs
104 END
Suchitra Vemurid7096132020-02-21 14:50:47 -0800105 # Deleting OLT after tests completes independently (as this test doesn't not run on each POD)
Suchitra Vemuri0656bc22020-03-21 19:07:49 -0700106 #Run Keyword If ${has_dataplane} Delete Device and Verify
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800107
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700108Verify OLT after rebooting physically
109 [Documentation] Test the physical reboot of the OLT
110 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
111 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
112 ... VOL-1956
113 [Tags] functional PhysicalOLTReboot
114 [Setup] Run Keywords Announce Message START TEST PhysicalOLTReboot
115 ... AND Start Logging PhysicalOLTReboot
116 [Teardown] Run Keywords Collect Logs
117 ... AND Stop Logging PhysicalOLTReboot
118 ... AND Announce Message END TEST PhysicalOLTReboot
Suchitra Vemurib3a92e42020-03-21 16:49:57 -0700119 ... AND Delete Device and Verify
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700120 # Add OLT device
121 setup
122 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
123 Run Keyword If ${has_dataplane} Clean Up Linux
124 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
125 # Reboot the OLT from the OLT CLI
126 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
127 ... sudo reboot ${olt_ip} ${olt_user} ${olt_pass} prompt=#
128 Run Keyword And Ignore Error Collect Logs
129 FOR ${I} IN RANGE 0 ${num_onus}
130 ${src}= Set Variable ${hosts.src[${I}]}
131 ${dst}= Set Variable ${hosts.dst[${I}]}
132 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
133 ... Wait Until Keyword Succeeds 60s 2s
134 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
135 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
136 END
137 # Wait for the OLT to come back up
138 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${olt_ip}
Suchitra Vemuri64db9652020-03-22 21:11:13 -0700139 # Waiting extra time for the ONUs to come up
140 Sleep 60s
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700141 Run Keyword And Ignore Error Collect Logs
142 Run Keyword If ${has_dataplane} Clean Up Linux
143 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
144 # Deleting OLT after test completes
Suchitra Vemurib3a92e42020-03-21 16:49:57 -0700145 #Run Keyword If ${has_dataplane} Delete Device and Verify
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700146
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700147Verify restart openolt-adapter container after subscriber provisioning
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800148 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
149 ... Prerequisite : ONUs are authenticated and pingable.
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800150 [Tags] functional VOL-1958 Restart-OpenOlt released
151 [Setup] Run Keywords Announce Message START TEST Restart-OpenOlt
152 ... AND Start Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800153 [Teardown] Run Keywords Collect Logs
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800154 ... AND Stop Logging Restart-OpenOlt
155 ... AND Announce Message END TEST Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800156 # Add OLT device
157 setup
158 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
159 Run Keyword If ${has_dataplane} Clean Up Linux
160 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
161 ${waitforRestart} Set Variable 120s
162 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
163 Log ${podStatusOutput}
164 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
165 ${podName} Set Variable adapter-open-olt
166 Restart Pod ${NAMESPACE} ${podName}
167 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${podName} ${NAMESPACE}
168 ... Running
Suchitra Vemuri312bfe02020-03-16 16:11:34 -0700169 # Wait for 1min after openolt adapter is restarted
170 Sleep 60s
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800171 Repeat Sanity Test
Suchitra Vemurid7096132020-02-21 14:50:47 -0800172 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800173 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
174 Log ${podStatusOutput}
175 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
176 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
177 Log to console Pod ${podName} restarted and sanity checks passed successfully
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800178
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800179Check OLT/ONU Authentication After Radius Pod Restart
180 [Documentation] After radius restart, triggers reassociation, checks status and
181 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
182 ... wpa supplicant is running in background hence it is recommended to remove
183 ... teardown from previous test or uncomment 'Teardown None'.
184 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
185 [Tags] functional RadiusRestart released
186 [Setup] Run Keywords Announce Message START TEST RadiusRestart
187 ... AND Start Logging RadiusRestart
188 [Teardown] Run Keywords Collect Logs
189 ... AND Stop Logging RadiusRestart
190 ... AND Announce Message END TEST RadiusRestart
191 ${waitforRestart} Set Variable 120s
192 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
193 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${RESTART_POD_NAME} ${NAMESPACE}
194 ... Running
195 FOR ${I} IN RANGE 0 ${num_onus}
196 ${src}= Set Variable ${hosts.src[${I}]}
197 ${dst}= Set Variable ${hosts.dst[${I}]}
198 ${onu_device_id}= Get Device ID From SN ${src['onu']}
199 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
200 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
201 Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800202 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800203 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
204 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
205 ... ${src['ip']} ${src['user']} ${src['pass']}
206 ... ${src['container_type']} ${src['container_name']}
ubuntu6b6e7d42020-03-02 12:35:42 -0800207 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800208 ... ${ONOS_SSH_PORT} ${onu_port}
209 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
210 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
211 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
212 ... ${src['ip']} ${src['user']} ${src['pass']}
213 ... ${src['container_type']} ${src['container_name']}
214 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
215 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
216 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800217 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800218 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
219 Run Keyword and Ignore Error Collect Logs
220 END
221
222Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
223 [Documentation] Deploys an device instance and waits for it to authenticate. After
224 ... authentication is successful the rw-core deployment is scaled to 0 instances to
225 ... simulate a POD crash. The test then scales the rw-core back to a single instance
226 ... and configures ONOS for access. The test succeeds if the device is able to
227 ... complete the DHCP sequence.
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700228 [Tags] functional rwcore-restart notready
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800229 [Setup] Run Keywords Announce Message START TEST RwCoreFailAndRestart
230 ... AND Start Logging RwCoreFailAndRestart
231 ... AND Clear All Devices Then Create New Device
232 [Teardown] Run Keywords Collect Logs
233 ... AND Stop Logging RwCoreFailAndRestart
234 ... AND Announce Message END TEST RwCoreFailAndRestart
235 ... AND Delete Device and Verify
236 Run Keyword and Ignore Error Collect Logs
237 Run Keyword If ${has_dataplane} Clean Up Linux
238 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
239 Set Global Variable ${of_id}
240 FOR ${I} IN RANGE 0 ${num_onus}
241 ${src}= Set Variable ${hosts.src[${I}]}
242 ${dst}= Set Variable ${hosts.dst[${I}]}
243 ${onu_device_id}= Get Device ID From SN ${src['onu']}
244 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
245 ... ${of_id}
246 # Bring up the device and verify it authenticates
247 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
248 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
ubuntu6b6e7d42020-03-02 12:35:42 -0800249 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800250 ... ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier84834d42020-02-25 13:49:50 -0700251 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
252 ... /tmp/wpa ${src['dp_iface_name']} log
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800253 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
254 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700255 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
ubuntu6b6e7d42020-03-02 12:35:42 -0800256 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800257 ... ${ONOS_SSH_PORT} ${onu_port}
258 END
259
260 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
261 Scale K8s Deployment voltha voltha-rw-core 0
262 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
263 # Ensure the ofagent POD goes "not-ready" as expected
264 Wait Until keyword Succeeds ${timeout} 2s
265 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
266 # Scale up the core deployment and make sure both it and the ofagent deployment are back
267 Scale K8s Deployment voltha voltha-rw-core 1
268 Wait Until Keyword Succeeds ${timeout} 2s
269 ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
270 Wait Until Keyword Succeeds ${timeout} 2s
271 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
272 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
273 # so restart the port forwarding for the API service
274 Restart VOLTHA Port Foward voltha-api-minimal
275 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
276 # represents system connectivity being restored
277 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
ubuntu6b6e7d42020-03-02 12:35:42 -0800278 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800279
280 FOR ${I} IN RANGE 0 ${num_onus}
281 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
ubuntu6b6e7d42020-03-02 12:35:42 -0800282 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800283 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
284 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
285 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
286 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
287 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
288 ... ${dst['container_name']}
289 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800290 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800291 END
292
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700293Verify openolt adapter restart before subscriber provisioning
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800294 [Documentation] Deploys an device instance and waits for it to authenticate. After
295 ... authentication is successful the rw-core deployment is scaled to 0 instances to
296 ... simulate a POD crash. The test then scales the rw-core back to a single instance
297 ... and configures ONOS for access. The test succeeds if the device is able to
298 ... complete the DHCP sequence.
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700299 [Tags] functional olt-adapter-restart
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800300 [Setup] Run Keywords Announce Message START TEST OltAdapterRestart
301 ... AND Start Logging OltAdapterRestart
Suchitra Vemuria0c086f2020-03-29 19:14:13 -0700302 #... AND Clear All Devices Then Create New Device
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800303 [Teardown] Run Keywords Collect Logs
304 ... AND Stop Logging OltAdapterRestart
305 ... AND Announce Message END TEST OltAdapterRestart
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700306 # Add OLT and perform sanity test
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700307 #setup
Suchitra Vemuri91537c82020-03-29 15:03:26 -0700308 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700309 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800310 Set Global Variable ${of_id}
311
312 FOR ${I} IN RANGE 0 ${num_onus}
313 ${src}= Set Variable ${hosts.src[${I}]}
314 ${dst}= Set Variable ${hosts.dst[${I}]}
315 ${onu_device_id}= Get Device ID From SN ${src['onu']}
316 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
317 ... ${of_id}
318
319 # Bring up the device and verify it authenticates
320 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
321 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
ubuntu6b6e7d42020-03-02 12:35:42 -0800322 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800323 ... ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier84834d42020-02-25 13:49:50 -0700324 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
325 ... /tmp/wpa ${src['dp_iface_name']} log
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800326 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
327 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700328 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
ubuntu6b6e7d42020-03-02 12:35:42 -0800329 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800330 ... ${ONOS_SSH_PORT} ${onu_port}
331 END
332 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
333 Scale K8s Deployment voltha adapter-open-olt 0
334 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha adapter-open-olt
335 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
336 Scale K8s Deployment voltha adapter-open-olt 1
337 Wait Until Keyword Succeeds ${timeout} 2s
338 ... Check Expected Available Deployment Replicas voltha adapter-open-olt 1
339
340 # Ensure the device is available in ONOS, this represents system connectivity being restored
341 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
ubuntu6b6e7d42020-03-02 12:35:42 -0800342 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800343
344 FOR ${I} IN RANGE 0 ${num_onus}
345 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
ubuntu6b6e7d42020-03-02 12:35:42 -0800346 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800347 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
348 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
349 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
350 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
351 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
352 ... ${dst['container_name']}
353 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800354 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800355 END
356
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000357Verify restart ofagent container after VOLTHA is operational
358 [Documentation] Restart ofagent container after VOLTHA is operational.
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000359 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530360 #TODO: remove "notready" tag once olt reboot and ofagent restart work items are done
Hardik Windlass7a846662020-03-19 20:42:59 +0530361 [Tags] functional VOL-2409 ofagentRestart
Andy Bavierabeba262020-02-07 16:22:16 -0700362 [Setup] Run Keywords Announce Message START TEST ofagentRestart
363 ... AND Start Logging ofagentRestart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700364 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700365 ... AND Stop Logging ofagentRestart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700366 ... AND Announce Message END TEST ofagentRestart
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530367 ... AND Scale K8s Deployment ${NAMESPACE} voltha-ofagent 1
Suchitra Vemuriceb002a2020-03-25 13:36:21 -0700368 # set timeout value
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000369 ${waitforRestart} Set Variable 120s
Andy Bavier4a8450e2020-02-04 08:58:37 -0700370 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000371 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700372 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530373 ${podName} Set Variable ofagent
374 Restart Pod ${NAMESPACE} ${podName}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000375 Sleep 60s
376 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
377 ... Running
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530378 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
379 Run Keyword If ${has_dataplane} Clean Up Linux
380 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Andy Bavier4a8450e2020-02-04 08:58:37 -0700381 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000382 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700383 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000384 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530385 # Scale Down the Of-Agent Deployment
386 Scale K8s Deployment ${NAMESPACE} voltha-ofagent 0
387 Sleep 30s
388 FOR ${I} IN RANGE 0 ${num_onus}
389 ${src}= Set Variable ${hosts.src[${I}]}
390 ${dst}= Set Variable ${hosts.dst[${I}]}
391 Run Keyword and Ignore Error Collect Logs
392 ${onu_device_id}= Get Device ID From SN ${src['onu']}
393 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
394 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
395 # Verify ONU state in voltha
396 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
397 ... ENABLED ACTIVE REACHABLE
398 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
399 # Check ONU port is Disabled in ONOS
400 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
401 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
402 # Verify EAPOL flows are present for the ONU port
403 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
404 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
405 # Verify ONU in AAA-Users
406 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
407 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
408 # Verify DHCP-Allocations
409 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
410 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
411 # Verify Ping
412 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
413 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
414 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
415 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
416 Run Keyword and Ignore Error Collect Logs
417 END
418 # Scale Up the Of-Agent Deployment
419 Scale K8s Deployment ${NAMESPACE} voltha-ofagent 1
420 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
421 ... Running
422 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
423 Run Keyword If ${has_dataplane} Clean Up Linux
424 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
425 Log to console Pod ${podName} restarted and sanity checks passed successfully
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000426
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000427Check ONU adapter crash not forcing authentication again
428 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
429 ... message count to make sure auth not started again and validates EAP status and ping.
430 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
431 [Tags] functional ONUAdaptCrash notready
Andy Bavierabeba262020-02-07 16:22:16 -0700432 [Setup] Run Keywords Announce Message START TEST ONUAdaptCrash
433 ... AND Start Logging ONUAdaptCrash
434 [Teardown] Run Keywords Collect Logs
435 ... AND Stop Logging ONUAdaptCrash
436 ... AND Announce Message END TEST ONUAdaptCrash
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000437 @{before_list}= Create List
438 @{after_list}= Create List
439 FOR ${I} IN RANGE 0 ${num_onus}
440 ${src}= Set Variable ${hosts.src[${I}]}
441 ${dst}= Set Variable ${hosts.dst[${I}]}
442 ${onu_device_id}= Get Device ID From SN ${src['onu']}
443 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
444 ... ${of_id}
445 ${before}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
446 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
447 ... ${src['container_type']} ${src['container_name']}
448 Append To List ${before_list} ${before}
449 END
450 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} adapter-open-onu
451 Wait Until Keyword Succeeds ${timeout} 2s Validate Pod Status ${podName} ${NAMESPACE}
452 ... Running
453 FOR ${I} IN RANGE 0 ${num_onus}
454 ${src}= Set Variable ${hosts.src[${I}]}
455 ${dst}= Set Variable ${hosts.dst[${I}]}
456 ${onu_device_id}= Get Device ID From SN ${src['onu']}
457 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
458 ... ${of_id}
459 ${after}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
460 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
461 ... ${src['container_type']} ${src['container_name']}
462 Append To List ${after_list} ${after}
463 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
464 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
465 ... ${src['container_type']} ${src['container_name']}
466 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
467 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
468 ... Wait Until Keyword Succeeds 60s 2s Check Ping
469 ... True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']} ${src['user']}
470 ... ${src['pass']} ${src['container_type']} ${src['container_name']}
471 END
472 Lists Should Be Equal ${after_list} ${before_list}
473 Log ${after_list}
474 Log ${before_list}
475 Run Keyword and Ignore Error Collect Logs
476
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000477ONU Reboot
478 [Documentation] Reboot ONU and verify that ONU comes up properly
479 [Tags] VOL-1957 RebootONU notready
Andy Bavierabeba262020-02-07 16:22:16 -0700480 [Setup] Run Keywords Announce Message START TEST RebootONU
481 ... AND Start Logging RebootONU
Andy Bavier4a8450e2020-02-04 08:58:37 -0700482 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700483 ... AND Stop Logging RebootONU
Andy Bavier4a8450e2020-02-04 08:58:37 -0700484 ... AND Announce Message END TEST RebootONU
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000485 FOR ${I} IN RANGE 0 ${num_onus}
486 ${src}= Set Variable ${hosts.src[${I}]}
487 ${dst}= Set Variable ${hosts.dst[${I}]}
488 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800489 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
490 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000491 Reboot ONU ${onu_device_id} ${src} ${dst}
492 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800493 # Check ONU port is Enabled in ONOS
494 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800495 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800496 # Verify EAPOL flows are added for the ONU port
497 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800498 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800499 # Verify ONU state in voltha
500 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
501 ... ENABLED ACTIVE REACHABLE
502 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
503 # Verify pings are successful after reboot on the current ONU
504 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
505 ... Wait Until Keyword Succeeds 60s 2s
506 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
507 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000508 END
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800509 #Run Keyword If ${has_dataplane} Clean Up Linux
510 #Check after reboot that ONUs are active, authenticated/DHCP/pingable
511 #Perform Sanity Test
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800512
513*** Keywords ***
514Setup Suite
515 [Documentation] Set up the test suite
516 Common Test Suite Setup
HungWei Chiu1408fb92020-03-03 19:43:30 -0500517 #power_switch.robot needs it to support different vendor's power switch
Hung-Wei Chiufcc6d8e2020-03-10 12:05:11 -0700518 ${switch_type}= Get Variable Value ${web_power_switch.type}
519 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800520
521Clear All Devices Then Create New Device
522 [Documentation] Remove any devices from VOLTHA and ONOS
523 # Remove all devices from voltha and nos
524 Delete All Devices and Verify
525 # Execute normal test Setup Keyword
526 Setup