blob: 9ceb95853fa541d4247ca1a2cdbeaf76c659a845 [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
45# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
51${has_dataplane} True
52${teardown_device} False
53${scripts} ../../scripts
54
55# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
58*** Test Cases ***
59Verify ONU after Rebooting Physically for DT
60 [Documentation] Test the ONU functionality by physically turning on/off ONU.
61 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
62 ... Test case runs only on the PODs that are configured with PowerSwitch that
63 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
64 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053065 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +053066 [Teardown] Run Keywords Collect Logs
67 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
68 ... AND Delete Device and Verify
69 # Add OLT device
70 Setup
71 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
72 Run Keyword If ${has_dataplane} Clean Up Linux
73 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
74 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053075 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlassb9bdd502020-04-17 14:46:21 +053076 FOR ${I} IN RANGE 0 ${num_onus}
77 ${src}= Set Variable ${hosts.src[${I}]}
78 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlass0560f802020-05-06 23:06:53 +053079 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053080 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
81 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass7c801e62020-05-04 19:33:21 +053082 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053083 Disable Switch Outlet ${src['power_switch_port']}
84 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
85 ... Wait Until Keyword Succeeds 60s 2s
86 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
87 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +053088 # Remove Subscriber Access (To replicate DT workflow)
89 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
90 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Suchitra Vemuri1b8d7b82020-05-29 21:10:12 -070091 Sleep 10s
Hardik Windlass7c801e62020-05-04 19:33:21 +053092 # Delete ONU Device (To replicate DT workflow)
93 Delete Device ${onu_device_id}
94 Sleep 5s
95 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053096 Enable Switch Outlet ${src['power_switch_port']}
Hardik Windlass40ccffb2020-05-11 14:12:07 +053097 # Waiting extra time for the ONU to come up
98 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +053099 # Check ONU port is Enabled in ONOS
100 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700101 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530102 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
103 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
104 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530105 # Verify ONU state in voltha
106 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
107 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530108 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530109 # Verify subscriber access flows are added for the ONU port
110 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
111 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
112 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530113 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530114 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
115 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530116 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530117 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
118 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530119 Run Keyword And Ignore Error Collect Logs
120 END
121 # Verify ONOS Flows
122 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
123 ${onos_flows_count}= Evaluate 4 * ${num_onus}
124 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
125 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
126 ... ${of_id} ${onos_flows_count}
127 # Verify VOLTHA Flows
128 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
129 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
130 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
131 ${onu_flows}= Set Variable 2
132 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
133 ${List_ONU_Serial} Create List
134 Set Suite Variable ${List_ONU_Serial}
135 Build ONU SN List ${List_ONU_Serial}
136 Log ${List_ONU_Serial}
137 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
138 ... ${List_ONU_Serial} ${onu_flows}
139
140Verify OLT after Rebooting Physically for DT
141 [Documentation] Test the physical reboot of the OLT
142 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
143 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
144 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530145 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530146 [Teardown] Run Keywords Collect Logs
147 ... AND Stop Logging RebootOlt_Physical_Dt
148 ... AND Delete Device and Verify
149 # Add OLT device
150 Setup
151 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
152 Run Keyword If ${has_dataplane} Clean Up Linux
153 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
154 # Reboot the OLT from the OLT CLI
155 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
156 ... sudo reboot ${olt_ip} ${olt_user} ${olt_pass} prompt=#
157 Run Keyword And Ignore Error Collect Logs
158 FOR ${I} IN RANGE 0 ${num_onus}
159 ${src}= Set Variable ${hosts.src[${I}]}
160 ${dst}= Set Variable ${hosts.dst[${I}]}
161 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
162 ... Wait Until Keyword Succeeds 60s 2s
163 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
164 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
165 END
166 # Wait for the OLT to come back up
167 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
168 ... Check Remote System Reachability True ${olt_ip}
169 # Waiting extra time for the ONUs to come up
170 Sleep 60s
171 Run Keyword And Ignore Error Collect Logs
172 Run Keyword If ${has_dataplane} Clean Up Linux
173 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
174
Hardik Windlass9df139e2020-04-24 14:54:54 +0530175Verify restart openolt-adapter container after subscriber provisioning for DT
176 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
177 ... Prerequisite : ONUs are authenticated and pingable.
178 [Tags] functionalDt Restart-OpenOlt-Dt
179 [Setup] Start Logging Restart-OpenOlt-Dt
180 [Teardown] Run Keywords Collect Logs
181 ... AND Stop Logging Restart-OpenOlt-Dt
182 # Add OLT device
183 setup
184 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
185 Run Keyword If ${has_dataplane} Clean Up Linux
186 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
187 ${waitforRestart} Set Variable 120s
188 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
189 Log ${podStatusOutput}
190 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530191 ${podName} Set Variable adapter-open-olt
192 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
193 Sleep 5s
194 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
195 ... app ${podName} Running
Hardik Windlass9df139e2020-04-24 14:54:54 +0530196 # Wait for 1min after openolt adapter is restarted
197 Sleep 60s
198 Run Keyword If ${has_dataplane} Clean Up Linux
199 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
200 Run Keyword and Ignore Error Collect Logs
201 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
202 Log ${podStatusOutput}
203 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
204 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
205 Log to console Pod ${podName} restarted and sanity checks passed successfully
206
207Verify openolt adapter restart before subscriber provisioning for DT
208 [Documentation] Deploys an device instance and waits for it to authenticate. After
209 ... authentication is successful the rw-core deployment is scaled to 0 instances to
210 ... simulate a POD crash. The test then scales the rw-core back to a single instance
211 ... and configures ONOS for access. The test succeeds if the device is able to
212 ... complete the DHCP sequence.
213 [Tags] functionalDt olt-adapter-restart-Dt
214 [Setup] Start Logging OltAdapterRestart-Dt
215 #... AND Clear All Devices Then Create New Device
216 [Teardown] Run Keywords Collect Logs
217 ... AND Stop Logging OltAdapterRestart-Dt
218 # Add OLT and perform sanity test
219 #setup
220 Run Keyword If ${has_dataplane} Clean Up Linux
221 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
222 Set Global Variable ${of_id}
223
224 FOR ${I} IN RANGE 0 ${num_onus}
225 ${src}= Set Variable ${hosts.src[${I}]}
226 ${dst}= Set Variable ${hosts.dst[${I}]}
227 ${onu_device_id}= Get Device ID From SN ${src['onu']}
228 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
229 ... ${of_id}
230 # Bring up the device and verify it authenticates
231 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
232 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
233 END
234 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530235 Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 0
236 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha open-olt-voltha-adapter-openolt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530237 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530238 Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530239 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530240 ... Check Expected Available Deployment Replicas voltha open-olt-voltha-adapter-openolt 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530241
242 # Ensure the device is available in ONOS, this represents system connectivity being restored
243 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
244 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
245
246 FOR ${I} IN RANGE 0 ${num_onus}
247 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
248 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
249 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
250 # Verify subscriber access flows are added for the ONU port
251 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
252 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
253 ... ${onu_port} ${nni_port} ${src['s_tag']}
254 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
255 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
256 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
257 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
258 ... ${dst['container_name']}
259 END
260
261Verify restart ofagent container after subscriber is provisioned for DT
262 [Documentation] Restart ofagent container after VOLTHA is operational.
263 ... Prerequisite : ONUs are authenticated and pingable.
264 [Tags] functionalDt ofagentRestart-Dt
265 [Setup] Start Logging ofagentRestart-Dt
266 [Teardown] Run Keywords Collect Logs
267 ... AND Stop Logging ofagentRestart-Dt
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530268 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530269 # set timeout value
270 ${waitforRestart} Set Variable 120s
271 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
272 Log ${podStatusOutput}
273 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
274 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530275 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530276 Sleep 60s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530277 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
278 ... app ${podName} Running
Hardik Windlass9df139e2020-04-24 14:54:54 +0530279 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
280 Run Keyword If ${has_dataplane} Clean Up Linux
281 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
282 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
283 Log ${podStatusOutput}
284 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
285 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
286 # Scale Down the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530287 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530288 Sleep 30s
289 FOR ${I} IN RANGE 0 ${num_onus}
290 ${src}= Set Variable ${hosts.src[${I}]}
291 ${dst}= Set Variable ${hosts.dst[${I}]}
292 Run Keyword and Ignore Error Collect Logs
293 ${onu_device_id}= Get Device ID From SN ${src['onu']}
294 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
295 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
296 # Verify ONU state in voltha
297 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
298 ... ENABLED ACTIVE REACHABLE
299 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
300 # Check ONU port is Disabled in ONOS
301 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700302 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530303 # Verify subscriber access flows are added for the ONU port
304 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
305 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
306 ... ${onu_port} ${nni_port} ${src['s_tag']}
307 # Verify Ping
308 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
309 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
310 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
311 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
312 Run Keyword and Ignore Error Collect Logs
313 END
314 # Scale Up the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530315 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530316 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
317 ... Running
318 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
319 Run Keyword If ${has_dataplane} Clean Up Linux
320 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
321 Log to console Pod ${podName} restarted and sanity checks passed successfully
322
323Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
324 [Documentation] Deploys an device instance and waits for it to authenticate. After
325 ... authentication is successful the rw-core deployment is scaled to 0 instances to
326 ... simulate a POD crash. The test then scales the rw-core back to a single instance
327 ... and configures ONOS for access. The test succeeds if the device is able to
328 ... complete the DHCP sequence.
329 [Tags] functionalDt rwcore-restart-Dt
330 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
331 ... AND Clear All Devices Then Create New Device
332 [Teardown] Run Keywords Collect Logs
333 ... AND Stop Logging RwCoreFailAndRestart-Dt
334 #... AND Delete Device and Verify
335 Run Keyword and Ignore Error Collect Logs
336 Run Keyword If ${has_dataplane} Clean Up Linux
337 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
338 Set Global Variable ${of_id}
339 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
340 ... Get NNI Port in ONOS ${of_id}
341 Set Global Variable ${nni_port}
342 FOR ${I} IN RANGE 0 ${num_onus}
343 ${src}= Set Variable ${hosts.src[${I}]}
344 ${dst}= Set Variable ${hosts.dst[${I}]}
345 ${onu_device_id}= Get Device ID From SN ${src['onu']}
346 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
347 ... ${of_id}
348 # Bring up the device and verify it authenticates
349 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
350 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded
351 END
352
353 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530354 Scale K8s Deployment voltha voltha-voltha-rw-core 0
355 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530356 # Ensure the ofagent POD goes "not-ready" as expected
357 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530358 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530359 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530360 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530361 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530362 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530363 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530364 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530365 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
366 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700367 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530368 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
369 # represents system connectivity being restored
370 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
371 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
372
373 FOR ${I} IN RANGE 0 ${num_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700374 ${src}= Set Variable ${hosts.src[${I}]}
375 ${dst}= Set Variable ${hosts.dst[${I}]}
376 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
377 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530378 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
379 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
380 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
381 # Verify subscriber access flows are added for the ONU port
382 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
383 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
384 ... ${onu_port} ${nni_port} ${src['s_tag']}
385 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
386 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
387 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
388 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
389 ... ${dst['container_name']}
390 END
391
Hemadf003682020-04-28 21:22:22 +0530392Verify OLT Soft Reboot for DT
393 [Documentation] Test soft reboot of the OLT using voltctl command
394 [Tags] VOL-2818 OLTSoftRebootDt notready
395 [Setup] Start Logging OLTSoftRebootDt
396 #... AND Setup
397 [Teardown] Run Keywords Collect Logs
398 ... AND Stop Logging OLTSoftRebootDt
399 #... AND Delete Device and Verify
400 ## Performing Sanity Test to make sure subscribers are all DHCP and pingable
401 #Run Keyword If ${has_dataplane} Clean Up Linux
402 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
403 # Reboot the OLT using "voltctl device reboot" command
404 Reboot Device ${olt_device_id}
405 Run Keyword And Ignore Error Collect Logs
406 #Verify that ping fails
407 FOR ${I} IN RANGE 0 ${num_onus}
408 ${src}= Set Variable ${hosts.src[${I}]}
409 ${dst}= Set Variable ${hosts.dst[${I}]}
410 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
411 ... Wait Until Keyword Succeeds 60s 2s
412 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
413 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
414 END
415 # Wait for the OLT to come back up
416 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
417 ... Check Remote System Reachability True ${olt_ip}
418 # Waiting extra time for the ONUs to come up
419 Sleep 60s
420 # Check OLT states
421 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE REACHABLE
422 ... ${olt_serial_number}
423 Run Keyword And Ignore Error Collect Logs
424 #Check after reboot that ONUs are active, DHCP and pingable
425 Run Keyword If ${has_dataplane} Clean Up Linux
426 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
427
Hema8c1a0322020-05-26 16:02:32 +0530428Verify ONU Soft Reboot for DT
429 [Documentation] Test soft reboot of the ONU using voltctl command
430 [Tags] VOL-2820 ONUSoftRebootDt notready
431 [Setup] Start Logging ONUSoftRebootDt
432 #... AND Setup
433 [Teardown] Run Keywords Collect Logs
434 ... AND Stop Logging ONUSoftRebootDt
435 #... AND Delete Device and Verify
436 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
437 #Run Keyword If ${has_dataplane} Clean Up Linux
438 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
439 #Reboot the ONU and verify that ping fails
440 FOR ${I} IN RANGE 0 ${num_onus}
441 ${src}= Set Variable ${hosts.src[${I}]}
442 ${dst}= Set Variable ${hosts.dst[${I}]}
443 ${onu_device_id}= Get Device ID From SN ${src['onu']}
444 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
445 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
446 Reboot Device ${onu_device_id}
447 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
448 ... Wait Until Keyword Succeeds 60s 2s
449 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
450 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
451 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
452 # Remove Subscriber Access (To replicate DT workflow)
453 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
454 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
455 # Delete ONU Device (To replicate DT workflow)
456 Delete Device ${onu_device_id}
457 Sleep 40s
458 # Check ONU port is Enabled in ONOS
459 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700460 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hema8c1a0322020-05-26 16:02:32 +0530461 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
462 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
463 ... volt-add-subscriber-access ${of_id} ${onu_port}
464 # Verify ONU state in voltha
465 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
466 ... ENABLED ACTIVE REACHABLE
467 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
468 # Verify subscriber access flows are added for the ONU port
469 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
470 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
471 ... ${onu_port} ${nni_port} ${src['s_tag']}
472 #Run Keyword If ${has_dataplane} Clean Up Linux
473 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
474 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
475 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
476 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
477 ... ${dst['container_name']}
478 Run Keyword And Ignore Error Collect Logs
479 END
480 # Verify ONOS Flows
481 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
482 ${onos_flows_count}= Evaluate 4 * ${num_onus}
483 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
484 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
485 ... ${of_id} ${onos_flows_count}
486 # Verify VOLTHA Flows
487 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
488 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
489 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
490 ${onu_flows}= Set Variable 2
491 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
492 ${List_ONU_Serial} Create List
493 Set Suite Variable ${List_ONU_Serial}
494 Build ONU SN List ${List_ONU_Serial}
495 Log ${List_ONU_Serial}
496 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
497 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530498
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530499*** Keywords ***
500Setup Suite
501 [Documentation] Set up the test suite
502 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200503 #power_switch.robot needs it to support different vendor's power switch
504 ${switch_type}= Get Variable Value ${web_power_switch.type}
505 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530506
507Clear All Devices Then Create New Device
508 [Documentation] Remove any devices from VOLTHA and ONOS
509 # Remove all devices from voltha and nos
510 Delete All Devices and Verify
511 # Execute normal test Setup Keyword
512 Setup
513