blob: 7b9146077bfff31f1d23315effea5f8caedddcb9 [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
45# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
51${has_dataplane} True
52${teardown_device} False
53${scripts} ../../scripts
54
55# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
58*** Test Cases ***
59Verify ONU after Rebooting Physically for DT
60 [Documentation] Test the ONU functionality by physically turning on/off ONU.
61 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
62 ... Test case runs only on the PODs that are configured with PowerSwitch that
63 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
64 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053065 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +053066 [Teardown] Run Keywords Collect Logs
67 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080068 ... AND Delete All Devices and Verify
Hardik Windlassb9bdd502020-04-17 14:46:21 +053069 # Add OLT device
70 Setup
71 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
72 Run Keyword If ${has_dataplane} Clean Up Linux
73 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
74 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053075 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080076 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053077 ${src}= Set Variable ${hosts.src[${I}]}
78 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080079 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass0560f802020-05-06 23:06:53 +053080 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053081 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
82 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass7c801e62020-05-04 19:33:21 +053083 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053084 Disable Switch Outlet ${src['power_switch_port']}
85 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
86 ... Wait Until Keyword Succeeds 60s 2s
87 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
88 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +053089 # Remove Subscriber Access (To replicate DT workflow)
90 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
91 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Suchitra Vemuri1b8d7b82020-05-29 21:10:12 -070092 Sleep 10s
Hardik Windlass7c801e62020-05-04 19:33:21 +053093 # Delete ONU Device (To replicate DT workflow)
94 Delete Device ${onu_device_id}
95 Sleep 5s
96 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053097 Enable Switch Outlet ${src['power_switch_port']}
Hardik Windlass40ccffb2020-05-11 14:12:07 +053098 # Waiting extra time for the ONU to come up
99 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530100 # Check ONU port is Enabled in ONOS
101 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700102 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530103 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
104 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
105 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530106 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700107 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530108 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530109 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530110 # Verify subscriber access flows are added for the ONU port
111 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
112 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
113 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530114 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530115 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
116 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530117 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530118 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
119 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530120 Run Keyword And Ignore Error Collect Logs
121 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800122 # Verify flows for all OLTs
123 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530124 # Verify ONOS Flows
125 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800126 #${onos_flows_count}= Evaluate 4 * ${num_onus}
127 #Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
128 #... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
129 #... ${of_id} ${onos_flows_count}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530130 # Verify VOLTHA Flows
131 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800132 #${olt_flows}= Evaluate 2 * ${num_onus} + 1
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530133 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800134 #${onu_flows}= Set Variable 2
135 #Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
136 #${List_ONU_Serial} Create List
137 #Set Suite Variable ${List_ONU_Serial}
Andrea Campanella6391f842020-11-16 10:01:18 +0100138 #Build ONU SN List ${List_ONU_Serial} ${olt_serial_number}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800139 #Log ${List_ONU_Serial}
140 #Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
141 #... ${List_ONU_Serial} ${onu_flows}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530142
143Verify OLT after Rebooting Physically for DT
144 [Documentation] Test the physical reboot of the OLT
145 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
146 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
147 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530148 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530149 [Teardown] Run Keywords Collect Logs
150 ... AND Stop Logging RebootOlt_Physical_Dt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800151 ... AND Delete All Devices and Verify
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530152 # Add OLT device
153 Setup
154 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
155 Run Keyword If ${has_dataplane} Clean Up Linux
156 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
157 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800158 FOR ${I} IN RANGE 0 ${olt_count}
159 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
160 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
161 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
162 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
163 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
164 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
165 ... sudo reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
166 Run Keyword And Ignore Error Collect Logs
167 END
168 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530169 ${src}= Set Variable ${hosts.src[${I}]}
170 ${dst}= Set Variable ${hosts.dst[${I}]}
171 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
172 ... Wait Until Keyword Succeeds 60s 2s
173 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
174 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
175 END
176 # Wait for the OLT to come back up
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800177 FOR ${I} IN RANGE 0 ${olt_count}
178 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
179 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
180 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
181 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
182 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
183 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
184 ... Check Remote System Reachability True ${olt_ssh_ip}
185 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
186 ... Validate OLT Device ENABLED ACTIVE
187 ... REACHABLE ${olt_serial_number}
188 END
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530189 # Waiting extra time for the ONUs to come up
190 Sleep 60s
191 Run Keyword And Ignore Error Collect Logs
192 Run Keyword If ${has_dataplane} Clean Up Linux
193 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
194
Hardik Windlass9df139e2020-04-24 14:54:54 +0530195Verify restart openolt-adapter container after subscriber provisioning for DT
196 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
197 ... Prerequisite : ONUs are authenticated and pingable.
198 [Tags] functionalDt Restart-OpenOlt-Dt
199 [Setup] Start Logging Restart-OpenOlt-Dt
200 [Teardown] Run Keywords Collect Logs
201 ... AND Stop Logging Restart-OpenOlt-Dt
202 # Add OLT device
203 setup
204 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
205 Run Keyword If ${has_dataplane} Clean Up Linux
206 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
207 ${waitforRestart} Set Variable 120s
208 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
209 Log ${podStatusOutput}
210 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530211 ${podName} Set Variable adapter-open-olt
212 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
213 Sleep 5s
214 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
215 ... app ${podName} Running
Hardik Windlass9df139e2020-04-24 14:54:54 +0530216 # Wait for 1min after openolt adapter is restarted
217 Sleep 60s
218 Run Keyword If ${has_dataplane} Clean Up Linux
219 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
220 Run Keyword and Ignore Error Collect Logs
221 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
222 Log ${podStatusOutput}
223 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
224 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
225 Log to console Pod ${podName} restarted and sanity checks passed successfully
226
227Verify openolt adapter restart before subscriber provisioning for DT
228 [Documentation] Deploys an device instance and waits for it to authenticate. After
229 ... authentication is successful the rw-core deployment is scaled to 0 instances to
230 ... simulate a POD crash. The test then scales the rw-core back to a single instance
231 ... and configures ONOS for access. The test succeeds if the device is able to
232 ... complete the DHCP sequence.
233 [Tags] functionalDt olt-adapter-restart-Dt
234 [Setup] Start Logging OltAdapterRestart-Dt
235 #... AND Clear All Devices Then Create New Device
236 [Teardown] Run Keywords Collect Logs
237 ... AND Stop Logging OltAdapterRestart-Dt
238 # Add OLT and perform sanity test
239 #setup
240 Run Keyword If ${has_dataplane} Clean Up Linux
241 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
242 Set Global Variable ${of_id}
243
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800244 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530245 ${src}= Set Variable ${hosts.src[${I}]}
246 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800247 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530248 ${onu_device_id}= Get Device ID From SN ${src['onu']}
249 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
250 ... ${of_id}
251 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700252 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass9df139e2020-04-24 14:54:54 +0530253 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
254 END
255 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530256 Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 0
257 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha open-olt-voltha-adapter-openolt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530258 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530259 Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530260 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530261 ... Check Expected Available Deployment Replicas voltha open-olt-voltha-adapter-openolt 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530262
263 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800264 FOR ${I} IN RANGE 0 ${olt_count}
265 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
266 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
267 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
268 ... ${olt_serial_number}
269 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
270 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
271 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530272
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800273 FOR ${I} IN RANGE 0 ${num_all_onus}
274 ${src}= Set Variable ${hosts.src[${I}]}
275 ${dst}= Set Variable ${hosts.dst[${I}]}
276 ${of_id}= Get ofID From OLT List ${src['olt']}
277 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
278 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530279 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
280 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
281 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
282 # Verify subscriber access flows are added for the ONU port
283 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
284 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
285 ... ${onu_port} ${nni_port} ${src['s_tag']}
286 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
287 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
288 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
289 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
290 ... ${dst['container_name']}
291 END
292
293Verify restart ofagent container after subscriber is provisioned for DT
294 [Documentation] Restart ofagent container after VOLTHA is operational.
295 ... Prerequisite : ONUs are authenticated and pingable.
296 [Tags] functionalDt ofagentRestart-Dt
297 [Setup] Start Logging ofagentRestart-Dt
298 [Teardown] Run Keywords Collect Logs
299 ... AND Stop Logging ofagentRestart-Dt
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530300 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530301 # set timeout value
302 ${waitforRestart} Set Variable 120s
303 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
304 Log ${podStatusOutput}
305 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
306 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530307 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530308 Sleep 60s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530309 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
310 ... app ${podName} Running
Hardik Windlass9df139e2020-04-24 14:54:54 +0530311 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
312 Run Keyword If ${has_dataplane} Clean Up Linux
313 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
314 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
315 Log ${podStatusOutput}
316 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
317 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
318 # Scale Down the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530319 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530320 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800321 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530322 ${src}= Set Variable ${hosts.src[${I}]}
323 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800324 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530325 Run Keyword and Ignore Error Collect Logs
326 ${onu_device_id}= Get Device ID From SN ${src['onu']}
327 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
328 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
329 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700330 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530331 ... ENABLED ACTIVE REACHABLE
332 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
333 # Check ONU port is Disabled in ONOS
334 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700335 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530336 # Verify subscriber access flows are added for the ONU port
337 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
338 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
339 ... ${onu_port} ${nni_port} ${src['s_tag']}
340 # Verify Ping
341 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
342 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
343 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
344 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
345 Run Keyword and Ignore Error Collect Logs
346 END
347 # Scale Up the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530348 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530349 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
350 ... Running
351 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
352 Run Keyword If ${has_dataplane} Clean Up Linux
353 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
354 Log to console Pod ${podName} restarted and sanity checks passed successfully
355
356Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
357 [Documentation] Deploys an device instance and waits for it to authenticate. After
358 ... authentication is successful the rw-core deployment is scaled to 0 instances to
359 ... simulate a POD crash. The test then scales the rw-core back to a single instance
360 ... and configures ONOS for access. The test succeeds if the device is able to
361 ... complete the DHCP sequence.
362 [Tags] functionalDt rwcore-restart-Dt
363 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
364 ... AND Clear All Devices Then Create New Device
365 [Teardown] Run Keywords Collect Logs
366 ... AND Stop Logging RwCoreFailAndRestart-Dt
367 #... AND Delete Device and Verify
368 Run Keyword and Ignore Error Collect Logs
369 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800370 FOR ${I} IN RANGE 0 ${olt_count}
371 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
372 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
373 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
374 ... ${olt_serial_number}
375 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
376 ... Get NNI Port in ONOS ${of_id}
377 # Set Global Variable ${nni_port}
378 END
379 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530380 ${src}= Set Variable ${hosts.src[${I}]}
381 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800382 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530383 ${onu_device_id}= Get Device ID From SN ${src['onu']}
384 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
385 ... ${of_id}
386 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700387 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass9df139e2020-04-24 14:54:54 +0530388 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded
389 END
390
391 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530392 Scale K8s Deployment voltha voltha-voltha-rw-core 0
393 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530394 # Ensure the ofagent POD goes "not-ready" as expected
395 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530396 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530397 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530398 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530399 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530400 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530401 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530402 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530403 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
404 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700405 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530406 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
407 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800408 FOR ${I} IN RANGE 0 ${olt_count}
409 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
410 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
411 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
412 ... ${olt_serial_number}
413 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
414 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
415 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530416
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800417 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700418 ${src}= Set Variable ${hosts.src[${I}]}
419 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800420 ${of_id}= Get ofID From OLT List ${src['olt']}
Andy Bavier52094622020-05-12 15:54:24 -0700421 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
422 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530423 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
424 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
425 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
426 # Verify subscriber access flows are added for the ONU port
427 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
428 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
429 ... ${onu_port} ${nni_port} ${src['s_tag']}
430 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
431 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
432 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
433 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
434 ... ${dst['container_name']}
435 END
436
Hemadf003682020-04-28 21:22:22 +0530437Verify OLT Soft Reboot for DT
438 [Documentation] Test soft reboot of the OLT using voltctl command
439 [Tags] VOL-2818 OLTSoftRebootDt notready
440 [Setup] Start Logging OLTSoftRebootDt
441 #... AND Setup
442 [Teardown] Run Keywords Collect Logs
443 ... AND Stop Logging OLTSoftRebootDt
444 #... AND Delete Device and Verify
445 ## Performing Sanity Test to make sure subscribers are all DHCP and pingable
446 #Run Keyword If ${has_dataplane} Clean Up Linux
447 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
448 # Reboot the OLT using "voltctl device reboot" command
449 Reboot Device ${olt_device_id}
450 Run Keyword And Ignore Error Collect Logs
451 #Verify that ping fails
452 FOR ${I} IN RANGE 0 ${num_onus}
453 ${src}= Set Variable ${hosts.src[${I}]}
454 ${dst}= Set Variable ${hosts.dst[${I}]}
455 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
456 ... Wait Until Keyword Succeeds 60s 2s
457 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
458 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
459 END
460 # Wait for the OLT to come back up
461 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
Suchitra Vemurib7253a52020-07-14 22:35:17 -0700462 ... Check Remote System Reachability True ${olt_ssh_ip}
Hemadf003682020-04-28 21:22:22 +0530463 # Waiting extra time for the ONUs to come up
464 Sleep 60s
465 # Check OLT states
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700466 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED ACTIVE REACHABLE
Hemadf003682020-04-28 21:22:22 +0530467 ... ${olt_serial_number}
468 Run Keyword And Ignore Error Collect Logs
469 #Check after reboot that ONUs are active, DHCP and pingable
470 Run Keyword If ${has_dataplane} Clean Up Linux
471 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
472
Hema8c1a0322020-05-26 16:02:32 +0530473Verify ONU Soft Reboot for DT
474 [Documentation] Test soft reboot of the ONU using voltctl command
475 [Tags] VOL-2820 ONUSoftRebootDt notready
476 [Setup] Start Logging ONUSoftRebootDt
477 #... AND Setup
478 [Teardown] Run Keywords Collect Logs
479 ... AND Stop Logging ONUSoftRebootDt
480 #... AND Delete Device and Verify
481 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
482 #Run Keyword If ${has_dataplane} Clean Up Linux
483 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
484 #Reboot the ONU and verify that ping fails
485 FOR ${I} IN RANGE 0 ${num_onus}
486 ${src}= Set Variable ${hosts.src[${I}]}
487 ${dst}= Set Variable ${hosts.dst[${I}]}
488 ${onu_device_id}= Get Device ID From SN ${src['onu']}
489 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
490 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
491 Reboot Device ${onu_device_id}
492 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
493 ... Wait Until Keyword Succeeds 60s 2s
494 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
495 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
496 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
497 # Remove Subscriber Access (To replicate DT workflow)
498 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
499 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
500 # Delete ONU Device (To replicate DT workflow)
501 Delete Device ${onu_device_id}
502 Sleep 40s
503 # Check ONU port is Enabled in ONOS
504 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700505 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hema8c1a0322020-05-26 16:02:32 +0530506 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
507 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
508 ... volt-add-subscriber-access ${of_id} ${onu_port}
509 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700510 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hema8c1a0322020-05-26 16:02:32 +0530511 ... ENABLED ACTIVE REACHABLE
512 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
513 # Verify subscriber access flows are added for the ONU port
514 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
515 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
516 ... ${onu_port} ${nni_port} ${src['s_tag']}
517 #Run Keyword If ${has_dataplane} Clean Up Linux
518 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
519 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
520 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
521 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
522 ... ${dst['container_name']}
523 Run Keyword And Ignore Error Collect Logs
524 END
525 # Verify ONOS Flows
526 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
527 ${onos_flows_count}= Evaluate 4 * ${num_onus}
528 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
529 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
530 ... ${of_id} ${onos_flows_count}
531 # Verify VOLTHA Flows
532 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
533 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
534 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
535 ${onu_flows}= Set Variable 2
536 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
537 ${List_ONU_Serial} Create List
538 Set Suite Variable ${List_ONU_Serial}
539 Build ONU SN List ${List_ONU_Serial}
540 Log ${List_ONU_Serial}
541 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
542 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530543
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530544*** Keywords ***
545Setup Suite
546 [Documentation] Set up the test suite
547 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200548 #power_switch.robot needs it to support different vendor's power switch
549 ${switch_type}= Get Variable Value ${web_power_switch.type}
550 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530551
552Clear All Devices Then Create New Device
553 [Documentation] Remove any devices from VOLTHA and ONOS
554 # Remove all devices from voltha and nos
555 Delete All Devices and Verify
556 # Execute normal test Setup Keyword
557 Setup
558