blob: 7389f5179bacdd6d21755a71fbfd2b0cdb98207a [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
45# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
51${has_dataplane} True
52${teardown_device} False
53${scripts} ../../scripts
54
55# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
58*** Test Cases ***
59Verify ONU after Rebooting Physically for DT
60 [Documentation] Test the ONU functionality by physically turning on/off ONU.
61 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
62 ... Test case runs only on the PODs that are configured with PowerSwitch that
63 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
64 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053065 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +053066 [Teardown] Run Keywords Collect Logs
67 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080068 ... AND Delete All Devices and Verify
Hardik Windlassb9bdd502020-04-17 14:46:21 +053069 # Add OLT device
70 Setup
71 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
72 Run Keyword If ${has_dataplane} Clean Up Linux
73 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
74 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053075 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080076 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053077 ${src}= Set Variable ${hosts.src[${I}]}
78 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080079 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass0560f802020-05-06 23:06:53 +053080 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053081 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
82 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass7c801e62020-05-04 19:33:21 +053083 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053084 Disable Switch Outlet ${src['power_switch_port']}
85 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
86 ... Wait Until Keyword Succeeds 60s 2s
87 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
88 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +053089 # Remove Subscriber Access (To replicate DT workflow)
90 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
91 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
92 # Delete ONU Device (To replicate DT workflow)
93 Delete Device ${onu_device_id}
94 Sleep 5s
95 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053096 Enable Switch Outlet ${src['power_switch_port']}
Hardik Windlass40ccffb2020-05-11 14:12:07 +053097 # Waiting extra time for the ONU to come up
98 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +053099 # Check ONU port is Enabled in ONOS
100 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700101 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530102 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
103 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
104 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530105 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700106 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530107 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530108 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530109 # Verify subscriber access flows are added for the ONU port
110 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
111 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
112 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530113 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530114 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
115 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530116 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530117 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
118 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530119 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800120 # Verify flows for all OLTs
121 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530122
123Verify OLT after Rebooting Physically for DT
124 [Documentation] Test the physical reboot of the OLT
125 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
126 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
127 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530128 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530129 [Teardown] Run Keywords Collect Logs
130 ... AND Stop Logging RebootOlt_Physical_Dt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800131 ... AND Delete All Devices and Verify
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530132 # Add OLT device
133 Setup
134 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
135 Run Keyword If ${has_dataplane} Clean Up Linux
136 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
137 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800138 FOR ${I} IN RANGE 0 ${olt_count}
139 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
140 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
141 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
142 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
143 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
144 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +0100145 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800146 END
147 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530148 ${src}= Set Variable ${hosts.src[${I}]}
149 ${dst}= Set Variable ${hosts.dst[${I}]}
150 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
151 ... Wait Until Keyword Succeeds 60s 2s
152 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
153 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
154 END
155 # Wait for the OLT to come back up
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800156 FOR ${I} IN RANGE 0 ${olt_count}
157 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
158 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
159 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
160 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
161 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
162 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
163 ... Check Remote System Reachability True ${olt_ssh_ip}
164 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
165 ... Validate OLT Device ENABLED ACTIVE
166 ... REACHABLE ${olt_serial_number}
167 END
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530168 # Waiting extra time for the ONUs to come up
169 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530170 Run Keyword If ${has_dataplane} Clean Up Linux
171 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
172
TorstenThiemeed3332f2021-02-02 10:30:48 +0000173Verify restart openonu-adapter container after subscriber provisioning for DT
174 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
175 ... Prerequisite : ONUs are authenticated and pingable.
176 [Tags] functionalDt Restart-OpenOnu-Dt
177 [Setup] Start Logging Restart-OpenOnu-Dt
178 [Teardown] Run Keywords Collect Logs
179 ... AND Stop Logging Restart-OpenOnu-Dt
180 ... AND Delete All Devices and Verify
181 # Add OLT device
182 Setup
183 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
184 Run Keyword If ${has_dataplane} Clean Up Linux
185 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
186 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
187 Log ${podStatusOutput}
188 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
189 ${podName} Set Variable adapter-open-onu
190 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
TorstenThiemeed3332f2021-02-02 10:30:48 +0000191 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
192 ... app ${podName} Running
193 # Wait for 1 min after openonu adapter is restarted
194 Sleep 60s
195 Run Keyword If ${has_dataplane} Clean Up Linux
196 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000197 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
198 Log ${podStatusOutput}
199 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
200 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
201 Log to console Pod ${podName} restarted and sanity checks passed successfully
202
Hardik Windlass9df139e2020-04-24 14:54:54 +0530203Verify restart openolt-adapter container after subscriber provisioning for DT
204 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
205 ... Prerequisite : ONUs are authenticated and pingable.
206 [Tags] functionalDt Restart-OpenOlt-Dt
207 [Setup] Start Logging Restart-OpenOlt-Dt
208 [Teardown] Run Keywords Collect Logs
209 ... AND Stop Logging Restart-OpenOlt-Dt
210 # Add OLT device
211 setup
212 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
213 Run Keyword If ${has_dataplane} Clean Up Linux
214 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530215 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
216 Log ${podStatusOutput}
217 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Andrea Campanella3dcce272021-01-15 16:04:47 +0100218 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530219 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530220 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
221 ... app ${podName} Running
TorstenThiemeed3332f2021-02-02 10:30:48 +0000222 # Wait for 1 min after openolt adapter is restarted
Hardik Windlass9df139e2020-04-24 14:54:54 +0530223 Sleep 60s
224 Run Keyword If ${has_dataplane} Clean Up Linux
225 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530226 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
227 Log ${podStatusOutput}
228 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
229 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
230 Log to console Pod ${podName} restarted and sanity checks passed successfully
231
232Verify openolt adapter restart before subscriber provisioning for DT
233 [Documentation] Deploys an device instance and waits for it to authenticate. After
234 ... authentication is successful the rw-core deployment is scaled to 0 instances to
235 ... simulate a POD crash. The test then scales the rw-core back to a single instance
236 ... and configures ONOS for access. The test succeeds if the device is able to
237 ... complete the DHCP sequence.
238 [Tags] functionalDt olt-adapter-restart-Dt
239 [Setup] Start Logging OltAdapterRestart-Dt
240 #... AND Clear All Devices Then Create New Device
241 [Teardown] Run Keywords Collect Logs
242 ... AND Stop Logging OltAdapterRestart-Dt
243 # Add OLT and perform sanity test
244 #setup
245 Run Keyword If ${has_dataplane} Clean Up Linux
246 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
247 Set Global Variable ${of_id}
248
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800249 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530250 ${src}= Set Variable ${hosts.src[${I}]}
251 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800252 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530253 ${onu_device_id}= Get Device ID From SN ${src['onu']}
254 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
255 ... ${of_id}
256 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700257 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass9df139e2020-04-24 14:54:54 +0530258 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
259 END
260 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Andrea Campanella3dcce272021-01-15 16:04:47 +0100261 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
262 Wait Until Keyword Succeeds ${timeout} 2s Pods Do Not Exist By Label ${NAMESPACE} app
263 ... ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530264 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Andrea Campanella3dcce272021-01-15 16:04:47 +0100265 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530266 Wait Until Keyword Succeeds ${timeout} 2s
Andrea Campanella3dcce272021-01-15 16:04:47 +0100267 ... Check Expected Available Deployment Replicas By Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530268
269 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800270 FOR ${I} IN RANGE 0 ${olt_count}
271 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
272 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
273 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
274 ... ${olt_serial_number}
275 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
276 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
277 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530278
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800279 FOR ${I} IN RANGE 0 ${num_all_onus}
280 ${src}= Set Variable ${hosts.src[${I}]}
281 ${dst}= Set Variable ${hosts.dst[${I}]}
282 ${of_id}= Get ofID From OLT List ${src['olt']}
283 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
284 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530285 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
286 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
287 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
288 # Verify subscriber access flows are added for the ONU port
289 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
290 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
291 ... ${onu_port} ${nni_port} ${src['s_tag']}
292 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
293 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
294 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
295 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
296 ... ${dst['container_name']}
297 END
298
299Verify restart ofagent container after subscriber is provisioned for DT
300 [Documentation] Restart ofagent container after VOLTHA is operational.
301 ... Prerequisite : ONUs are authenticated and pingable.
302 [Tags] functionalDt ofagentRestart-Dt
303 [Setup] Start Logging ofagentRestart-Dt
304 [Teardown] Run Keywords Collect Logs
305 ... AND Stop Logging ofagentRestart-Dt
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530306 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530307 # set timeout value
308 ${waitforRestart} Set Variable 120s
309 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
310 Log ${podStatusOutput}
311 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
312 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530313 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530314 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
315 ... app ${podName} Running
Hardik Windlass5e214b22021-02-26 10:37:14 +0000316 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530317 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
318 Run Keyword If ${has_dataplane} Clean Up Linux
319 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
320 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
321 Log ${podStatusOutput}
322 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
323 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
324 # Scale Down the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530325 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530326 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800327 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530328 ${src}= Set Variable ${hosts.src[${I}]}
329 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800330 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530331 ${onu_device_id}= Get Device ID From SN ${src['onu']}
332 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
333 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
334 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700335 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530336 ... ENABLED ACTIVE REACHABLE
337 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
338 # Check ONU port is Disabled in ONOS
339 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700340 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530341 # Verify subscriber access flows are added for the ONU port
342 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
343 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
344 ... ${onu_port} ${nni_port} ${src['s_tag']}
345 # Verify Ping
346 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
347 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
348 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530349 END
350 # Scale Up the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530351 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530352 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
353 ... Running
354 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
355 Run Keyword If ${has_dataplane} Clean Up Linux
356 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
357 Log to console Pod ${podName} restarted and sanity checks passed successfully
358
359Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
360 [Documentation] Deploys an device instance and waits for it to authenticate. After
361 ... authentication is successful the rw-core deployment is scaled to 0 instances to
362 ... simulate a POD crash. The test then scales the rw-core back to a single instance
363 ... and configures ONOS for access. The test succeeds if the device is able to
364 ... complete the DHCP sequence.
365 [Tags] functionalDt rwcore-restart-Dt
366 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
367 ... AND Clear All Devices Then Create New Device
368 [Teardown] Run Keywords Collect Logs
369 ... AND Stop Logging RwCoreFailAndRestart-Dt
370 #... AND Delete Device and Verify
Hardik Windlass9df139e2020-04-24 14:54:54 +0530371 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800372 FOR ${I} IN RANGE 0 ${olt_count}
373 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
374 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
375 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
376 ... ${olt_serial_number}
377 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
378 ... Get NNI Port in ONOS ${of_id}
379 # Set Global Variable ${nni_port}
380 END
381 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530382 ${src}= Set Variable ${hosts.src[${I}]}
383 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800384 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530385 ${onu_device_id}= Get Device ID From SN ${src['onu']}
386 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
387 ... ${of_id}
388 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700389 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass9df139e2020-04-24 14:54:54 +0530390 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded
391 END
392
393 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530394 Scale K8s Deployment voltha voltha-voltha-rw-core 0
395 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530396 # Ensure the ofagent POD goes "not-ready" as expected
397 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530398 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530399 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530400 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530401 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530402 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530403 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530404 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530405 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
406 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700407 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530408 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
409 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800410 FOR ${I} IN RANGE 0 ${olt_count}
411 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
412 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
413 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
414 ... ${olt_serial_number}
Girish Gowdrae4de3b32021-03-23 22:20:00 -0700415 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800416 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
417 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530418
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800419 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700420 ${src}= Set Variable ${hosts.src[${I}]}
421 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800422 ${of_id}= Get ofID From OLT List ${src['olt']}
Andy Bavier52094622020-05-12 15:54:24 -0700423 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
424 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530425 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
426 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
427 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
428 # Verify subscriber access flows are added for the ONU port
429 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
430 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
431 ... ${onu_port} ${nni_port} ${src['s_tag']}
432 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
433 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
434 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
435 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
436 ... ${dst['container_name']}
437 END
438
Hemadf003682020-04-28 21:22:22 +0530439Verify OLT Soft Reboot for DT
440 [Documentation] Test soft reboot of the OLT using voltctl command
441 [Tags] VOL-2818 OLTSoftRebootDt notready
442 [Setup] Start Logging OLTSoftRebootDt
443 #... AND Setup
444 [Teardown] Run Keywords Collect Logs
445 ... AND Stop Logging OLTSoftRebootDt
446 #... AND Delete Device and Verify
Hemadf003682020-04-28 21:22:22 +0530447 # Reboot the OLT using "voltctl device reboot" command
448 Reboot Device ${olt_device_id}
Hemadf003682020-04-28 21:22:22 +0530449 #Verify that ping fails
450 FOR ${I} IN RANGE 0 ${num_onus}
451 ${src}= Set Variable ${hosts.src[${I}]}
452 ${dst}= Set Variable ${hosts.dst[${I}]}
453 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
454 ... Wait Until Keyword Succeeds 60s 2s
455 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
456 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
457 END
458 # Wait for the OLT to come back up
459 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
Suchitra Vemurib7253a52020-07-14 22:35:17 -0700460 ... Check Remote System Reachability True ${olt_ssh_ip}
Hemadf003682020-04-28 21:22:22 +0530461 # Waiting extra time for the ONUs to come up
462 Sleep 60s
463 # Check OLT states
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700464 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED ACTIVE REACHABLE
Hemadf003682020-04-28 21:22:22 +0530465 ... ${olt_serial_number}
Hemadf003682020-04-28 21:22:22 +0530466 #Check after reboot that ONUs are active, DHCP and pingable
467 Run Keyword If ${has_dataplane} Clean Up Linux
468 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
469
Hema8c1a0322020-05-26 16:02:32 +0530470Verify ONU Soft Reboot for DT
471 [Documentation] Test soft reboot of the ONU using voltctl command
472 [Tags] VOL-2820 ONUSoftRebootDt notready
473 [Setup] Start Logging ONUSoftRebootDt
474 #... AND Setup
475 [Teardown] Run Keywords Collect Logs
476 ... AND Stop Logging ONUSoftRebootDt
477 #... AND Delete Device and Verify
Hema8c1a0322020-05-26 16:02:32 +0530478 #Reboot the ONU and verify that ping fails
479 FOR ${I} IN RANGE 0 ${num_onus}
480 ${src}= Set Variable ${hosts.src[${I}]}
481 ${dst}= Set Variable ${hosts.dst[${I}]}
482 ${onu_device_id}= Get Device ID From SN ${src['onu']}
483 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
484 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
485 Reboot Device ${onu_device_id}
486 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
487 ... Wait Until Keyword Succeeds 60s 2s
488 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
489 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
490 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
491 # Remove Subscriber Access (To replicate DT workflow)
492 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
493 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
494 # Delete ONU Device (To replicate DT workflow)
495 Delete Device ${onu_device_id}
496 Sleep 40s
497 # Check ONU port is Enabled in ONOS
498 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700499 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hema8c1a0322020-05-26 16:02:32 +0530500 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
501 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
502 ... volt-add-subscriber-access ${of_id} ${onu_port}
503 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700504 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hema8c1a0322020-05-26 16:02:32 +0530505 ... ENABLED ACTIVE REACHABLE
506 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
507 # Verify subscriber access flows are added for the ONU port
508 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
509 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
510 ... ${onu_port} ${nni_port} ${src['s_tag']}
511 #Run Keyword If ${has_dataplane} Clean Up Linux
512 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
513 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
514 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
515 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
516 ... ${dst['container_name']}
Hema8c1a0322020-05-26 16:02:32 +0530517 END
518 # Verify ONOS Flows
519 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
520 ${onos_flows_count}= Evaluate 4 * ${num_onus}
521 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
522 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
523 ... ${of_id} ${onos_flows_count}
524 # Verify VOLTHA Flows
525 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
526 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
527 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
528 ${onu_flows}= Set Variable 2
529 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
530 ${List_ONU_Serial} Create List
531 Set Suite Variable ${List_ONU_Serial}
532 Build ONU SN List ${List_ONU_Serial}
533 Log ${List_ONU_Serial}
534 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
535 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530536
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530537*** Keywords ***
538Setup Suite
539 [Documentation] Set up the test suite
540 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200541 #power_switch.robot needs it to support different vendor's power switch
542 ${switch_type}= Get Variable Value ${web_power_switch.type}
543 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530544
545Clear All Devices Then Create New Device
546 [Documentation] Remove any devices from VOLTHA and ONOS
547 # Remove all devices from voltha and nos
548 Delete All Devices and Verify
549 # Execute normal test Setup Keyword
550 Setup
551