blob: 767fede497cf257f80ed7e28f65de5d63e765fc7 [file] [log] [blame]
Gayathri.Selvan283a63c2020-01-23 04:09:18 +00001# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000014
15*** Settings ***
HungWei Chiu87edb1f2020-01-28 00:10:58 +000016Documentation Test various failure scenarios
HungWei Chiu1408fb92020-03-03 19:43:30 -050017Suite Setup Setup Suite
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000018Test Setup Setup
19Test Teardown Teardown
20#Suite Teardown Teardown Suite
21Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../libraries/DependencyLibrary.py
27Resource ../../libraries/onos.robot
28Resource ../../libraries/voltctl.robot
29Resource ../../libraries/voltha.robot
30Resource ../../libraries/utils.robot
31Resource ../../libraries/k8s.robot
32Resource ../../variables/variables.robot
HungWei Chiu87edb1f2020-01-28 00:10:58 +000033Resource ../../libraries/power_switch.robot
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000034
35*** Variables ***
36${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
43${NAMESPACE} voltha
Suchitra Vemurica1c5d32020-04-09 18:45:29 -070044${DEFAULTSPACE} default
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000045# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
Suchitra Vemuri109feb12020-01-23 16:25:02 -080051${has_dataplane} True
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000052${teardown_device} False
53${scripts} ../../scripts
54
Andy Bavierabeba262020-02-07 16:22:16 -070055# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000058*** Test Cases ***
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080059Verify ONU after rebooting physically
60 [Documentation] Test the ONU functionality by physically turning on/off ONU.
61 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
62 ... Test case runs only on the PODs that are configured with PowerSwitch that
63 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
64 ... VOL-2634
65 [Tags] functional PowerSwitch
Andy Baviere187eda2020-04-20 15:00:02 -070066 [Setup] Start Logging ONUreboot_PowerSwitch
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080067 [Teardown] Run Keywords Collect Logs
Suchitra Vemuri9a6dd6d2020-02-28 17:46:26 -080068 ... AND Stop Logging ONUreboot_PowerSwitch
Suchitra Vemuri0656bc22020-03-21 19:07:49 -070069 ... AND Delete Device and Verify
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080070 # Add OLT device
71 setup
72 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
73 Run Keyword If ${has_dataplane} Clean Up Linux
74 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
75 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
76 FOR ${I} IN RANGE 0 ${num_onus}
77 ${src}= Set Variable ${hosts.src[${I}]}
78 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlass0560f802020-05-06 23:06:53 +053079 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080080 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
81 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
82 Disable Switch Outlet ${src['power_switch_port']}
83 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
84 ... Wait Until Keyword Succeeds 60s 2s
85 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
86 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +053087 # Remove Subscriber Access (To replicate ATT workflow)
88 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
89 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080090
91 Enable Switch Outlet ${src['power_switch_port']}
92 # Check ONU port is Enabled in ONOS
93 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -070094 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080095 # Verify EAPOL flows are added for the ONU port
96 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -080097 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080098 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -070099 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800100 ... ENABLED ACTIVE REACHABLE
101 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass7c801e62020-05-04 19:33:21 +0530102 Run Keyword If ${has_dataplane} Clean Up Linux
103 # Perform Authentication
104 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
105 ... /tmp/wpa ${src['dp_iface_name']} log
106 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
107 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
108 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
109 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
110 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
111 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
112 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
113 ... volt-add-subscriber-access ${of_id} ${onu_port}
114 # Verify that no pending flows exist for the ONU port
115 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
116 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530117 # Verify subscriber access flows are added for the ONU port
118 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
119 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
120 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530121 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
122 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800123 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530124 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
125 ... ${dst['container_name']}
126 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
127 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
128 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800129 Run Keyword And Ignore Error Collect Logs
130 END
Suchitra Vemurid7096132020-02-21 14:50:47 -0800131 # Deleting OLT after tests completes independently (as this test doesn't not run on each POD)
Suchitra Vemuri0656bc22020-03-21 19:07:49 -0700132 #Run Keyword If ${has_dataplane} Delete Device and Verify
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800133
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700134Verify OLT after rebooting physically
135 [Documentation] Test the physical reboot of the OLT
136 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
137 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
138 ... VOL-1956
139 [Tags] functional PhysicalOLTReboot
Andy Baviere187eda2020-04-20 15:00:02 -0700140 [Setup] Start Logging PhysicalOLTReboot
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700141 [Teardown] Run Keywords Collect Logs
142 ... AND Stop Logging PhysicalOLTReboot
Suchitra Vemurib3a92e42020-03-21 16:49:57 -0700143 ... AND Delete Device and Verify
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700144 # Add OLT device
145 setup
146 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
147 Run Keyword If ${has_dataplane} Clean Up Linux
148 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
149 # Reboot the OLT from the OLT CLI
150 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Suchitra Vemurib7253a52020-07-14 22:35:17 -0700151 ... sudo reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700152 Run Keyword And Ignore Error Collect Logs
153 FOR ${I} IN RANGE 0 ${num_onus}
154 ${src}= Set Variable ${hosts.src[${I}]}
155 ${dst}= Set Variable ${hosts.dst[${I}]}
156 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
157 ... Wait Until Keyword Succeeds 60s 2s
158 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
159 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
160 END
161 # Wait for the OLT to come back up
Andy Bavier8da0e132020-04-13 10:25:16 -0700162 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
Suchitra Vemurib7253a52020-07-14 22:35:17 -0700163 ... Check Remote System Reachability True ${olt_ssh_ip}
Suchitra Vemuri44219d02020-07-29 09:21:32 -0700164 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
165 ... Validate OLT Device ENABLED ACTIVE
166 ... REACHABLE ${olt_serial_number}
Suchitra Vemuri64db9652020-03-22 21:11:13 -0700167 # Waiting extra time for the ONUs to come up
168 Sleep 60s
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700169 Run Keyword And Ignore Error Collect Logs
170 Run Keyword If ${has_dataplane} Clean Up Linux
171 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
172 # Deleting OLT after test completes
Suchitra Vemurib3a92e42020-03-21 16:49:57 -0700173 #Run Keyword If ${has_dataplane} Delete Device and Verify
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700174
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700175Verify restart openolt-adapter container after subscriber provisioning
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800176 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
177 ... Prerequisite : ONUs are authenticated and pingable.
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800178 [Tags] functional VOL-1958 Restart-OpenOlt released
Andy Baviere187eda2020-04-20 15:00:02 -0700179 [Setup] Start Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800180 [Teardown] Run Keywords Collect Logs
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800181 ... AND Stop Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800182 # Add OLT device
183 setup
184 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
185 Run Keyword If ${has_dataplane} Clean Up Linux
186 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
187 ${waitforRestart} Set Variable 120s
188 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
189 Log ${podStatusOutput}
190 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530191 ${podName} Set Variable adapter-open-olt
192 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
193 Sleep 5s
194 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
195 ... app ${podName} Running
Suchitra Vemuri312bfe02020-03-16 16:11:34 -0700196 # Wait for 1min after openolt adapter is restarted
197 Sleep 60s
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800198 Repeat Sanity Test
Suchitra Vemurid7096132020-02-21 14:50:47 -0800199 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800200 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
201 Log ${podStatusOutput}
202 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
203 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
204 Log to console Pod ${podName} restarted and sanity checks passed successfully
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800205
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800206Check OLT/ONU Authentication After Radius Pod Restart
207 [Documentation] After radius restart, triggers reassociation, checks status and
208 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
209 ... wpa supplicant is running in background hence it is recommended to remove
210 ... teardown from previous test or uncomment 'Teardown None'.
211 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
212 [Tags] functional RadiusRestart released
Andy Baviere187eda2020-04-20 15:00:02 -0700213 [Setup] Start Logging RadiusRestart
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800214 [Teardown] Run Keywords Collect Logs
215 ... AND Stop Logging RadiusRestart
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800216 ${waitforRestart} Set Variable 120s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530217 ${podName} Set Variable radius
218 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${DEFAULTSPACE} app ${podName}
219 Sleep 5s
220 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pods Status By Label ${DEFAULTSPACE}
221 ... app ${podName} Running
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800222 FOR ${I} IN RANGE 0 ${num_onus}
223 ${src}= Set Variable ${hosts.src[${I}]}
224 ${dst}= Set Variable ${hosts.dst[${I}]}
225 ${onu_device_id}= Get Device ID From SN ${src['onu']}
226 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
227 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
228 Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800229 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800230 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
231 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
232 ... ${src['ip']} ${src['user']} ${src['pass']}
233 ... ${src['container_type']} ${src['container_name']}
ubuntu6b6e7d42020-03-02 12:35:42 -0800234 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800235 ... ${ONOS_SSH_PORT} ${onu_port}
236 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
237 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
238 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
239 ... ${src['ip']} ${src['user']} ${src['pass']}
240 ... ${src['container_type']} ${src['container_name']}
241 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
242 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
243 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800244 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800245 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
246 Run Keyword and Ignore Error Collect Logs
247 END
248
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700249Verify openolt adapter restart before subscriber provisioning
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800250 [Documentation] Deploys an device instance and waits for it to authenticate. After
251 ... authentication is successful the rw-core deployment is scaled to 0 instances to
252 ... simulate a POD crash. The test then scales the rw-core back to a single instance
253 ... and configures ONOS for access. The test succeeds if the device is able to
254 ... complete the DHCP sequence.
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700255 [Tags] functional olt-adapter-restart
Andy Baviere187eda2020-04-20 15:00:02 -0700256 [Setup] Start Logging OltAdapterRestart
Suchitra Vemuria0c086f2020-03-29 19:14:13 -0700257 #... AND Clear All Devices Then Create New Device
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800258 [Teardown] Run Keywords Collect Logs
259 ... AND Stop Logging OltAdapterRestart
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700260 # Add OLT and perform sanity test
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700261 #setup
Suchitra Vemuri91537c82020-03-29 15:03:26 -0700262 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700263 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800264 Set Global Variable ${of_id}
265
266 FOR ${I} IN RANGE 0 ${num_onus}
267 ${src}= Set Variable ${hosts.src[${I}]}
268 ${dst}= Set Variable ${hosts.dst[${I}]}
269 ${onu_device_id}= Get Device ID From SN ${src['onu']}
270 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
271 ... ${of_id}
272
273 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700274 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800275 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
ubuntu6b6e7d42020-03-02 12:35:42 -0800276 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800277 ... ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier84834d42020-02-25 13:49:50 -0700278 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
279 ... /tmp/wpa ${src['dp_iface_name']} log
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800280 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
281 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700282 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
ubuntu6b6e7d42020-03-02 12:35:42 -0800283 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800284 ... ${ONOS_SSH_PORT} ${onu_port}
285 END
286 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530287 Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 0
288 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha open-olt-voltha-adapter-openolt
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800289 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530290 Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 1
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800291 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530292 ... Check Expected Available Deployment Replicas voltha open-olt-voltha-adapter-openolt 1
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800293
294 # Ensure the device is available in ONOS, this represents system connectivity being restored
295 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
ubuntu6b6e7d42020-03-02 12:35:42 -0800296 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800297
298 FOR ${I} IN RANGE 0 ${num_onus}
299 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
ubuntu6b6e7d42020-03-02 12:35:42 -0800300 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800301 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530302 # Verify subscriber access flows are added for the ONU port
303 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
304 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
305 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800306 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
307 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
308 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
309 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
310 ... ${dst['container_name']}
311 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800312 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800313 END
314
hwchiu14c05102020-03-27 00:44:30 +0000315Verify restart ofagent container after subscriber is provisioned
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000316 [Documentation] Restart ofagent container after VOLTHA is operational.
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000317 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass7a846662020-03-19 20:42:59 +0530318 [Tags] functional VOL-2409 ofagentRestart
Andy Baviere187eda2020-04-20 15:00:02 -0700319 [Setup] Start Logging ofagentRestart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700320 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700321 ... AND Stop Logging ofagentRestart
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530322 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Suchitra Vemuriceb002a2020-03-25 13:36:21 -0700323 # set timeout value
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000324 ${waitforRestart} Set Variable 120s
Andy Bavier4a8450e2020-02-04 08:58:37 -0700325 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000326 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700327 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530328 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530329 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000330 Sleep 60s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530331 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
332 ... app ${podName} Running
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530333 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
334 Run Keyword If ${has_dataplane} Clean Up Linux
335 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Andy Bavier4a8450e2020-02-04 08:58:37 -0700336 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000337 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700338 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000339 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530340 # Scale Down the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530341 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530342 Sleep 30s
343 FOR ${I} IN RANGE 0 ${num_onus}
344 ${src}= Set Variable ${hosts.src[${I}]}
345 ${dst}= Set Variable ${hosts.dst[${I}]}
346 Run Keyword and Ignore Error Collect Logs
347 ${onu_device_id}= Get Device ID From SN ${src['onu']}
348 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
349 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
350 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700351 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530352 ... ENABLED ACTIVE REACHABLE
353 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
354 # Check ONU port is Disabled in ONOS
355 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700356 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530357 # Verify EAPOL flows are present for the ONU port
358 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
359 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
360 # Verify ONU in AAA-Users
361 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
362 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
363 # Verify DHCP-Allocations
364 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
365 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530366 # Verify subscriber access flows are added for the ONU port
367 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
368 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
369 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530370 # Verify Ping
371 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
372 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
373 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
374 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
375 Run Keyword and Ignore Error Collect Logs
376 END
377 # Scale Up the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530378 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530379 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
380 ... Running
381 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
382 Run Keyword If ${has_dataplane} Clean Up Linux
383 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
384 Log to console Pod ${podName} restarted and sanity checks passed successfully
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000385
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000386Check ONU adapter crash not forcing authentication again
387 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
388 ... message count to make sure auth not started again and validates EAP status and ping.
389 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
hwchiu14c05102020-03-27 00:44:30 +0000390 [Tags] functional ONUAdaptCrash
Andy Baviere187eda2020-04-20 15:00:02 -0700391 [Setup] Start Logging ONUAdaptCrash
Andy Bavierabeba262020-02-07 16:22:16 -0700392 [Teardown] Run Keywords Collect Logs
393 ... AND Stop Logging ONUAdaptCrash
hwchiu14c05102020-03-27 00:44:30 +0000394 # Wait for adapter to resync
395 Sleep 60s
396 # Restart the onu
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530397 ${podName} Set Variable adapter-open-onu
Hung-Wei Chiu2bee4d42020-04-24 11:31:50 -0700398 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
hwchiu14c05102020-03-27 00:44:30 +0000399 # Validate ONU Ports
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000400 FOR ${I} IN RANGE 0 ${num_onus}
401 ${src}= Set Variable ${hosts.src[${I}]}
402 ${dst}= Set Variable ${hosts.dst[${I}]}
403 ${onu_device_id}= Get Device ID From SN ${src['onu']}
404 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
405 ... ${of_id}
hwchiu14c05102020-03-27 00:44:30 +0000406 Run Keyword And Continue On Failure Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700407 ... ${src['onu']}
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000408 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
409 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
410 ... ${src['container_type']} ${src['container_name']}
411 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000412 END
Hung-Wei Chiu2bee4d42020-04-24 11:31:50 -0700413 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
414 ... app ${podName} Running
hwchiu14c05102020-03-27 00:44:30 +0000415 # Wait for adapter to resync
416 Sleep 60s
Andy Bavier8da0e132020-04-13 10:25:16 -0700417 Run Keyword If ${has_dataplane} Clean Up Linux
hwchiu14c05102020-03-27 00:44:30 +0000418 # Perform all steps in Sanity Test except the subscriber addition
419 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
420 Set Global Variable ${of_id}
421 FOR ${I} IN RANGE 0 ${num_onus}
422 ${src}= Set Variable ${hosts.src[${I}]}
423 ${dst}= Set Variable ${hosts.dst[${I}]}
424 Run Keyword and Ignore Error Collect Logs
425 ${onu_device_id}= Get Device ID From SN ${src['onu']}
426 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
427 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
428 # Check ONU port is Enabled in ONOS
429 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700430 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
hwchiu14c05102020-03-27 00:44:30 +0000431 # Verify EAPOL flows are added for the ONU port
432 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
433 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
434 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700435 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
hwchiu14c05102020-03-27 00:44:30 +0000436 ... ENABLED ACTIVE REACHABLE
437 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
438 # Perform Authentication
439 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
440 ... /tmp/wpa ${src['dp_iface_name']} log
441 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
442 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
443 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
444 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
445 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
446 # Verify that no pending flows exist for the ONU port
447 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
448 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530449 # Verify subscriber access flows are added for the ONU port
450 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
451 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
452 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
hwchiu14c05102020-03-27 00:44:30 +0000453 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
454 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
455 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
456 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
457 ... ${dst['container_name']}
458 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
459 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
460 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
461 Run Keyword and Ignore Error Collect Logs
462 END
463
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000464 Run Keyword and Ignore Error Collect Logs
465
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700466Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
467 [Documentation] Deploys an device instance and waits for it to authenticate. After
468 ... authentication is successful the rw-core deployment is scaled to 0 instances to
469 ... simulate a POD crash. The test then scales the rw-core back to a single instance
470 ... and configures ONOS for access. The test succeeds if the device is able to
471 ... complete the DHCP sequence.
472 [Tags] functional rwcore-restart
Andy Baviere187eda2020-04-20 15:00:02 -0700473 [Setup] Run Keywords Start Logging RwCoreFailAndRestart
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700474 ... AND Clear All Devices Then Create New Device
475 [Teardown] Run Keywords Collect Logs
476 ... AND Stop Logging RwCoreFailAndRestart
Suchitra Vemuri5d3383f2020-04-03 11:02:26 -0700477 #... AND Delete Device and Verify
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700478 Run Keyword and Ignore Error Collect Logs
479 Run Keyword If ${has_dataplane} Clean Up Linux
480 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
481 Set Global Variable ${of_id}
Hardik Windlasse05e4712020-04-29 10:40:42 +0530482 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
483 ... Get NNI Port in ONOS ${of_id}
484 Set Global Variable ${nni_port}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700485 FOR ${I} IN RANGE 0 ${num_onus}
486 ${src}= Set Variable ${hosts.src[${I}]}
487 ${dst}= Set Variable ${hosts.dst[${I}]}
488 ${onu_device_id}= Get Device ID From SN ${src['onu']}
489 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
490 ... ${of_id}
491 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700492 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700493 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
494 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
495 ... ${ONOS_SSH_PORT} ${onu_port}
496 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
497 ... /tmp/wpa ${src['dp_iface_name']} log
498 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
499 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
500 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
501 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
502 ... ${ONOS_SSH_PORT} ${onu_port}
503 END
504
505 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530506 Scale K8s Deployment voltha voltha-voltha-rw-core 0
507 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700508 # Ensure the ofagent POD goes "not-ready" as expected
509 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530510 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700511 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530512 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700513 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530514 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700515 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530516 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700517 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
518 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700519 Restart VOLTHA Port Forward voltha-api
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700520 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
521 # represents system connectivity being restored
522 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
523 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
524
525 FOR ${I} IN RANGE 0 ${num_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700526 ${src}= Set Variable ${hosts.src[${I}]}
527 ${dst}= Set Variable ${hosts.dst[${I}]}
528 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
529 ... ${of_id}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700530 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
531 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
532 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530533 # Verify subscriber access flows are added for the ONU port
534 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
535 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
536 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700537 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
538 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
539 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
540 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
541 ... ${dst['container_name']}
542 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
543 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
544 END
545
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700546Verify OLT Soft Reboot
547 [Documentation] Test soft reboot of the OLT using voltctl command
548 [Tags] VOL-2745 OLTSoftReboot functional
549 [Setup] Start Logging OLTSoftReboot
550 [Teardown] Run Keywords Collect Logs
551 ... AND Stop Logging OLTSoftReboot
Suchitra Vemuri6282ba12020-08-13 12:50:54 -0700552 #Delete All Devices and Verify
553 #Setup
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700554 ## Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
Suchitra Vemuri6282ba12020-08-13 12:50:54 -0700555 #Run Keyword If ${has_dataplane} Clean Up Linux
556 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700557 # Reboot the OLT using "voltctl device reboot" command
Suchitra Vemuri6282ba12020-08-13 12:50:54 -0700558 Wait Until Keyword Succeeds 360s 5s
559 ... Validate OLT Device ENABLED ACTIVE
560 ... REACHABLE ${olt_serial_number}
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700561 Reboot Device ${olt_device_id}
562 Run Keyword And Ignore Error Collect Logs
563 #Verify that ping fails
564 FOR ${I} IN RANGE 0 ${num_onus}
565 ${src}= Set Variable ${hosts.src[${I}]}
566 ${dst}= Set Variable ${hosts.dst[${I}]}
567 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
568 ... Wait Until Keyword Succeeds 60s 2s
569 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
570 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
571 END
572 # Wait for the OLT to come back up
573 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
574 ... Check Remote System Reachability True ${olt_ssh_ip}
575 # Waiting extra time for the ONUs to come up
576 # Check OLT states
577 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED ACTIVE REACHABLE
578 ... ${olt_serial_number}
579 Run Keyword And Ignore Error Collect Logs
580 #Check after reboot that ONUs are active, authenticated/DHCP/pingable
581 Run Keyword If ${has_dataplane} Clean Up Linux
582 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
583
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700584Verify restart ofagent container before subscriber is provisioned
585 [Documentation] Restart ofagent container before subscriber is provisioned.
586 [Tags] functional VOL-2962 ofagentRestart2
587 [Setup] Start Logging ofagentRestart2
588 [Teardown] Run Keywords Collect Logs
Suchitra Vemuri8130be82020-09-15 16:47:58 -0700589 ... AND Stop Logging ofagentRestart2
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700590 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
591 Delete Device And Verify
592 setup
593 Run Keyword If ${has_dataplane} Clean Up Linux
594 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
595 ... Validate OLT Device ENABLED ACTIVE
596 ... REACHABLE ${olt_serial_number}
597 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
598 Set Global Variable ${of_id}
599 FOR ${I} IN RANGE 0 ${num_onus}
600 ${src}= Set Variable ${hosts.src[${I}]}
601 ${dst}= Set Variable ${hosts.dst[${I}]}
602 ${onu_device_id}= Get Device ID From SN ${src['onu']}
603 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
604 ... ${of_id}
605 # Bring up the device and verify it authenticates
606 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
607 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
608 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
609 ... ${ONOS_SSH_PORT} ${onu_port}
610 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
611 ... /tmp/wpa ${src['dp_iface_name']} log
612 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
613 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
614 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
615 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
616 ... ${ONOS_SSH_PORT} ${onu_port}
617 END
618 # Restart POD ofagent
619 ${waitforRestart} Set Variable 120s
620 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
621 Log ${podStatusOutput}
622 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
623 ${podName} Set Variable ofagent
624 Restart Pod ${NAMESPACE} ${podName}
625 Sleep 60s
626 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
627 ... Running
628 FOR ${I} IN RANGE 0 ${num_onus}
629 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
630 #Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
631 #... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
632 ${src}= Set Variable ${hosts.src[${I}]}
633 ${dst}= Set Variable ${hosts.dst[${I}]}
634 Run Keyword and Ignore Error Collect Logs
635 ${onu_device_id}= Get Device ID From SN ${src['onu']}
636 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
637 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
638 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
639 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
640 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
641 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
642 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
643 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
644 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
645 ... ${dst['container_name']}
646 # Verify DHCP-Allocations
647 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
648 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
649 Run Keyword and Ignore Error Collect Logs
650 END
651
Hema123aff52020-05-26 15:41:47 +0530652Verify ONU Soft Reboot
653 [Documentation] Test soft reboot of the ONU using voltctl command
Suchitra Vemuri8e64c542020-09-24 21:51:55 -0700654 [Tags] VOL-1957 ONUSoftReboot functional notready
Hema123aff52020-05-26 15:41:47 +0530655 [Setup] Start Logging ONUSoftReboot
656 #... AND Setup
Andy Bavier4a8450e2020-02-04 08:58:37 -0700657 [Teardown] Run Keywords Collect Logs
Hema123aff52020-05-26 15:41:47 +0530658 ... AND Stop Logging ONUSoftReboot
659 #... AND Delete Device and Verify
660 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
661 #Run Keyword If ${has_dataplane} Clean Up Linux
662 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
663 #Reboot the ONU and verify that ping fails
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000664 FOR ${I} IN RANGE 0 ${num_onus}
665 ${src}= Set Variable ${hosts.src[${I}]}
666 ${dst}= Set Variable ${hosts.dst[${I}]}
667 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800668 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
669 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hema123aff52020-05-26 15:41:47 +0530670 Reboot Device ${onu_device_id}
671 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
672 ... Wait Until Keyword Succeeds 60s 2s
673 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
674 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
675 # Remove Subscriber Access (To replicate ATT workflow)
676 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
677 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000678 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
Hema123aff52020-05-26 15:41:47 +0530679 Sleep 40s
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800680 # Check ONU port is Enabled in ONOS
681 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700682 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800683 # Verify EAPOL flows are added for the ONU port
684 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800685 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800686 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700687 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800688 ... ENABLED ACTIVE REACHABLE
689 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hema123aff52020-05-26 15:41:47 +0530690 Run Keyword If ${has_dataplane} Clean Up Linux
691 # Perform Authentication
692 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
693 ... /tmp/wpa ${src['dp_iface_name']} log
694 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
695 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
696 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
697 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
698 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
699 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
700 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
701 ... volt-add-subscriber-access ${of_id} ${onu_port}
702 # Verify that no pending flows exist for the ONU port
703 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
704 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530705 # Verify subscriber access flows are added for the ONU port
706 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
707 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
708 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Hema123aff52020-05-26 15:41:47 +0530709 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
710 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800711 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hema123aff52020-05-26 15:41:47 +0530712 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
713 ... ${dst['container_name']}
714 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
715 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
716 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
717 Run Keyword And Ignore Error Collect Logs
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000718 END
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800719
720*** Keywords ***
721Setup Suite
722 [Documentation] Set up the test suite
723 Common Test Suite Setup
HungWei Chiu1408fb92020-03-03 19:43:30 -0500724 #power_switch.robot needs it to support different vendor's power switch
Hung-Wei Chiufcc6d8e2020-03-10 12:05:11 -0700725 ${switch_type}= Get Variable Value ${web_power_switch.type}
726 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800727
728Clear All Devices Then Create New Device
729 [Documentation] Remove any devices from VOLTHA and ONOS
730 # Remove all devices from voltha and nos
731 Delete All Devices and Verify
732 # Execute normal test Setup Keyword
733 Setup