blob: f91c8ff84d3d30aec9bf16d1daad14268dbf6312 [file] [log] [blame]
Gayathri.Selvan283a63c2020-01-23 04:09:18 +00001# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000014
15*** Settings ***
HungWei Chiu87edb1f2020-01-28 00:10:58 +000016Documentation Test various failure scenarios
HungWei Chiu1408fb92020-03-03 19:43:30 -050017Suite Setup Setup Suite
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000018Test Setup Setup
19Test Teardown Teardown
20#Suite Teardown Teardown Suite
21Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../libraries/DependencyLibrary.py
27Resource ../../libraries/onos.robot
28Resource ../../libraries/voltctl.robot
29Resource ../../libraries/voltha.robot
30Resource ../../libraries/utils.robot
31Resource ../../libraries/k8s.robot
32Resource ../../variables/variables.robot
HungWei Chiu87edb1f2020-01-28 00:10:58 +000033Resource ../../libraries/power_switch.robot
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000034
35*** Variables ***
36${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
43${NAMESPACE} voltha
Suchitra Vemurica1c5d32020-04-09 18:45:29 -070044${DEFAULTSPACE} default
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000045# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
Suchitra Vemuri109feb12020-01-23 16:25:02 -080051${has_dataplane} True
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000052${teardown_device} False
53${scripts} ../../scripts
54
Andy Bavierabeba262020-02-07 16:22:16 -070055# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000058*** Test Cases ***
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080059Verify ONU after rebooting physically
60 [Documentation] Test the ONU functionality by physically turning on/off ONU.
61 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
62 ... Test case runs only on the PODs that are configured with PowerSwitch that
63 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
64 ... VOL-2634
65 [Tags] functional PowerSwitch
Andy Baviere187eda2020-04-20 15:00:02 -070066 [Setup] Start Logging ONUreboot_PowerSwitch
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080067 [Teardown] Run Keywords Collect Logs
Suchitra Vemuri9a6dd6d2020-02-28 17:46:26 -080068 ... AND Stop Logging ONUreboot_PowerSwitch
Suchitra Vemuri0656bc22020-03-21 19:07:49 -070069 ... AND Delete Device and Verify
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080070 # Add OLT device
71 setup
72 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
73 Run Keyword If ${has_dataplane} Clean Up Linux
74 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
75 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
76 FOR ${I} IN RANGE 0 ${num_onus}
77 ${src}= Set Variable ${hosts.src[${I}]}
78 ${dst}= Set Variable ${hosts.dst[${I}]}
79 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
80 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
81 Disable Switch Outlet ${src['power_switch_port']}
82 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
83 ... Wait Until Keyword Succeeds 60s 2s
84 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
85 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
86
87 Enable Switch Outlet ${src['power_switch_port']}
88 # Check ONU port is Enabled in ONOS
89 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
ubuntu6b6e7d42020-03-02 12:35:42 -080090 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080091 # Verify EAPOL flows are added for the ONU port
92 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -080093 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080094 # Verify ONU state in voltha
95 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
96 ... ENABLED ACTIVE REACHABLE
97 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass21807632020-04-14 16:24:55 +053098 # Verify subscriber access flows are added for the ONU port
99 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
100 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
101 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800102 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
103 ... Wait Until Keyword Succeeds 60s 2s
104 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
105 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
106 Run Keyword And Ignore Error Collect Logs
107 END
Suchitra Vemurid7096132020-02-21 14:50:47 -0800108 # Deleting OLT after tests completes independently (as this test doesn't not run on each POD)
Suchitra Vemuri0656bc22020-03-21 19:07:49 -0700109 #Run Keyword If ${has_dataplane} Delete Device and Verify
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800110
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700111Verify OLT after rebooting physically
112 [Documentation] Test the physical reboot of the OLT
113 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
114 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
115 ... VOL-1956
116 [Tags] functional PhysicalOLTReboot
Andy Baviere187eda2020-04-20 15:00:02 -0700117 [Setup] Start Logging PhysicalOLTReboot
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700118 [Teardown] Run Keywords Collect Logs
119 ... AND Stop Logging PhysicalOLTReboot
Suchitra Vemurib3a92e42020-03-21 16:49:57 -0700120 ... AND Delete Device and Verify
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700121 # Add OLT device
122 setup
123 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
124 Run Keyword If ${has_dataplane} Clean Up Linux
125 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
126 # Reboot the OLT from the OLT CLI
127 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
128 ... sudo reboot ${olt_ip} ${olt_user} ${olt_pass} prompt=#
129 Run Keyword And Ignore Error Collect Logs
130 FOR ${I} IN RANGE 0 ${num_onus}
131 ${src}= Set Variable ${hosts.src[${I}]}
132 ${dst}= Set Variable ${hosts.dst[${I}]}
133 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
134 ... Wait Until Keyword Succeeds 60s 2s
135 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
136 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
137 END
138 # Wait for the OLT to come back up
Andy Bavier8da0e132020-04-13 10:25:16 -0700139 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
140 ... Check Remote System Reachability True ${olt_ip}
Suchitra Vemuri64db9652020-03-22 21:11:13 -0700141 # Waiting extra time for the ONUs to come up
142 Sleep 60s
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700143 Run Keyword And Ignore Error Collect Logs
144 Run Keyword If ${has_dataplane} Clean Up Linux
145 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
146 # Deleting OLT after test completes
Suchitra Vemurib3a92e42020-03-21 16:49:57 -0700147 #Run Keyword If ${has_dataplane} Delete Device and Verify
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700148
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700149Verify restart openolt-adapter container after subscriber provisioning
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800150 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
151 ... Prerequisite : ONUs are authenticated and pingable.
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800152 [Tags] functional VOL-1958 Restart-OpenOlt released
Andy Baviere187eda2020-04-20 15:00:02 -0700153 [Setup] Start Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800154 [Teardown] Run Keywords Collect Logs
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800155 ... AND Stop Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800156 # Add OLT device
157 setup
158 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
159 Run Keyword If ${has_dataplane} Clean Up Linux
160 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
161 ${waitforRestart} Set Variable 120s
162 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
163 Log ${podStatusOutput}
164 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
165 ${podName} Set Variable adapter-open-olt
166 Restart Pod ${NAMESPACE} ${podName}
167 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${podName} ${NAMESPACE}
168 ... Running
Suchitra Vemuri312bfe02020-03-16 16:11:34 -0700169 # Wait for 1min after openolt adapter is restarted
170 Sleep 60s
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800171 Repeat Sanity Test
Suchitra Vemurid7096132020-02-21 14:50:47 -0800172 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800173 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
174 Log ${podStatusOutput}
175 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
176 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
177 Log to console Pod ${podName} restarted and sanity checks passed successfully
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800178
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800179Check OLT/ONU Authentication After Radius Pod Restart
180 [Documentation] After radius restart, triggers reassociation, checks status and
181 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
182 ... wpa supplicant is running in background hence it is recommended to remove
183 ... teardown from previous test or uncomment 'Teardown None'.
184 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
185 [Tags] functional RadiusRestart released
Andy Baviere187eda2020-04-20 15:00:02 -0700186 [Setup] Start Logging RadiusRestart
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800187 [Teardown] Run Keywords Collect Logs
188 ... AND Stop Logging RadiusRestart
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800189 ${waitforRestart} Set Variable 120s
Suchitra Vemurica1c5d32020-04-09 18:45:29 -0700190 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${DEFAULTSPACE} ${RESTART_POD_NAME}
191 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${RESTART_POD_NAME}
192 ... ${DEFAULTSPACE} Running
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800193 FOR ${I} IN RANGE 0 ${num_onus}
194 ${src}= Set Variable ${hosts.src[${I}]}
195 ${dst}= Set Variable ${hosts.dst[${I}]}
196 ${onu_device_id}= Get Device ID From SN ${src['onu']}
197 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
198 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
199 Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800200 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800201 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
202 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
203 ... ${src['ip']} ${src['user']} ${src['pass']}
204 ... ${src['container_type']} ${src['container_name']}
ubuntu6b6e7d42020-03-02 12:35:42 -0800205 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800206 ... ${ONOS_SSH_PORT} ${onu_port}
207 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
208 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
209 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
210 ... ${src['ip']} ${src['user']} ${src['pass']}
211 ... ${src['container_type']} ${src['container_name']}
212 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
213 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
214 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800215 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800216 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
217 Run Keyword and Ignore Error Collect Logs
218 END
219
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700220Verify openolt adapter restart before subscriber provisioning
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800221 [Documentation] Deploys an device instance and waits for it to authenticate. After
222 ... authentication is successful the rw-core deployment is scaled to 0 instances to
223 ... simulate a POD crash. The test then scales the rw-core back to a single instance
224 ... and configures ONOS for access. The test succeeds if the device is able to
225 ... complete the DHCP sequence.
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700226 [Tags] functional olt-adapter-restart
Andy Baviere187eda2020-04-20 15:00:02 -0700227 [Setup] Start Logging OltAdapterRestart
Suchitra Vemuria0c086f2020-03-29 19:14:13 -0700228 #... AND Clear All Devices Then Create New Device
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800229 [Teardown] Run Keywords Collect Logs
230 ... AND Stop Logging OltAdapterRestart
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700231 # Add OLT and perform sanity test
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700232 #setup
Suchitra Vemuri91537c82020-03-29 15:03:26 -0700233 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700234 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800235 Set Global Variable ${of_id}
236
237 FOR ${I} IN RANGE 0 ${num_onus}
238 ${src}= Set Variable ${hosts.src[${I}]}
239 ${dst}= Set Variable ${hosts.dst[${I}]}
240 ${onu_device_id}= Get Device ID From SN ${src['onu']}
241 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
242 ... ${of_id}
243
244 # Bring up the device and verify it authenticates
245 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
246 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
ubuntu6b6e7d42020-03-02 12:35:42 -0800247 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800248 ... ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier84834d42020-02-25 13:49:50 -0700249 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
250 ... /tmp/wpa ${src['dp_iface_name']} log
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800251 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
252 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700253 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
ubuntu6b6e7d42020-03-02 12:35:42 -0800254 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800255 ... ${ONOS_SSH_PORT} ${onu_port}
256 END
257 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
258 Scale K8s Deployment voltha adapter-open-olt 0
259 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha adapter-open-olt
260 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
261 Scale K8s Deployment voltha adapter-open-olt 1
262 Wait Until Keyword Succeeds ${timeout} 2s
263 ... Check Expected Available Deployment Replicas voltha adapter-open-olt 1
264
265 # Ensure the device is available in ONOS, this represents system connectivity being restored
266 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
ubuntu6b6e7d42020-03-02 12:35:42 -0800267 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800268
269 FOR ${I} IN RANGE 0 ${num_onus}
270 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
ubuntu6b6e7d42020-03-02 12:35:42 -0800271 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800272 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530273 # Verify subscriber access flows are added for the ONU port
274 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
275 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
276 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800277 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
278 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
279 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
280 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
281 ... ${dst['container_name']}
282 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800283 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800284 END
285
hwchiu14c05102020-03-27 00:44:30 +0000286Verify restart ofagent container after subscriber is provisioned
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000287 [Documentation] Restart ofagent container after VOLTHA is operational.
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000288 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass7a846662020-03-19 20:42:59 +0530289 [Tags] functional VOL-2409 ofagentRestart
Andy Baviere187eda2020-04-20 15:00:02 -0700290 [Setup] Start Logging ofagentRestart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700291 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700292 ... AND Stop Logging ofagentRestart
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530293 ... AND Scale K8s Deployment ${NAMESPACE} voltha-ofagent 1
Suchitra Vemuriceb002a2020-03-25 13:36:21 -0700294 # set timeout value
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000295 ${waitforRestart} Set Variable 120s
Andy Bavier4a8450e2020-02-04 08:58:37 -0700296 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000297 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700298 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530299 ${podName} Set Variable ofagent
300 Restart Pod ${NAMESPACE} ${podName}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000301 Sleep 60s
302 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
303 ... Running
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530304 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
305 Run Keyword If ${has_dataplane} Clean Up Linux
306 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Andy Bavier4a8450e2020-02-04 08:58:37 -0700307 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000308 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700309 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000310 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530311 # Scale Down the Of-Agent Deployment
312 Scale K8s Deployment ${NAMESPACE} voltha-ofagent 0
313 Sleep 30s
314 FOR ${I} IN RANGE 0 ${num_onus}
315 ${src}= Set Variable ${hosts.src[${I}]}
316 ${dst}= Set Variable ${hosts.dst[${I}]}
317 Run Keyword and Ignore Error Collect Logs
318 ${onu_device_id}= Get Device ID From SN ${src['onu']}
319 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
320 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
321 # Verify ONU state in voltha
322 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
323 ... ENABLED ACTIVE REACHABLE
324 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
325 # Check ONU port is Disabled in ONOS
326 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
327 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
328 # Verify EAPOL flows are present for the ONU port
329 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
330 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
331 # Verify ONU in AAA-Users
332 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
333 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
334 # Verify DHCP-Allocations
335 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
336 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530337 # Verify subscriber access flows are added for the ONU port
338 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
339 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
340 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530341 # Verify Ping
342 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
343 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
344 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
345 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
346 Run Keyword and Ignore Error Collect Logs
347 END
348 # Scale Up the Of-Agent Deployment
349 Scale K8s Deployment ${NAMESPACE} voltha-ofagent 1
350 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
351 ... Running
352 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
353 Run Keyword If ${has_dataplane} Clean Up Linux
354 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
355 Log to console Pod ${podName} restarted and sanity checks passed successfully
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000356
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000357Check ONU adapter crash not forcing authentication again
358 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
359 ... message count to make sure auth not started again and validates EAP status and ping.
360 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
hwchiu14c05102020-03-27 00:44:30 +0000361 [Tags] functional ONUAdaptCrash
Andy Baviere187eda2020-04-20 15:00:02 -0700362 [Setup] Start Logging ONUAdaptCrash
Andy Bavierabeba262020-02-07 16:22:16 -0700363 [Teardown] Run Keywords Collect Logs
364 ... AND Stop Logging ONUAdaptCrash
hwchiu14c05102020-03-27 00:44:30 +0000365 # Wait for adapter to resync
366 Sleep 60s
367 # Restart the onu
368 ${podName} Set Variable adapter-open-onu
Hung-Wei Chiu2bee4d42020-04-24 11:31:50 -0700369 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
hwchiu14c05102020-03-27 00:44:30 +0000370 # Validate ONU Ports
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000371 FOR ${I} IN RANGE 0 ${num_onus}
372 ${src}= Set Variable ${hosts.src[${I}]}
373 ${dst}= Set Variable ${hosts.dst[${I}]}
374 ${onu_device_id}= Get Device ID From SN ${src['onu']}
375 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
376 ... ${of_id}
hwchiu14c05102020-03-27 00:44:30 +0000377 Run Keyword And Continue On Failure Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
378 ... ${onu_port}
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000379 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
380 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
381 ... ${src['container_type']} ${src['container_name']}
382 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000383 END
Hung-Wei Chiu2bee4d42020-04-24 11:31:50 -0700384 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
385 ... app ${podName} Running
hwchiu14c05102020-03-27 00:44:30 +0000386 # Wait for adapter to resync
387 Sleep 60s
Andy Bavier8da0e132020-04-13 10:25:16 -0700388 Run Keyword If ${has_dataplane} Clean Up Linux
hwchiu14c05102020-03-27 00:44:30 +0000389 # Perform all steps in Sanity Test except the subscriber addition
390 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
391 Set Global Variable ${of_id}
392 FOR ${I} IN RANGE 0 ${num_onus}
393 ${src}= Set Variable ${hosts.src[${I}]}
394 ${dst}= Set Variable ${hosts.dst[${I}]}
395 Run Keyword and Ignore Error Collect Logs
396 ${onu_device_id}= Get Device ID From SN ${src['onu']}
397 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
398 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
399 # Check ONU port is Enabled in ONOS
400 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
401 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
402 # Verify EAPOL flows are added for the ONU port
403 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
404 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
405 # Verify ONU state in voltha
406 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
407 ... ENABLED ACTIVE REACHABLE
408 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
409 # Perform Authentication
410 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
411 ... /tmp/wpa ${src['dp_iface_name']} log
412 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
413 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
414 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
415 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
416 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
417 # Verify that no pending flows exist for the ONU port
418 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
419 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530420 # Verify subscriber access flows are added for the ONU port
421 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
422 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
423 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
hwchiu14c05102020-03-27 00:44:30 +0000424 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
425 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
426 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
427 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
428 ... ${dst['container_name']}
429 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
430 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
431 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
432 Run Keyword and Ignore Error Collect Logs
433 END
434
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000435 Run Keyword and Ignore Error Collect Logs
436
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700437Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
438 [Documentation] Deploys an device instance and waits for it to authenticate. After
439 ... authentication is successful the rw-core deployment is scaled to 0 instances to
440 ... simulate a POD crash. The test then scales the rw-core back to a single instance
441 ... and configures ONOS for access. The test succeeds if the device is able to
442 ... complete the DHCP sequence.
443 [Tags] functional rwcore-restart
Andy Baviere187eda2020-04-20 15:00:02 -0700444 [Setup] Run Keywords Start Logging RwCoreFailAndRestart
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700445 ... AND Clear All Devices Then Create New Device
446 [Teardown] Run Keywords Collect Logs
447 ... AND Stop Logging RwCoreFailAndRestart
Suchitra Vemuri5d3383f2020-04-03 11:02:26 -0700448 #... AND Delete Device and Verify
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700449 Run Keyword and Ignore Error Collect Logs
450 Run Keyword If ${has_dataplane} Clean Up Linux
451 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
452 Set Global Variable ${of_id}
Hardik Windlasse05e4712020-04-29 10:40:42 +0530453 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
454 ... Get NNI Port in ONOS ${of_id}
455 Set Global Variable ${nni_port}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700456 FOR ${I} IN RANGE 0 ${num_onus}
457 ${src}= Set Variable ${hosts.src[${I}]}
458 ${dst}= Set Variable ${hosts.dst[${I}]}
459 ${onu_device_id}= Get Device ID From SN ${src['onu']}
460 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
461 ... ${of_id}
462 # Bring up the device and verify it authenticates
463 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
464 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
465 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
466 ... ${ONOS_SSH_PORT} ${onu_port}
467 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
468 ... /tmp/wpa ${src['dp_iface_name']} log
469 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
470 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
471 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
472 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
473 ... ${ONOS_SSH_PORT} ${onu_port}
474 END
475
476 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
477 Scale K8s Deployment voltha voltha-rw-core 0
478 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
479 # Ensure the ofagent POD goes "not-ready" as expected
480 Wait Until keyword Succeeds ${timeout} 2s
481 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
482 # Scale up the core deployment and make sure both it and the ofagent deployment are back
483 Scale K8s Deployment voltha voltha-rw-core 1
484 Wait Until Keyword Succeeds ${timeout} 2s
485 ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
486 Wait Until Keyword Succeeds ${timeout} 2s
487 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
488 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
489 # so restart the port forwarding for the API service
490 Restart VOLTHA Port Foward voltha-api-minimal
491 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
492 # represents system connectivity being restored
493 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
494 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
495
496 FOR ${I} IN RANGE 0 ${num_onus}
497 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
498 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
499 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530500 # Verify subscriber access flows are added for the ONU port
501 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
502 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
503 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700504 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
505 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
506 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
507 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
508 ... ${dst['container_name']}
509 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
510 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
511 END
512
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000513ONU Reboot
514 [Documentation] Reboot ONU and verify that ONU comes up properly
515 [Tags] VOL-1957 RebootONU notready
Andy Baviere187eda2020-04-20 15:00:02 -0700516 [Setup] Start Logging RebootONU
Andy Bavier4a8450e2020-02-04 08:58:37 -0700517 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700518 ... AND Stop Logging RebootONU
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000519 FOR ${I} IN RANGE 0 ${num_onus}
520 ${src}= Set Variable ${hosts.src[${I}]}
521 ${dst}= Set Variable ${hosts.dst[${I}]}
522 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800523 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
524 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000525 Reboot ONU ${onu_device_id} ${src} ${dst}
526 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800527 # Check ONU port is Enabled in ONOS
528 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800529 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800530 # Verify EAPOL flows are added for the ONU port
531 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800532 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800533 # Verify ONU state in voltha
534 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
535 ... ENABLED ACTIVE REACHABLE
536 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass21807632020-04-14 16:24:55 +0530537 # Verify subscriber access flows are added for the ONU port
538 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
539 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
540 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800541 # Verify pings are successful after reboot on the current ONU
542 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
543 ... Wait Until Keyword Succeeds 60s 2s
544 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
545 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000546 END
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800547 #Run Keyword If ${has_dataplane} Clean Up Linux
548 #Check after reboot that ONUs are active, authenticated/DHCP/pingable
549 #Perform Sanity Test
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800550
551*** Keywords ***
552Setup Suite
553 [Documentation] Set up the test suite
554 Common Test Suite Setup
HungWei Chiu1408fb92020-03-03 19:43:30 -0500555 #power_switch.robot needs it to support different vendor's power switch
Hung-Wei Chiufcc6d8e2020-03-10 12:05:11 -0700556 ${switch_type}= Get Variable Value ${web_power_switch.type}
557 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800558
559Clear All Devices Then Create New Device
560 [Documentation] Remove any devices from VOLTHA and ONOS
561 # Remove all devices from voltha and nos
562 Delete All Devices and Verify
563 # Execute normal test Setup Keyword
564 Setup