blob: 7b8cb4676f93bcf2be999a01118bf0793453dc5f [file] [log] [blame]
David Bainbridgef81cd642019-11-20 00:14:47 +00001# Copyright 2017 - present Open Networking Foundation
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -07002#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Matteo Scandolo1294aeb2019-09-24 16:20:32 -070014# FIXME Can we use the same test against BBSim and Hardware?
15
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070016*** Settings ***
Zack Williamsec53a1b2019-09-16 15:50:52 -070017Documentation Test various end-to-end scenarios
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070018Suite Setup Setup Suite
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070019Test Setup Setup
20Test Teardown Teardown
Andy Bavierba9866b2019-10-11 07:11:53 -070021Suite Teardown Teardown Suite
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070022Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
Andy Bavierba9866b2019-10-11 07:11:53 -070027Library ../../libraries/DependencyLibrary.py
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070028Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
David Bainbridgef81cd642019-11-20 00:14:47 +000030Resource ../../libraries/voltha.robot
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070031Resource ../../libraries/utils.robot
Andy Bavierba9866b2019-10-11 07:11:53 -070032Resource ../../libraries/k8s.robot
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070033Resource ../../variables/variables.robot
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070034
35*** Variables ***
Zack Williamsec53a1b2019-09-16 15:50:52 -070036${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
Zack Williamsa8fe75a2020-01-10 14:25:27 -070043${NAMESPACE} voltha
suraj gourd64356b2019-11-07 13:26:20 +000044# For below variable value, using deployment name as using grep for
45# parsing radius pod name, we can also use full radius pod name
Zack Williamsa8fe75a2020-01-10 14:25:27 -070046${RESTART_POD_NAME} radius
Andy Bavierba9866b2019-10-11 07:11:53 -070047${timeout} 60s
Zack Williamsec53a1b2019-09-16 15:50:52 -070048${of_id} 0
49${logical_id} 0
Andy Bavierba9866b2019-10-11 07:11:53 -070050${has_dataplane} True
51${external_libs} True
52${teardown_device} False
Zack Williamsa8fe75a2020-01-10 14:25:27 -070053${scripts} ../../scripts
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070054
55*** Test Cases ***
56Sanity E2E Test for OLT/ONU on POD
57 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
58 ... Validate successful authentication/DHCP/E2E ping for the tech profile that is used
Andy Bavierba9866b2019-10-11 07:11:53 -070059 [Tags] sanity test1
Andy Bavier4a8450e2020-02-04 08:58:37 -070060 [Setup] Run Keywords Announce Message START TEST SanityTest
61 ... AND Setup
62 [Teardown] Run Keywords Collect Logs
63 ... AND Announce Message END TEST SanityTest
Suchitra Vemuric5295a32019-12-15 20:32:04 -080064 Run Keyword If ${has_dataplane} Clean Up Linux
Zack Williamsa8fe75a2020-01-10 14:25:27 -070065 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070066
Hema93762cf2020-01-29 19:59:28 +053067Test Disable and Enable OLT
68 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
69 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
70 ... Perform disable on the OLT and validate that the pings do not succeed
71 ... Perform enable on the OLT and validate that the pings are successful
72 [Tags] VOL-2410 DisableEnableOLT notready
Andy Bavier4a8450e2020-02-04 08:58:37 -070073 [Setup] Announce Message START TEST DisableEnableOLT
74 [Teardown] Run Keywords Collect Logs
75 ... AND Announce Message END TEST DisableEnableOLT
Hema93762cf2020-01-29 19:59:28 +053076 #Disable the OLT and verify the OLT/ONUs are disabled properly
77 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
78 Should Be Equal As Integers ${rc} 0
79 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN REACHABLE
80 ... ${olt_serial_number}
81 FOR ${I} IN RANGE 0 ${num_onus}
82 ${src}= Set Variable ${hosts.src[${I}]}
83 ${dst}= Set Variable ${hosts.dst[${I}]}
84 ${onu_device_id}= Get Device ID From SN ${src['onu']}
85 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED DISCOVERED
86 ... UNREACHABLE ${src['onu']} onu=false
87 #Verify that ping fails
88 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 60s 2s
89 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
90 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}}
91 END
92 #Enable the OLT back and check ONU, OLT status are back to "ACTIVE"
93 Enable Device ${olt_device_id}
94 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE REACHABLE
95 ... ${olt_serial_number}
96 FOR ${I} IN RANGE 0 ${num_onus}
97 ${src}= Set Variable ${hosts.src[${I}]}
98 ${dst}= Set Variable ${hosts.dst[${I}]}
99 ${onu_device_id}= Get Device ID From SN ${src['onu']}
100 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
101 ... ${of_id}
102 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE
103 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
104 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
105 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
106 #Verify that ping workss fine again
107 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 60s 2s
108 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
109 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}}
110 Run Keyword and Ignore Error Collect Logs
111 END
112
113
Suchitra Vemuri6db89412019-11-14 14:52:54 -0800114Test Disable and Enable ONU
115 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
116 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
117 ... Perform disable on the ONUs and validate that the pings do not succeed
118 ... Perform enable on the ONUs and validate that the pings are successful
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800119 [Tags] functional DisableEnableONU released
Andy Bavier4a8450e2020-02-04 08:58:37 -0700120 [Setup] Announce Message START TEST DisableEnableONU
121 [Teardown] Run Keywords Collect Logs
122 ... AND Announce Message END TEST DisableEnableONU
Suchitra Vemuri6db89412019-11-14 14:52:54 -0800123 FOR ${I} IN RANGE 0 ${num_onus}
124 ${src}= Set Variable ${hosts.src[${I}]}
125 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri6db89412019-11-14 14:52:54 -0800126 ${onu_device_id}= Get Device ID From SN ${src['onu']}
127 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
128 ... ${of_id}
129 Disable Device ${onu_device_id}
Suchitra Vemuri15e15252020-01-20 17:38:49 -0800130 #Test Devices Disabled in VOLTHA Id=${onu_device_id}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700131 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
132 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700133 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
134 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri6db89412019-11-14 14:52:54 -0800135 Enable Device ${onu_device_id}
136 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
137 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700138 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
139 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700140 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
141 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
142 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
143 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri6db89412019-11-14 14:52:54 -0800144 END
145
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800146Test Subscriber Delete and Add
147 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
148 ... Assuming that all the ONUs are authenticated/DHCP/pingable
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700149 ... Delete a subscriber and validate that the pings do not succeed
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800150 ... Re-add the subscriber and validate that the pings are successful
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800151 [Tags] functional SubAddDelete released
Andy Bavier4a8450e2020-02-04 08:58:37 -0700152 [Setup] Announce Message START TEST SubAddDelete
153 [Teardown] Run Keywords Collect Logs
154 ... AND Announce Message END TEST SubAddDelete
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800155 FOR ${I} IN RANGE 0 ${num_onus}
156 ${src}= Set Variable ${hosts.src[${I}]}
157 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800158 ${onu_device_id}= Get Device ID From SN ${src['onu']}
159 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
160 ... ${of_id}
161 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
162 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
163 Sleep 10s
Andy Bavier33e6dd32020-01-16 13:35:20 -0700164 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
165 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700166 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
167 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800168 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
169 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
170 Sleep 10s
171 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
172 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700173 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
174 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700175 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
176 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
177 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
178 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800179 END
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800180
suraj gourd64356b2019-11-07 13:26:20 +0000181Check OLT/ONU Authentication After Radius Pod Restart
suraj gour7f6d5fe2019-11-29 10:56:35 +0000182 [Documentation] After radius restart, triggers reassociation, checks status and
183 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
184 ... wpa supplicant is running in background hence it is recommended to remove
185 ... teardown from previous test or uncomment 'Teardown None'.
186 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800187 [Tags] functional RadiusRestart released
Andy Bavier4a8450e2020-02-04 08:58:37 -0700188 [Setup] Announce Message START TEST RadiusRestart
189 [Teardown] Run Keywords Collect Logs
190 ... AND Announce Message END TEST RadiusRestart
suraj gourd64356b2019-11-07 13:26:20 +0000191 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
suraj gourd64356b2019-11-07 13:26:20 +0000192 FOR ${I} IN RANGE 0 ${num_onus}
193 ${src}= Set Variable ${hosts.src[${I}]}
194 ${dst}= Set Variable ${hosts.dst[${I}]}
suraj gourd64356b2019-11-07 13:26:20 +0000195 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700196 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
197 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
198 Wait Until Keyword Succeeds ${timeout} 2s
199 ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
200 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
201 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
202 ... ${src['ip']} ${src['user']} ${src['pass']}
suraj gourd64356b2019-11-07 13:26:20 +0000203 ... ${src['container_type']} ${src['container_name']}
204 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700205 ... ${ONOS_SSH_PORT} ${onu_port}
206 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
207 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
208 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
209 ... ${src['ip']} ${src['user']} ${src['pass']}
210 ... ${src['container_type']} ${src['container_name']}
211 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
212 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
suraj gourd64356b2019-11-07 13:26:20 +0000213 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
214 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700215 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
216 Run Keyword and Ignore Error Collect Logs
suraj gourd64356b2019-11-07 13:26:20 +0000217 END
218
suraj gourd5cfdbb2019-12-13 12:44:55 +0000219Check DHCP attempt fails when subscriber is not added
220 [Documentation] Validates when removed subscriber access, DHCP attempt, ping fails and
221 ... when again added subscriber access, DHCP attempt, ping succeeds
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700222 ... Assuming that test1 or sanity test was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800223 [Tags] functional SubsRemoveDHCP released
Andy Bavier4a8450e2020-02-04 08:58:37 -0700224 [Setup] Announce Message START TEST SubsRemoveDHCP
225 [Teardown] Run Keywords Collect Logs
226 ... AND Announce Message END TEST SubsRemoveDHCP
suraj gourd5cfdbb2019-12-13 12:44:55 +0000227 FOR ${I} IN RANGE 0 ${num_onus}
228 ${src}= Set Variable ${hosts.src[${I}]}
229 ${dst}= Set Variable ${hosts.dst[${I}]}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700230 ${onu_device_id}= Get Device ID From SN ${src['onu']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000231 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
232 ... ${of_id}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800233 Run Keyword And Ignore Error Login And Run Command On Remote System killall dhclient ${src['ip']}
234 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
235 Run Keyword And Ignore Error Login And Run Command On Remote System ps -ef | grep dhclient ${src['ip']}
236 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000237 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
238 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700239 Sleep 5s
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800240 Run Keyword And Ignore Error Login And Run Command On Remote System ps -ef | grep dhclient ${src['ip']}
241 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700242 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds ${timeout} 2s
243 ... Delete IP Addresses from Interface on Remote Host ${src['dp_iface_name']} ${src['ip']}
244 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
245 Run Keyword And Ignore Error Login And Run Command On Remote System
246 ... ifconfig | grep -A 10 ens ${src['ip']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800247 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000248 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping False
249 ... False ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
250 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
251 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
252 ... ${dst['container_name']}
253 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
254 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
255 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
256 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
257 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
258 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
259 ... ${dst['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800260 Run Keyword and Ignore Error Collect Logs
261 END
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800262
suraj gour1ecfae92019-12-20 15:11:40 +0000263Check ONU adapter crash not forcing authentication again
264 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700265 ... message count to make sure auth not started again and validates EAP status and ping.
suraj gour1ecfae92019-12-20 15:11:40 +0000266 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800267 [Tags] functional ONUAdaptCrash notready
Andy Bavier4a8450e2020-02-04 08:58:37 -0700268 [Setup] Announce Message START TEST ONUAdaptCrash
269 [Teardown] Run Keywords Collect Logs
270 ... AND Announce Message END TEST ONUAdaptCrash
suraj gour1ecfae92019-12-20 15:11:40 +0000271 @{before_list}= Create List
272 @{after_list}= Create List
273 FOR ${I} IN RANGE 0 ${num_onus}
274 ${src}= Set Variable ${hosts.src[${I}]}
275 ${dst}= Set Variable ${hosts.dst[${I}]}
276 ${onu_device_id}= Get Device ID From SN ${src['onu']}
277 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
278 ... ${of_id}
279 ${before}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
280 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
281 ... ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700282 Append To List ${before_list} ${before}
suraj gour1ecfae92019-12-20 15:11:40 +0000283 END
284 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} adapter-open-onu
285 Wait Until Keyword Succeeds ${timeout} 2s Validate Pod Status ${podName} ${NAMESPACE}
286 ... Running
287 FOR ${I} IN RANGE 0 ${num_onus}
288 ${src}= Set Variable ${hosts.src[${I}]}
289 ${dst}= Set Variable ${hosts.dst[${I}]}
290 ${onu_device_id}= Get Device ID From SN ${src['onu']}
291 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
292 ... ${of_id}
293 ${after}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
294 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
295 ... ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700296 Append To List ${after_list} ${after}
297 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
298 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
299 ... ${src['container_type']} ${src['container_name']}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700300 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700301 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
302 ... Wait Until Keyword Succeeds 60s 2s Check Ping
suraj gour1ecfae92019-12-20 15:11:40 +0000303 ... True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']} ${src['user']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700304 ... ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gour1ecfae92019-12-20 15:11:40 +0000305 END
306 Lists Should Be Equal ${after_list} ${before_list}
307 Log ${after_list}
308 Log ${before_list}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700309
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800310
311Test Disable and Enable ONU scenario for ATT workflow
312 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
313 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
314 ... Perform disable on the ONUs, call volt-remove-subscriber and validate that the pings do not succeed
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700315 ... Perform enable on the ONUs, authentication check, volt-add-subscriber-access and
316 ... validate that the pings are successful
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800317 ... VOL-2284
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800318 [Tags] functional ATT_DisableEnableONU
Andy Bavier4a8450e2020-02-04 08:58:37 -0700319 [Setup] Announce Message START TEST ATT_DisableEnableONU
320 [Teardown] Run Keywords Collect Logs
321 ... AND Announce Message END TEST ATT_DisableEnableONU
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800322 FOR ${I} IN RANGE 0 ${num_onus}
323 ${src}= Set Variable ${hosts.src[${I}]}
324 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800325 ${onu_device_id}= Get Device ID From SN ${src['onu']}
326 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
327 ... ${of_id}
328 Disable Device ${onu_device_id}
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800329 Sleep 5s
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800330 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
331 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700332 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
333 ... Wait Until Keyword Succeeds 60s 2s Check Ping
334 ... False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
335 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800336 ... ELSE sleep 60s
337 Enable Device ${onu_device_id}
338 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
339 ... ${ONOS_SSH_PORT} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700340 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
341 ... Validate Authentication After Reassociate True
342 ... ${src['dp_iface_name']} ${src['ip']} ${src['user']} ${src['pass']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800343 ... ${src['container_type']} ${src['container_name']}
344 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700345 ... ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800346 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
347 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
348 Sleep 10s
349 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
Andy Bavier08ae5852019-12-19 09:12:42 -0700350 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800351 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
352 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
353 ... ${dst['container_name']}
354 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
355 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800356 Run Keyword and Ignore Error Collect Logs
suraj gourd5cfdbb2019-12-13 12:44:55 +0000357 END
suraj gourd5cfdbb2019-12-13 12:44:55 +0000358
Suchitra Vemuri937cd392020-01-12 17:10:42 -0800359Delete OLT, ReAdd OLT and Perform Sanity Test
360 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
361 ... Disable and Delete the OLT
362 ... Create/Enable the same OLT again
363 ... Validate authentication/DHCP/E2E pings succeed for all the ONUs connected to the OLT
Suchitra Vemuri2fa9bba2020-01-22 17:38:48 -0800364 [Tags] functional DeleteOLT
Andy Bavier4a8450e2020-02-04 08:58:37 -0700365 [Setup] Announce Message START TEST DeleteOLT
366 [Teardown] Run Keywords Collect Logs
367 ... AND Announce Message END TEST DeleteOLT
Suchitra Vemuri937cd392020-01-12 17:10:42 -0800368 Run Keyword If ${has_dataplane} Clean Up Linux
369 Run Keyword If ${has_dataplane} Delete Device and Verify
370 Run Keyword and Ignore Error Collect Logs
371 # Recreate the OLT
372 Run Keyword If ${has_dataplane} Setup
373 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemuri937cd392020-01-12 17:10:42 -0800374
Hema783279b2020-01-22 15:37:37 +0530375Test disable ONUs and OLT then delete ONUs and OLT
376 [Documentation] On deployed POD, disable the ONU, disable the OLT and then delete ONU and OLT.
377 ... This TC is to confirm that ONU removal is not impacting OLT
378 ... Devices will be removed during the execution of this TC
379 ... so calling setup at the end to add the devices back to avoid the confusion.
Suchitra Vemuriba4f3712020-01-24 16:18:26 -0800380 [Tags] functional VOL-2354 DisableDeleteONUandOLT
Andy Bavier4a8450e2020-02-04 08:58:37 -0700381 [Setup] Announce Message START TEST DisableDeleteONUandOLT
382 [Teardown] Run Keywords Collect Logs
383 ... AND Announce Message END TEST DisableDeleteONUandOLT
Suchitra Vemuriba4f3712020-01-24 16:18:26 -0800384 ${olt_device_id}= Get Device ID From SN ${olt_serial_number}
Hema783279b2020-01-22 15:37:37 +0530385 FOR ${I} IN RANGE 0 ${num_onus}
386 ${src}= Set Variable ${hosts.src[${I}]}
387 ${dst}= Set Variable ${hosts.dst[${I}]}
388 ${onu_device_id}= Get Device ID From SN ${src['onu']}
389 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE
390 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
391 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE
392 ... REACHABLE ${olt_serial_number}
393 ${rc} ${output}= Run and Return Rc and Output
394 ... ${VOLTCTL_CONFIG}; voltctl device disable ${onu_device_id}
395 Should Be Equal As Integers ${rc} 0
396 Wait Until Keyword Succeeds ${timeout} 5s Validate Device DISABLED UNKNOWN
397 ... REACHABLE ${src['onu']} onu=false
398 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE
399 ... REACHABLE ${olt_serial_number}
400 END
401 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
402 Should Be Equal As Integers ${rc} 0
403 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN REACHABLE
404 ... ${olt_serial_number}
405 FOR ${I} IN RANGE 0 ${num_onus}
406 ${src}= Set Variable ${hosts.src[${I}]}
407 ${dst}= Set Variable ${hosts.dst[${I}]}
408 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuriba4f3712020-01-24 16:18:26 -0800409 Wait Until Keyword Succeeds ${timeout} 5s Validate Device DISABLED UNKNOWN
410 ... REACHABLE ${src['onu']} onu=false
Hema783279b2020-01-22 15:37:37 +0530411 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device delete ${onu_device_id}
412 Should Be Equal As Integers ${rc} 0
413 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN
414 ... REACHABLE ${olt_serial_number}
415 END
416 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device delete ${olt_device_id}
417 Should Be Equal As Integers ${rc} 0
418 Wait Until Keyword Succeeds ${timeout} 5s Test Empty Device List
419 #Adding setup here to add the devices back since this TC removes the devices
420 Run Keyword If ${has_dataplane} sleep 180s
421 setup
422
David Bainbridgef81cd642019-11-20 00:14:47 +0000423Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
424 [Documentation] Deploys an device instance and waits for it to authenticate. After
425 ... authentication is successful the rw-core deployment is scaled to 0 instances to
426 ... simulate a POD crash. The test then scales the rw-core back to a single instance
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700427 ... and configures ONOS for access. The test succeeds if the device is able to
David Bainbridgef81cd642019-11-20 00:14:47 +0000428 ... complete the DHCP sequence.
429 [Tags] bbsim rwcore-restart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700430 [Setup] Run Keywords Announce Message START TEST RwCoreFailAndRestart
431 ... AND Clear All Devices Then Create New Device
432 [Teardown] Run Keywords Collect Logs
433 ... AND Announce Message END TEST RwCoreFailAndRestart
David Bainbridgef81cd642019-11-20 00:14:47 +0000434 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
435 Set Global Variable ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000436 FOR ${I} IN RANGE 0 ${num_onus}
437 ${src}= Set Variable ${hosts.src[${I}]}
438 ${dst}= Set Variable ${hosts.dst[${I}]}
439 ${onu_device_id}= Get Device ID From SN ${src['onu']}
440 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
441 ... ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000442 # Bring up the device and verify it authenticates
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700443 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
David Bainbridgef81cd642019-11-20 00:14:47 +0000444 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
445 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
446 ... ${ONOS_SSH_PORT} ${onu_port}
447 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
448 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
449 ... ${src['container_type']} ${src['container_name']}
450 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
David Bainbridge3d6d5d32019-12-17 19:05:35 +0000451 ... ${ONOS_SSH_PORT} ${onu_port}
452 END
453
454 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
455 Scale K8s Deployment voltha voltha-rw-core 0
456 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
457 # Ensure the ofagent POD goes "not-ready" as expected
458 Wait Until keyword Succeeds ${timeout} 2s
459 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
460 # Scale up the core deployment and make sure both it and the ofagent deployment are back
461 Scale K8s Deployment voltha voltha-rw-core 1
462 Wait Until Keyword Succeeds ${timeout} 2s
463 ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
464 Wait Until Keyword Succeeds ${timeout} 2s
465 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
466 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
467 # so restart the port forwarding for the API service
468 Restart VOLTHA Port Foward voltha-api-minimal
469 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
470 # represents system connectivity being restored
471 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
472 ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
473
474 FOR ${I} IN RANGE 0 ${num_onus}
475 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
476 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
477 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
478 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
479 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
480 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
481 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
482 ... ${dst['container_name']}
483 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
484 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
485 END
486
487Sanity E2E Test for OLT/ONU on POD With OLT Adapters Fail and Restart
488 [Documentation] Deploys an device instance and waits for it to authenticate. After
489 ... authentication is successful the rw-core deployment is scaled to 0 instances to
490 ... simulate a POD crash. The test then scales the rw-core back to a single instance
491 ... and configures ONOS for access. The test succeeds if the device is able to
492 ... complete the DHCP sequence.
493 [Tags] bbsim olt-adapter-restart
494 [Setup] Clear All Devices Then Create New Device
495 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
496 Set Global Variable ${of_id}
497
498 FOR ${I} IN RANGE 0 ${num_onus}
499 ${src}= Set Variable ${hosts.src[${I}]}
500 ${dst}= Set Variable ${hosts.dst[${I}]}
501 ${onu_device_id}= Get Device ID From SN ${src['onu']}
502 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
503 ... ${of_id}
504
505 # Bring up the device and verify it authenticates
506 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
507 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
508 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700509 ... ${ONOS_SSH_PORT} ${onu_port}
David Bainbridge3d6d5d32019-12-17 19:05:35 +0000510 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
511 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
512 ... ${src['container_type']} ${src['container_name']}
513 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
514 ... ${ONOS_SSH_PORT} ${onu_port}
515 END
516
517 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
518 Scale K8s Deployment voltha adapter-open-olt 0
519 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha adapter-open-olt
520 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
521 Scale K8s Deployment voltha adapter-open-olt 1
522 Wait Until Keyword Succeeds ${timeout} 2s
523 ... Check Expected Available Deployment Replicas voltha adapter-open-olt 1
524
525 # Ensure the device is available in ONOS, this represents system connectivity being restored
526 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
527 ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
528
529 FOR ${I} IN RANGE 0 ${num_onus}
David Bainbridgef81cd642019-11-20 00:14:47 +0000530 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
531 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
532 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
533 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
534 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
535 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
536 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
537 ... ${dst['container_name']}
538 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
539 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
540 END
541
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700542*** Keywords ***
543Setup Suite
Andy Bavier88cd9f62019-11-26 16:22:33 -0700544 [Documentation] Set up the test suite
545 Common Test Suite Setup
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700546
David Bainbridgef81cd642019-11-20 00:14:47 +0000547Clear All Devices Then Create New Device
548 [Documentation] Remove any devices from VOLTHA and ONOS
David Bainbridgef81cd642019-11-20 00:14:47 +0000549 # Remove all devices from voltha and nos
550 Delete All Devices and Verify
David Bainbridgef81cd642019-11-20 00:14:47 +0000551 # Execute normal test Setup Keyword
552 Setup
553