blob: a82a889a2d5cce70b48f78485c21acd010c496af [file] [log] [blame]
Joey Armstrong9fadcbe2024-01-17 19:00:37 -05001# Copyright 2020-2024 Open Networking Foundation (ONF) and the ONF Contributors
Suchitra Vemurife309412020-12-01 19:19:52 -08002#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15*** Settings ***
16Documentation Test various failure scenarios
17Suite Setup Setup Suite
18Test Setup Setup
19Test Teardown Teardown
TorstenThieme4e2168e2021-06-22 14:01:47 +000020Suite Teardown Teardown Suite
Suchitra Vemurife309412020-12-01 19:19:52 -080021Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../libraries/DependencyLibrary.py
27Resource ../../libraries/onos.robot
28Resource ../../libraries/voltctl.robot
29Resource ../../libraries/voltha.robot
30Resource ../../libraries/utils.robot
31Resource ../../libraries/k8s.robot
32Resource ../../variables/variables.robot
33Resource ../../libraries/power_switch.robot
34
35*** Variables ***
36${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
43${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000044${INFRA_NAMESPACE} default
Suchitra Vemurife309412020-12-01 19:19:52 -080045# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
51${has_dataplane} True
52${teardown_device} False
53${scripts} ../../scripts
54
55# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070058# logging flag to enable Collect Logs, can be passed via the command line too
59# example: -v logging:False
60${logging} True
61
Suchitra Vemurife309412020-12-01 19:19:52 -080062*** Test Cases ***
63Verify OLT after rebooting physically - MultipleOLT
64 [Documentation] Test the physical reboot of the OLT
65 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
66 ... Test performs a physical reboot on one of the OLTs, performs "reboot" from the OLT CLI
67 ... Validates that ONU on other OLTs are still functional
Hardik Windlass5ceb84e2021-02-11 11:17:08 +000068 [Tags] functional MultiOLTPhysicalReboot
Suchitra Vemurife309412020-12-01 19:19:52 -080069 [Setup] Start Logging MultipleOlt-PhysicalOLTReboot
70 [Teardown] Run Keywords Collect Logs
71 ... AND Stop Logging MultipleOlt-PhysicalOLTReboot
Suchitra Vemurife309412020-12-01 19:19:52 -080072 # Execute the test when the number of OLTs are greater than one
73 Pass Execution If ${olt_count} == 1 Skipping test: just one OLT
Hardik Windlassa119a102021-02-15 14:52:07 +053074 Clear All Devices Then Perform Setup And Sanity
Suchitra Vemurife309412020-12-01 19:19:52 -080075 # Reboot the first OLT from the list of olts - rebooting from the OLT CLI
76 ${olt_user}= Get From Dictionary ${list_olts}[0] user
77 ${olt_pass}= Get From Dictionary ${list_olts}[0] pass
78 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[0] sship
79 ${olt_serial_number}= Get From Dictionary ${list_olts}[0] sn
80 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
81 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +010082 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemurife309412020-12-01 19:19:52 -080083 # validate that the ONUs on the other OLTs are still functional
84 Verify ping is successful for ONUs not on this OLT ${num_all_onus} ${olt_device_id}
Suchitra Vemurife309412020-12-01 19:19:52 -080085 # Validate that the ONUs on the rebooted OLT are not pingable
86 FOR ${I} IN RANGE 0 ${num_all_onus}
87 ${src}= Set Variable ${hosts.src[${I}]}
88 ${dst}= Set Variable ${hosts.dst[${I}]}
89 ${olt_serial_number}= Get From Dictionary ${list_olts}[0] sn
90 Continue For Loop If "${olt_serial_number}"!="${src['olt']}"
91 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass58b23a32022-05-10 11:06:48 +053092 ... Wait Until Keyword Succeeds ${timeout} 2s
Suchitra Vemurife309412020-12-01 19:19:52 -080093 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
94 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
95 END
96 # Wait for the rebooted OLT to come back up
97 ${olt_serial_number}= Get From Dictionary ${olt_ids}[0] sn
98 ${olt_user}= Get From Dictionary ${list_olts}[0] user
99 ${olt_pass}= Get From Dictionary ${list_olts}[0] pass
100 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[0] sship
101 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
102 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
103 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530104 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemurife309412020-12-01 19:19:52 -0800105 ... Validate OLT Device ENABLED ACTIVE
106 ... REACHABLE ${olt_serial_number}
107 # Waiting extra time for the ONUs to come up
108 Sleep 60s
Suchitra Vemurife309412020-12-01 19:19:52 -0800109 Run Keyword If ${has_dataplane} Clean Up Linux
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700110 FOR ${J} IN RANGE 0 ${num_olts}
111 ${olt_serial_number}= Set Variable ${list_olts}[${J}][sn]
112 ${onu_count}= Set Variable ${list_olts}[${J}][onucount]
113 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
114 ... ${olt_serial_number}
115 Set Global Variable ${of_id}
TorstenThiemedda37652021-11-11 16:22:05 +0000116 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700117 ${suppressaddsubscriber}= Set Variable If '${J}'=='0' False True
118 Perform Sanity Test Per OLT ${of_id} ${nni_port} ${olt_serial_number} ${onu_count}
119 ... ${suppressaddsubscriber}
120 END
Suchitra Vemurife309412020-12-01 19:19:52 -0800121 # Deleting OLT after test completes
122 #Run Keyword If ${has_dataplane} Delete All Devices and Verify
Suchitra Vemurife309412020-12-01 19:19:52 -0800123
124Verify OLT Soft Reboot - MultipleOLT
125 [Documentation] Test soft reboot of the OLT using voltctl command
126 ... Verifies that when one OLT is rebooted other ONUs on other OLTs are
127 ... still functional
Hardik Windlass5ceb84e2021-02-11 11:17:08 +0000128 [Tags] MultiOLTSoftReboot functional
Suchitra Vemurife309412020-12-01 19:19:52 -0800129 [Setup] Start Logging MultiOlt-OLTSoftReboot
130 [Teardown] Run Keywords Collect Logs
Suchitra Vemuri6971c392020-12-08 10:41:33 -0800131 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700132 ... AND Run Keyword If ${logging} Collect Logs
133 ... AND Stop Logging MultiOlt-OLTSoftReboot
Suchitra Vemurife309412020-12-01 19:19:52 -0800134 # Execute the test when the number of OLTs are greater than one
135 Pass Execution If ${olt_count} == 1 Skipping test: just one OLT
Hardik Windlassa119a102021-02-15 14:52:07 +0530136 Clear All Devices Then Perform Setup And Sanity
Suchitra Vemurife309412020-12-01 19:19:52 -0800137 # Reboot the first OLT
138 ${olt_user}= Get From Dictionary ${list_olts}[0] user
139 ${olt_pass}= Get From Dictionary ${list_olts}[0] pass
140 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[0] sship
141 ${olt_serial_number}= Get From Dictionary ${list_olts}[0] sn
142 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530143 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemurife309412020-12-01 19:19:52 -0800144 ... Validate OLT Device ENABLED ACTIVE
145 ... REACHABLE ${olt_serial_number}
146 # Reboot the OLT using "voltctl device reboot" command
147 Reboot Device ${olt_device_id}
Andrea Campanella9d8d3a52021-06-17 12:18:23 +0200148 # Wait for the OLT to actually go down
149 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
150 ... ${olt_serial_number}
Suchitra Vemurife309412020-12-01 19:19:52 -0800151 # validate that the ONUs on the other OLTs are still functional
152 Verify ping is successful for ONUs not on this OLT ${num_all_onus} ${olt_device_id}
153 #Verify that ping fails for the ONUs where the OLT has been rebooted
154 FOR ${I} IN RANGE 0 ${num_all_onus}
155 ${src}= Set Variable ${hosts.src[${I}]}
156 ${dst}= Set Variable ${hosts.dst[${I}]}
157 ${olt_serial_number}= Get From Dictionary ${list_olts}[0] sn
158 Continue For Loop If "${olt_serial_number}"!="${src['olt']}"
159 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass58b23a32022-05-10 11:06:48 +0530160 ... Wait Until Keyword Succeeds ${timeout} 2s
Suchitra Vemurife309412020-12-01 19:19:52 -0800161 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
162 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
163 END
164 # Check OLT state of the rebooted OLT
165 ${olt_serial_number}= Get From Dictionary ${list_olts}[0] sn
166 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[0] sship
167 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
168 # Wait for the OLT to come back up
169 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
170 ... Check Remote System Reachability True ${olt_ssh_ip}
171 # Check OLT states
Hardik Windlass58b23a32022-05-10 11:06:48 +0530172 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemurife309412020-12-01 19:19:52 -0800173 ... Validate OLT Device ENABLED ACTIVE
174 ... REACHABLE ${olt_serial_number}
175 # Waiting extra time for the ONUs to come up
176 Sleep 60s
Suchitra Vemurife309412020-12-01 19:19:52 -0800177 #Check after reboot that ONUs are active, authenticated/DHCP/pingable
178 Run Keyword If ${has_dataplane} Clean Up Linux
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700179 FOR ${J} IN RANGE 0 ${num_olts}
180 ${olt_serial_number}= Set Variable ${list_olts}[${J}][sn]
181 ${onu_count}= Set Variable ${list_olts}[${J}][onucount]
182 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
183 ... ${olt_serial_number}
184 Set Global Variable ${of_id}
TorstenThiemedda37652021-11-11 16:22:05 +0000185 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700186 ${suppressaddsubscriber}= Set Variable If '${J}'=='0' False True
187 Perform Sanity Test Per OLT ${of_id} ${nni_port} ${olt_serial_number} ${onu_count}
188 ... ${suppressaddsubscriber}
189 END
Suchitra Vemurife309412020-12-01 19:19:52 -0800190
191
192*** Keywords ***
193Setup Suite
194 [Documentation] Set up the test suite
195 Common Test Suite Setup
196 #power_switch.robot needs it to support different vendor's power switch
197 ${switch_type}= Get Variable Value ${web_power_switch.type}
198 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
199
Hardik Windlassa119a102021-02-15 14:52:07 +0530200Clear All Devices Then Perform Setup And Sanity
201 [Documentation] Remove any devices from VOLTHA and Verify in ONOS
202 ... Create New Device through Setup and Perform Sanity
203 # Remove all devices from voltha and onos
204 Run Keyword and Ignore Error Delete All Devices and Verify
Suchitra Vemurife309412020-12-01 19:19:52 -0800205 # Execute normal test Setup Keyword
206 Setup
Hardik Windlassa119a102021-02-15 14:52:07 +0530207 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
208 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530209 Perform Sanity Test
TorstenThieme4e2168e2021-06-22 14:01:47 +0000210
211Teardown Suite
212 [Documentation] Clean up ONOS SSH connections
213 Close All ONOS SSH Connections
Hardik Windlass81375222021-12-02 10:48:01 +0000214 Run Keyword If ${has_dataplane} Clean Up All Nodes