blob: 3c6bcb0c87d056c102b7b79521b0474e45361532 [file] [log] [blame]
Hardik Windlassa3d04b92020-02-17 15:06:05 +00001# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -070034Resource ../../libraries/power_switch.robot
Hardik Windlassa3d04b92020-02-17 15:06:05 +000035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Hardik Windlassa3d04b92020-02-17 15:06:05 +000046# For below variable value, using deployment name as using grep for
47# parsing radius pod name, we can also use full radius pod name
48${RESTART_POD_NAME} radius
Hardik Windlass5d537ad2021-11-16 06:01:54 +000049${timeout} 60s
Hardik Windlassa3d04b92020-02-17 15:06:05 +000050${of_id} 0
51${logical_id} 0
Andrea Campanella86e24062020-08-05 19:09:18 +020052${uprate} 0
53${dnrate} 0
Hardik Windlassa3d04b92020-02-17 15:06:05 +000054${has_dataplane} True
Hardik Windlassffbf0602020-04-24 23:14:45 +053055${teardown_device} True
Hardik Windlassa3d04b92020-02-17 15:06:05 +000056${scripts} ../../scripts
57
Andy Baviere19991a2020-05-08 10:42:40 -070058# For dataplane bandwidth testing
59${upper_margin_pct} 105 # Allow 5% over the limit
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080060${lower_margin_pct} 90 # Allow 8% under the limit
Andy Baviere19991a2020-05-08 10:42:40 -070061${udp_rate_multiplier} 1.10 # Send UDP at bw profile limit * rate_multiplier
Andy Bavierc80a2742020-08-28 10:59:05 -070062${udp_packet_bytes} 1470 # UDP payload in bytes
Andy Baviere19991a2020-05-08 10:42:40 -070063
Hardik Windlassa3d04b92020-02-17 15:06:05 +000064# Per-test logging on failure is turned off by default; set this variable to enable
65${container_log_dir} ${None}
66
Hardik Windlass43531332021-05-19 07:07:03 +000067# logging flag to enable Collect Logs, can be passed via the command line too
68# example: -v logging:False
69${logging} True
70
Hardik Windlass4ed54502021-05-14 07:34:15 +000071# Flag specific to Soak Jobs
72${SOAK_TEST} False
73${bbsim_port} 50060
74
Hardik Windlassa3d04b92020-02-17 15:06:05 +000075*** Test Cases ***
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -070076Reboot DT ONUs Physically
77 [Documentation] This test reboots ONUs physically before execution all the tests
78 ... Test case runs only on the PODs that are configured with PowerSwitch that
79 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
Suchitra Vemuri228f8802021-03-10 13:14:17 -080080 [Tags] functionalDt PowerSwitch RebootAllDTONUs soak
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -070081 [Setup] Start Logging RebootAllDTONUs
Hardik Windlass43531332021-05-19 07:07:03 +000082 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -070083 ... AND Stop Logging RebootAllDTONUs
84 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080085 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -070086 ${src}= Set Variable ${hosts.src[${I}]}
87 ${dst}= Set Variable ${hosts.dst[${I}]}
88 Disable Switch Outlet ${src['power_switch_port']}
Hardik Windlass5e214b22021-02-26 10:37:14 +000089 Sleep 10s
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -070090 Enable Switch Outlet ${src['power_switch_port']}
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -070091 END
92
Hardik Windlass4ed54502021-05-14 07:34:15 +000093Create Soak BBSim Device
94 [Documentation] This creates and enables the BBSim device as required by the soak testing
95 ... The BBSim OLT and ONUs created as part of this test are not part of active testing
96 ... but only to mock the load on Soak POD.
97 [Tags] soak
98 [Setup] Start Logging soakPodCreateBBSimLoad
Hardik Windlass43531332021-05-19 07:07:03 +000099 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass4ed54502021-05-14 07:34:15 +0000100 ... AND Stop Logging soakPodCreateBBSimLoad
101 ${num_bbsim} Get Length ${bbsim}
Hardik Windlassea4caf72021-07-30 07:22:12 +0000102 @{bbsim_olt_of_id_list}= Create List
Hardik Windlass4ed54502021-05-14 07:34:15 +0000103 FOR ${I} IN RANGE 0 ${num_bbsim}
104 ${ip} Evaluate ${bbsim}[${I}].get("ip")
105 ${serial_number} Evaluate ${bbsim}[${I}].get("serial")
106 ${bbsim_olt_device_id}= Create Device ${ip} ${bbsim_port}
107 Log ${bbsim_olt_device_id}
108 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass8b446492021-06-10 06:25:23 +0000109 ... Validate OLT Device PREPROVISIONED UNKNOWN UNKNOWN ${bbsim_olt_device_id} by_dev_id=True
Hardik Windlass4ed54502021-05-14 07:34:15 +0000110 Enable Device ${bbsim_olt_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000111 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass4ed54502021-05-14 07:34:15 +0000112 ... Validate OLT Device ENABLED ACTIVE REACHABLE ${serial_number}
Hardik Windlassea4caf72021-07-30 07:22:12 +0000113 ${bbsim_olt_of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
Hardik Windlass4ed54502021-05-14 07:34:15 +0000114 ... ${serial_number}
Hardik Windlassea4caf72021-07-30 07:22:12 +0000115 Log ${bbsim_olt_of_id}
116 Append To List ${bbsim_olt_of_id_list} ${bbsim_olt_of_id}
Hardik Windlass4ed54502021-05-14 07:34:15 +0000117 END
118 # Extra sleep time for ONUs to come up Active
Hardik Windlassea4caf72021-07-30 07:22:12 +0000119 Sleep 60s
120 FOR ${bbsim_olt_of_id} IN @{bbsim_olt_of_id_list}
121 Provision all subscribers on device ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${ONOS_SSH_IP} ${ONOS_REST_PORT}
122 ... ${bbsim_olt_of_id}
123 ${total_onus_per_bbsim_olt}= Count Enabled UNI Ports ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
124 ... ${bbsim_olt_of_id}
125 Wait for all flows to in ADDED state ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
126 ... ${bbsim_olt_of_id} dt ${total_onus_per_bbsim_olt} 1 true
127 ... false false false true
128 END
Hardik Windlass4ed54502021-05-14 07:34:15 +0000129
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000130Sanity E2E Test for OLT/ONU on POD for DT
131 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
132 ... Validate successful DHCP/E2E ping (no EAPOL and DHCP flows) for the tech profile that is used
133 ... Traffic sent with same vlan from different RGs,
134 ... should reach the NNI port on the OLT with the expected double tagged vlan ids
135 ... Inner vlans from the RG should not change
Suchitra Vemuri228f8802021-03-10 13:14:17 -0800136 [Tags] sanityDt soak
Hardik Windlass4ed54502021-05-14 07:34:15 +0000137 [Setup] Start Logging SanityTestDt
Hardik Windlass43531332021-05-19 07:07:03 +0000138 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000139 ... AND Stop Logging SanityTestDt
Hardik Windlass4ed54502021-05-14 07:34:15 +0000140 Setup ${SOAK_TEST}
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000141 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000142 Perform Sanity Test DT
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000143
Hardik Windlass35706ba2020-02-20 08:16:42 +0000144Test Subscriber Delete and Add for DT
145 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
Hardik Windlass63d5e002020-03-06 21:07:09 +0530146 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
Hardik Windlass35706ba2020-02-20 08:16:42 +0000147 ... Delete a subscriber and validate that the pings do not succeed and state is purged
148 ... Disable and Enable the ONU (This is to replicate the existing DT behaviour)
149 ... Re-add the subscriber, and validate that the flows are present and pings are successful
Suchitra Vemuri228f8802021-03-10 13:14:17 -0800150 [Tags] functionalDt SubAddDeleteDt soak
Andy Baviere187eda2020-04-20 15:00:02 -0700151 [Setup] Start Logging SubAddDeleteDt
Hardik Windlass43531332021-05-19 07:07:03 +0000152 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass35706ba2020-02-20 08:16:42 +0000153 ... AND Stop Logging SubAddDeleteDt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800154 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000155 ${src}= Set Variable ${hosts.src[${I}]}
156 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800157 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000158 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000159 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800160 ${olt_device_id}= Get OLTDeviceID From OLT List ${src['olt']}
Andrea Campanellad848de02020-11-24 10:23:41 +0100161 ${num_of_olt_onus}= Get Num of Onus From OLT SN ${src['olt']}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000162 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
163 ... ${of_id}
164 # Remove Subscriber Access
TorstenThieme4e2168e2021-06-22 14:01:47 +0000165 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000166 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000167 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000168 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass35706ba2020-02-20 08:16:42 +0000169 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
170 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Andrea Campanellad848de02020-11-24 10:23:41 +0100171
Hardik Windlassfb5eace2020-03-26 14:49:01 +0530172 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
Andrea Campanellad848de02020-11-24 10:23:41 +0100173 ${onos_flows_count}= Evaluate 4 * ( ${num_of_olt_onus} - 1 )
Hardik Windlassfb5eace2020-03-26 14:49:01 +0530174 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
175 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Suchitra Vemuri93db32b2020-11-23 21:27:05 -0800176 ... ${of_id} ${onos_flows_count}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000177 # Verify VOLTHA flows for OLT equals twice the number of ONUS (minus ONU under test) + 1 for LLDP
Andrea Campanellad848de02020-11-24 10:23:41 +0100178 ${olt_flows}= Evaluate 2 * ( ${num_of_olt_onus} - 1 ) + 1
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000179 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows
180 ... ${olt_flows} ${olt_device_id}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000181 # Verify VOLTHA flows for ONU under test is Zero
182 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate Device Flows
183 ... ${onu_device_id} 0
184 # Disable and Re-Enable the ONU (To replicate DT current workflow)
185 # TODO: Delete and Auto-Discovery Add of ONU (not yet supported)
186 Disable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000187 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass35706ba2020-02-20 08:16:42 +0000188 ... Validate Device DISABLED UNKNOWN
189 ... REACHABLE ${src['onu']}
190 Enable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000191 Wait Until Keyword Succeeds 360s 5s
Hardik Windlass35706ba2020-02-20 08:16:42 +0000192 ... Validate Device ENABLED ACTIVE
193 ... REACHABLE ${src['onu']}
194 # Add Subscriber Access
TorstenThieme4e2168e2021-06-22 14:01:47 +0000195 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000196 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassfb5eace2020-03-26 14:49:01 +0530197 # Verify subscriber access flows are added for the ONU port
198 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass25e11702020-03-30 20:05:19 +0530199 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
200 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000201 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass35706ba2020-02-20 08:16:42 +0000202 ... Validate Device ENABLED ACTIVE
203 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
Girish Gowdrad2c0e9d2021-11-23 14:40:23 -0800204 # Workaround for issue seen in VOL-4489. Keep this workaround until VOL-4489 is fixed.
205 Run Keyword If ${has_dataplane} Reboot XGSPON ONU ${src['olt']} ${src['onu']} omci-flows-pushed
206 # Workaround ends here for issue seen in VOL-4489.
Hardik Windlass35706ba2020-02-20 08:16:42 +0000207 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000208 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass35706ba2020-02-20 08:16:42 +0000209 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
210 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass35706ba2020-02-20 08:16:42 +0000211 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800212 # Verify flows for all OLTs
213 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlass35706ba2020-02-20 08:16:42 +0000214
Hardik Windlass63d5e002020-03-06 21:07:09 +0530215Test Disable and Enable ONU for DT
216 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
217 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
218 ... Perform disable on the ONUs and validate that the pings do not succeed
219 ... Perform enable on the ONUs and validate that the pings are successful
Suchitra Vemuri228f8802021-03-10 13:14:17 -0800220 [Tags] functionalDt DisableEnableONUDt soak
Andy Baviere187eda2020-04-20 15:00:02 -0700221 [Setup] Start Logging DisableEnableONUDt
Hardik Windlass43531332021-05-19 07:07:03 +0000222 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass63d5e002020-03-06 21:07:09 +0530223 ... AND Stop Logging DisableEnableONUDt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800224 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass63d5e002020-03-06 21:07:09 +0530225 ${src}= Set Variable ${hosts.src[${I}]}
226 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800227 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass63d5e002020-03-06 21:07:09 +0530228 ${onu_device_id}= Get Device ID From SN ${src['onu']}
229 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
230 ... ${of_id}
231 Disable Device ${onu_device_id}
232 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
233 ... Validate Device DISABLED UNKNOWN
234 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-admin-lock
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000235 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000236 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassfb5eace2020-03-26 14:49:01 +0530237 # TODO: Yet to Verify on the GPON based Physical POD (VOL-2652)
Hardik Windlass63d5e002020-03-06 21:07:09 +0530238 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000239 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass63d5e002020-03-06 21:07:09 +0530240 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
241 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
242 Sleep 5s
243 Enable Device ${onu_device_id}
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700244 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
Hardik Windlass63d5e002020-03-06 21:07:09 +0530245 ... Validate Device ENABLED ACTIVE
246 ... REACHABLE ${src['onu']} onu=True onu_reason=onu-reenabled
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000247 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000248 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassfb5eace2020-03-26 14:49:01 +0530249 # TODO: Yet to Verify on the GPON based Physical POD (VOL-2652)
Hardik Windlass63d5e002020-03-06 21:07:09 +0530250 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000251 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass63d5e002020-03-06 21:07:09 +0530252 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
253 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass63d5e002020-03-06 21:07:09 +0530254 END
255
Hardik Windlassaaea3402020-03-10 19:45:45 +0530256Test Disable and Delete OLT for DT
257 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
258 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
259 ... Perform disable on the OLT and validate ONUs state and that the pings do not succeed
260 ... Perform delete on the OLT, Re-do Setup (Recreate the OLT) and Perform Sanity Test DT
Hardik Windlass43531332021-05-19 07:07:03 +0000261 [Tags] functionalDt DisableDeleteOLTDt soak
Suchitra Vemuri221398a2020-05-29 22:45:17 -0700262 [Setup] Start Logging DisableDeleteOLTDt
Hardik Windlass43531332021-05-19 07:07:03 +0000263 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlassaaea3402020-03-10 19:45:45 +0530264 ... AND Stop Logging DisableDeleteOLTDt
Hardik Windlassaaea3402020-03-10 19:45:45 +0530265 # Disable and Validate OLT Device
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800266 FOR ${I} IN RANGE 0 ${olt_count}
267 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
268 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
269 Disable Device ${olt_device_id}
270 ${of_id}= Get ofID From OLT List ${olt_serial_number}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000271 Wait Until Keyword Succeeds ${timeout} 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800272 ... Validate OLT Device DISABLED UNKNOWN REACHABLE
273 ... ${olt_serial_number}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800274 ${num_onus}= Set Variable ${list_olts}[${I}][onucount]
275 # Validate ONUs
276 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONUs After OLT Disable
Andrea Campanella4e507882020-11-26 11:33:11 +0100277 ... ${num_onus} ${olt_serial_number}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800278 # Verify ONOS Flows
Matteo Scandolo8b6dc262021-11-22 12:30:27 -0800279 # When we disable the device we shouldn't have any flows provisioned on it
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800280 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
281 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Matteo Scandolo8b6dc262021-11-22 12:30:27 -0800282 ... ${of_id} 0
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800283 # Verify VOLTHA Flows
Hardik Windlass3a294892021-11-23 17:05:29 +0000284 # OLT should only have 1 default flow for LLDP
285 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows 1
Matteo Scandolo8b6dc262021-11-22 12:30:27 -0800286 ... ${olt_device_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800287 ${List_ONU_Serial} Create List
288 Set Suite Variable ${List_ONU_Serial}
Andrea Campanella99a548b2020-11-18 09:50:32 +0100289 Build ONU SN List ${List_ONU_Serial} ${olt_serial_number} ${num_onus}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800290 Log ${List_ONU_Serial}
291 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
Matteo Scandolo8b6dc262021-11-22 12:30:27 -0800292 ... ${List_ONU_Serial} 0
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800293 # Delete OLT and Validate Empty Device List
294 Delete Device ${olt_device_id}
Andrea Campanella4e507882020-11-26 11:33:11 +0100295 # Check that the OLT and the ONUs are actually removed from the system
296 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device Removed
297 ... ${olt_serial_number}
298 Run Keyword and Continue On Failure Validate all ONUS for OLT Removed ${num_all_onus} ${hosts}
299 ... ${olt_serial_number} ${timeout}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000300 Wait Until Keyword Succeeds ${timeout} 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800301 ... Verify Device Flows Removed ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800302 END
Hardik Windlassaaea3402020-03-10 19:45:45 +0530303 # Re-do Setup (Recreate the OLT) and Perform Sanity Test DT
Hardik Windlass43531332021-05-19 07:07:03 +0000304 Run Keyword Setup ${SOAK_TEST}
Suchitra Vemuriae9cecb2020-03-12 12:59:14 -0700305 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000306 Perform Sanity Test DT
Hardik Windlassaaea3402020-03-10 19:45:45 +0530307
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530308Test Disable and Enable OLT for DT
309 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
310 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
311 ... Perform disable on the OLT and validate that the pings do not succeed
312 ... Perform enable on the OLT and validate that the pings are successful
Suchitra Vemuri228f8802021-03-10 13:14:17 -0800313 [Tags] functionalDt DisableEnableOLTDt soak
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530314 [Setup] Start Logging DisableEnableOLTDt
Hardik Windlass43531332021-05-19 07:07:03 +0000315 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530316 ... AND Stop Logging DisableEnableOLTDt
317 # Disable and Validate OLT Device
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800318 FOR ${I} IN RANGE 0 ${olt_count}
319 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
320 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
321 ${rc} ${output}= Run and Return Rc and Output
322 ... ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
323 Should Be Equal As Integers ${rc} 0
324 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN REACHABLE
325 ... ${olt_serial_number}
326 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800327 # Validate ONUs
328 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530329 ${src}= Set Variable ${hosts.src[${I}]}
330 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800331 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530332 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000333 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass7c801e62020-05-04 19:33:21 +0530334 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000335 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000336 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530337 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000338 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass7c801e62020-05-04 19:33:21 +0530339 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
340 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
341 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000342 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530343 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
344 # Delete ONU Device (To replicate DT workflow)
345 Delete Device ${onu_device_id}
346 END
Hardik Windlass2b37e712020-06-12 02:13:17 +0530347 Sleep 5s
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530348 # Enable the OLT back and check ONU, OLT status are back to "ACTIVE"
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800349 FOR ${I} IN RANGE 0 ${olt_count}
350 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
351 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
352 Enable Device ${olt_device_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800353 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE REACHABLE
354 ... ${olt_serial_number}
355 #TODO: Update for PON_OLT ETHERNET_NNI
356 #Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Port Types
357 #... PON_OLT ETHERNET_NNI
358 END
Hardik Windlass7c801e62020-05-04 19:33:21 +0530359 # Waiting extra time for the ONUs to come up
360 Sleep 60s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800361 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000362 Perform Sanity Test DT
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530363
364Test Delete and ReAdd OLT for DT
365 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
366 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
367 ... Disable and Delete the OLT
368 ... Create/Enable the same OLT again
369 ... Validate DHCP/E2E pings succeed for all the ONUs connected to the OLT
Hardik Windlass43531332021-05-19 07:07:03 +0000370 [Tags] functionalDt DeleteReAddOLTDt soak
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530371 [Setup] Start Logging DeleteReAddOLTDt
Hardik Windlass43531332021-05-19 07:07:03 +0000372 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530373 ... AND Stop Logging DeleteReAddOLTDt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800374 FOR ${I} IN RANGE 0 ${olt_count}
375 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
376 ${of_id}= Get ofID From OLT List ${olt_serial_number}
377 Delete Device and Verify ${olt_serial_number}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000378 Wait Until Keyword Succeeds ${timeout} 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800379 ... Verify Device Flows Removed ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800380 END
381 # Recreate the OLTs
Hardik Windlass43531332021-05-19 07:07:03 +0000382 Setup ${SOAK_TEST}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800383 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000384 Perform Sanity Test DT
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530385
386Test Disable ONUs and OLT Then Delete ONUs and OLT for DT
387 [Documentation] On deployed POD, disable the ONU, disable the OLT and then delete ONU and OLT.
388 ... This TC is to confirm that ONU removal is not impacting OLT
389 ... Devices will be removed during the execution of this TC
390 ... so calling setup at the end to add the devices back to avoid the confusion.
391 [Tags] functionalDt DisableDeleteONUOLTDt
392 [Setup] Start Logging DisableDeleteONUOLTDt
Hardik Windlass43531332021-05-19 07:07:03 +0000393 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530394 ... AND Stop Logging DisableDeleteONUOLTDt
Suchitra Vemuri02170552021-03-23 16:40:50 -0700395 @{onu_reason}= Create List initial-mib-downloaded omci-flows-pushed
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800396 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530397 ${src}= Set Variable ${hosts.src[${I}]}
398 ${dst}= Set Variable ${hosts.dst[${I}]}
399 ${onu_device_id}= Get Device ID From SN ${src['onu']}
400 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
401 ... Validate Device ENABLED ACTIVE
Suchitra Vemuri02170552021-03-23 16:40:50 -0700402 ... REACHABLE ${src['onu']} onu=True onu_reason=${onu_reason}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000403 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530404 ... Validate OLT Device ENABLED ACTIVE
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800405 ... REACHABLE ${src['olt']}
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530406 Disable Device ${onu_device_id}
407 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
408 ... Validate Device DISABLED UNKNOWN
409 ... REACHABLE ${src['onu']} onu=false
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000410 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530411 ... Validate OLT Device ENABLED ACTIVE
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800412 ... REACHABLE ${src['olt']}
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530413 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800414 # Disable all OLTs
415 FOR ${I} IN RANGE 0 ${olt_count}
416 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
417 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
418 ${rc} ${output}= Run and Return Rc and Output
419 ... ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
420 Should Be Equal As Integers ${rc} 0
421 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN REACHABLE
422 ... ${olt_serial_number}
423 END
424 # Validate ONUs after OLT disable
425 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530426 ${src}= Set Variable ${hosts.src[${I}]}
427 ${dst}= Set Variable ${hosts.dst[${I}]}
428 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000429 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530430 ... Validate Device DISABLED DISCOVERED
431 ... UNREACHABLE ${src['onu']} onu=false
432 Delete Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000433 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530434 ... Validate OLT Device DISABLED UNKNOWN
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800435 ... REACHABLE ${src['olt']}
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530436 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800437 # Delete all OLTs
438 Delete All Devices and Verify
439
440 #Delete Device ${olt_device_id}
441 #TODO: Fix the following assertion
442 #Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Test Empty Device List
443 #Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
444 #... Verify Device Flows Removed ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
445
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530446 # Re-do Setup (Recreate the OLT) and Perform Sanity Test DT
447 Run Keyword Setup
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800448 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000449 Perform Sanity Test DT
Hardik Windlass3b531fa2020-04-23 18:51:44 +0530450
Andy Baviere19991a2020-05-08 10:42:40 -0700451Data plane verification using TCP for DT
452 [Documentation] Test bandwidth profile is met and not exceeded for each subscriber.
453 ... Assumes iperf3 and jq installed on client and iperf -s running on DHCP server
Matteo Scandoloab48f9e2021-06-11 17:08:00 +0200454 [Tags] non-critical dataplaneDt BandwidthProfileTCPDt VOL-3061 soakDataplane
Andy Bavierdfbcaef2020-05-27 15:34:18 -0700455 [Setup] Start Logging BandwidthProfileTCPDt
Hardik Windlass43531332021-05-19 07:07:03 +0000456 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Andy Bavierdfbcaef2020-05-27 15:34:18 -0700457 ... AND Stop Logging BandwidthProfileTCPDt
Andy Baviere19991a2020-05-08 10:42:40 -0700458 Pass Execution If '${has_dataplane}'=='False' Bandwidth profile validation can be done only in
459 ... physical pod. Skipping this test in BBSIM.
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000460 Run Keyword If '${SOAK_TEST}'=='False' Clear All Devices Then Create New Device
Hardik Windlass62db9d92021-05-17 09:32:40 +0000461 ... ELSE Setup Soak
Suchitra Vemuria4e3e072020-11-06 14:23:47 -0800462 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000463 Perform Sanity Test DT
Suchitra Vemuria4e3e072020-11-06 14:23:47 -0800464
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800465 #${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
466 #... ${olt_serial_number}
467 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Baviere19991a2020-05-08 10:42:40 -0700468 ${src}= Set Variable ${hosts.src[${I}]}
469 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800470 ${of_id}= Get ofID From OLT List ${src['olt']}
Andy Bavierf6ab19c2020-05-14 10:34:47 -0700471 # Check for iperf3 and jq tools
472 ${stdout} ${stderr} ${rc}= Execute Remote Command which iperf3 jq
473 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
474 Pass Execution If ${rc} != 0 Skipping test: iperf3 / jq not found on the RG
475
Andy Baviere19991a2020-05-08 10:42:40 -0700476 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
477 ... ${of_id}
478 ${subscriber_id}= Set Variable ${of_id}/${onu_port}
479 ${bandwidth_profile_name} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id}
480 ... upstreamBandwidthProfile
Girish Gowdra21ec25d2021-05-24 10:07:01 -0700481 ${us_cir} ${us_cbs} ${us_pir} ${us_pbs} ${us_gir}= Get Bandwidth Profile Details Ietf Rest
482 ... ${bandwidth_profile_name}
483 ${limiting_bw_value_upstream}= Set Variable If ${us_pir} != 0 ${us_pir} ${us_gir}
Andy Baviere19991a2020-05-08 10:42:40 -0700484 ${bandwidth_profile_name} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id}
485 ... downstreamBandwidthProfile
Girish Gowdra21ec25d2021-05-24 10:07:01 -0700486 ${ds_cir} ${ds_cbs} ${ds_pir} ${ds_pbs} ${ds_gir}= Get Bandwidth Profile Details Ietf Rest
487 ... ${bandwidth_profile_name}
488 ${limiting_bw_value_dnstream}= Set Variable If ${ds_pir} != 0 ${ds_pir} ${ds_gir}
Andy Baviere19991a2020-05-08 10:42:40 -0700489
490 # Stream TCP packets from RG to server
491 ${updict}= Run Iperf3 Test Client ${src} server=${dst['dp_iface_ip_qinq']}
492 ... args=-t 30
493 ${actual_upstream_bw_used}= Evaluate ${updict['end']['sum_received']['bits_per_second']}/1000
494
495 # Stream TCP packets from server to RG
496 ${dndict}= Run Iperf3 Test Client ${src} server=${dst['dp_iface_ip_qinq']}
497 ... args=-R -t 30
498 ${actual_dnstream_bw_used}= Evaluate ${dndict['end']['sum_received']['bits_per_second']}/1000
499
500 ${pct_limit_up}= Evaluate 100*${actual_upstream_bw_used}/${limiting_bw_value_upstream}
501 ${pct_limit_dn}= Evaluate 100*${actual_dnstream_bw_used}/${limiting_bw_value_dnstream}
502 Log Up: bwprof ${limiting_bw_value_upstream}Kbps, got ${actual_upstream_bw_used}Kbps (${pct_limit_up}%)
503 Log Down: bwprof ${limiting_bw_value_dnstream}Kbps, got ${actual_dnstream_bw_used}Kbps (${pct_limit_dn}%)
504
505 Should Be True ${pct_limit_up} <= ${upper_margin_pct}
506 ... The upstream bandwidth exceeded the limit (${pct_limit_up}% of limit)
Andy Bavier7613b9a2020-07-08 07:25:47 -0700507 # VOL-3125: downstream bw limit not enforced. Uncomment when fixed.
508 #Should Be True ${pct_limit_dn} <= ${upper_margin_pct}
509 #... The downstream bandwidth exceeded the limit (${pct_limit_dn}% of limit)
Andy Baviere19991a2020-05-08 10:42:40 -0700510 Should Be True ${pct_limit_up} >= ${lower_margin_pct}
511 ... The upstream bandwidth guarantee was not met (${pct_limit_up}% of resv)
512 Should Be True ${pct_limit_dn} >= ${lower_margin_pct}
513 ... The downstream bandwidth guarantee was not met (${pct_limit_dn}% of resv)
514 END
515
516Data plane verification using UDP for DT
517 [Documentation] Test bandwidth profile is met and not exceeded for each subscriber.
518 ... Assumes iperf3 and jq installed on client and iperf -s running on DHCP server
Matteo Scandoloab48f9e2021-06-11 17:08:00 +0200519 [Tags] non-critical dataplaneDt BandwidthProfileUDPDt VOL-3061 soakDataplane
Andy Bavierdfbcaef2020-05-27 15:34:18 -0700520 [Setup] Start Logging BandwidthProfileUDPDt
Hardik Windlass43531332021-05-19 07:07:03 +0000521 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Andy Bavierdfbcaef2020-05-27 15:34:18 -0700522 ... AND Stop Logging BandwidthProfileUDPDt
Andy Baviere19991a2020-05-08 10:42:40 -0700523 Pass Execution If '${has_dataplane}'=='False' Bandwidth profile validation can be done only in
524 ... physical pod. Skipping this test in BBSIM.
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800525 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Baviere19991a2020-05-08 10:42:40 -0700526 ${src}= Set Variable ${hosts.src[${I}]}
527 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800528 ${of_id}= Get ofID From OLT List ${src['olt']}
Andy Bavier7613b9a2020-07-08 07:25:47 -0700529
530 # Check for iperf3 and jq tools
531 ${stdout} ${stderr} ${rc}= Execute Remote Command which iperf3 jq
532 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
533 Pass Execution If ${rc} != 0 Skipping test: iperf3 / jq not found on the RG
534
Andy Baviere19991a2020-05-08 10:42:40 -0700535 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
536 ... ${of_id}
537 ${subscriber_id}= Set Variable ${of_id}/${onu_port}
538 ${bandwidth_profile_name} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id}
539 ... upstreamBandwidthProfile
Girish Gowdra21ec25d2021-05-24 10:07:01 -0700540 ${us_cir} ${us_cbs} ${us_pir} ${us_pbs} ${us_gir}= Get Bandwidth Profile Details Ietf Rest
541 ... ${bandwidth_profile_name}
542 ${limiting_bw_value_upstream}= Set Variable If ${us_pir} != 0 ${us_pir} ${us_gir}
Andy Baviere19991a2020-05-08 10:42:40 -0700543 ${bandwidth_profile_name} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id}
544 ... downstreamBandwidthProfile
Girish Gowdra21ec25d2021-05-24 10:07:01 -0700545 ${ds_cir} ${ds_cbs} ${ds_pir} ${ds_pbs} ${ds_gir}= Get Bandwidth Profile Details Ietf Rest
546 ... ${bandwidth_profile_name}
547 ${limiting_bw_value_dnstream}= Set Variable If ${ds_pir} != 0 ${ds_pir} ${ds_gir}
Andy Baviere19991a2020-05-08 10:42:40 -0700548
549 # Stream UDP packets from RG to server
Andrea Campanella1d34bfc2020-08-06 09:58:49 +0200550 ${uprate}= Run Keyword If ${limiting_bw_value_upstream} != 1000000
551 ... Evaluate ${limiting_bw_value_upstream}*${udp_rate_multiplier}
Andrea Campanella580e0492020-08-05 15:35:48 +0200552 ... ELSE
Andrea Campanella1d34bfc2020-08-06 09:58:49 +0200553 ... Set Variable ${limiting_bw_value_upstream}
Andrea Campanella580e0492020-08-05 15:35:48 +0200554
Andy Baviere19991a2020-05-08 10:42:40 -0700555 ${updict}= Run Iperf3 Test Client ${src} server=${dst['dp_iface_ip_qinq']}
Andy Bavier7dc50622020-05-28 14:34:12 -0700556 ... args=-u -b ${uprate}K -t 30 -l ${udp_packet_bytes} --pacing-timer 0
Andy Baviere19991a2020-05-08 10:42:40 -0700557 # With UDP test, bits per second is the sending rate. Multiply by the loss rate to get the throughput.
558 ${actual_upstream_bw_used}= Evaluate
559 ... (100 - ${updict['end']['sum']['lost_percent']})*${updict['end']['sum']['bits_per_second']}/100000
560
561 # Stream UDP packets from server to RG
Andrea Campanella1d34bfc2020-08-06 09:58:49 +0200562 ${dnrate}= Run Keyword If ${limiting_bw_value_dnstream} != 1000000
563 ... Evaluate ${limiting_bw_value_dnstream}*${udp_rate_multiplier}
Andrea Campanella580e0492020-08-05 15:35:48 +0200564 ... ELSE
Andrea Campanella1d34bfc2020-08-06 09:58:49 +0200565 ... Set Variable ${limiting_bw_value_dnstream}
Andy Baviere19991a2020-05-08 10:42:40 -0700566 ${dndict}= Run Iperf3 Test Client ${src} server=${dst['dp_iface_ip_qinq']}
Andy Bavier7dc50622020-05-28 14:34:12 -0700567 ... args=-u -b ${dnrate}K -R -t 30 -l ${udp_packet_bytes} --pacing-timer 0
Andy Baviere19991a2020-05-08 10:42:40 -0700568 # With UDP test, bits per second is the sending rate. Multiply by the loss rate to get the throughput.
569 ${actual_dnstream_bw_used}= Evaluate
570 ... (100 - ${dndict['end']['sum']['lost_percent']})*${dndict['end']['sum']['bits_per_second']}/100000
571
572 ${pct_limit_up}= Evaluate 100*${actual_upstream_bw_used}/${limiting_bw_value_upstream}
573 ${pct_limit_dn}= Evaluate 100*${actual_dnstream_bw_used}/${limiting_bw_value_dnstream}
574 Log Up: bwprof ${limiting_bw_value_upstream}Kbps, got ${actual_upstream_bw_used}Kbps (${pct_limit_up}%)
575 Log Down: bwprof ${limiting_bw_value_dnstream}Kbps, got ${actual_dnstream_bw_used}Kbps (${pct_limit_dn}%)
576
577 Should Be True ${pct_limit_up} <= ${upper_margin_pct}
578 ... The upstream bandwidth exceeded the limit (${pct_limit_up}% of limit)
Andy Bavier7613b9a2020-07-08 07:25:47 -0700579 # VOL-3125: downstream bw limit not enforced. Uncomment when fixed.
580 #Should Be True ${pct_limit_dn} <= ${upper_margin_pct}
581 #... The downstream bandwidth exceeded the limit (${pct_limit_dn}% of limit)
Andy Baviere19991a2020-05-08 10:42:40 -0700582 Should Be True ${pct_limit_up} >= ${lower_margin_pct}
583 ... The upstream bandwidth guarantee was not met (${pct_limit_up}% of resv)
584 Should Be True ${pct_limit_dn} >= ${lower_margin_pct}
585 ... The downstream bandwidth guarantee was not met (${pct_limit_dn}% of resv)
586 END
587
Andy Bavier7613b9a2020-07-08 07:25:47 -0700588Validate parsing of data traffic through voltha using tech profile
589 [Documentation] Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
590 ... Prerequisite tools : Tcpdump and Mausezahn traffic generator on both RG and DHCP/BNG VMs
591 ... Install jq tool to read json file, where test suite is being running
Girish Gowdrabbfbbc32021-05-17 14:36:58 -0700592 ... This test sends TCP packets with pbits between 0 and 7 and validates that
Andy Bavier7613b9a2020-07-08 07:25:47 -0700593 ... the pbits are preserved by the PON.
Hardik Windlass62db9d92021-05-17 09:32:40 +0000594 [Tags] dataplaneDt TechProfileDt VOL-3291 soakDataplane
Andy Bavier7613b9a2020-07-08 07:25:47 -0700595 [Setup] Start Logging TechProfileDt
Hardik Windlass43531332021-05-19 07:07:03 +0000596 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Andy Bavier7613b9a2020-07-08 07:25:47 -0700597 ... AND Stop Logging TechProfileDt
598 Pass Execution If '${has_dataplane}'=='False'
599 ... Skipping test: Technology profile validation can be done only in physical pod
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800600 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier7613b9a2020-07-08 07:25:47 -0700601 ${src}= Set Variable ${hosts.src[${I}]}
602 ${dst}= Set Variable ${hosts.dst[${I}]}
603
Andy Bavier275414a2020-07-16 13:07:17 -0700604 ${src_iface_name}= Fetch From Left ${src['dp_iface_name']} .
605
Andy Bavier7613b9a2020-07-08 07:25:47 -0700606 ${bng_ip}= Get Variable Value ${dst['noroot_ip']}
607 ${bng_user}= Get Variable Value ${dst['noroot_user']}
608 ${bng_pass}= Get Variable Value ${dst['noroot_pass']}
609 Pass Execution If "${bng_ip}" == "${NONE}" or "${bng_user}" == "${NONE}" or "${bng_pass}" == "${NONE}"
610 ... Skipping test: credentials for BNG login required in deployment config
611
612 ${stdout} ${stderr} ${rc}= Execute Remote Command which mausezahn tcpdump
613 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
614 Pass Execution If ${rc} != 0 Skipping test: mausezahn / tcpdump not found on the RG
615 ${stdout} ${stderr} ${rc}= Execute Remote Command which mausezahn tcpdump
616 ... ${bng_ip} ${bng_user} ${bng_pass} ${dst['container_type']} ${dst['container_name']}
617 Pass Execution If ${rc} != 0 Skipping test: mausezahn / tcpdump not found on the BNG
618 Log Upstream test
619 Run Keyword If ${has_dataplane} Create traffic with each pbit and capture at other end
Andy Bavier275414a2020-07-16 13:07:17 -0700620 ... ${dst['dp_iface_ip_qinq']} ${dst['dp_iface_name']} ${src_iface_name}
Girish Gowdra66cd95d2021-07-08 10:31:54 -0700621 ... 0 tcp ${src['c_tag']} ${src['s_tag']} upstream vlan
Andy Bavier7613b9a2020-07-08 07:25:47 -0700622 ... ${bng_ip} ${bng_user} ${bng_pass} ${dst['container_type']} ${dst['container_name']}
623 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
624 Log Downstream test
625 ${rg_ip} ${stderr} ${rc}= Execute Remote Command
626 ... ifconfig ${src['dp_iface_name']} | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1 }'
627 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
628 Should Be Equal As Integers ${rc} 0 Could not get RG's IP address
629 Run Keyword If ${has_dataplane} Create traffic with each pbit and capture at other end
Girish Gowdra66cd95d2021-07-08 10:31:54 -0700630 ... ${rg_ip} ${src_iface_name} ${dst['dp_iface_name']}
631 ... 0 tcp ${src['c_tag']} ${src['s_tag']} downstream tcp
Andy Bavier7613b9a2020-07-08 07:25:47 -0700632 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
633 ... ${bng_ip} ${bng_user} ${bng_pass} ${dst['container_type']} ${dst['container_name']}
634 END
635
Hardik Windlass16cdf962020-04-29 15:26:50 +0530636Test Disable and Enable OLT PON Port for DT
637 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
638 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
639 ... Perform disable on the OLT PON Port and validate that the pings do not succeed
640 ... Perform enable on the OLT PON Port and validate that the pings are successful
Hardik Windlass43531332021-05-19 07:07:03 +0000641 [Tags] functionalDt DisableEnableOltPonPortDt VOL-2577 soak
Hardik Windlass16cdf962020-04-29 15:26:50 +0530642 [Setup] Start Logging DisableEnableOltPonPortDt
Hardik Windlass43531332021-05-19 07:07:03 +0000643 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass16cdf962020-04-29 15:26:50 +0530644 ... AND Stop Logging DisableEnableOltPonPortDt
Hardik Windlass41701e82021-01-12 17:44:44 +0000645 FOR ${I} IN RANGE 0 ${olt_count}
646 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
647 Disable Enable PON Port Per OLT DT ${olt_serial_number}
Hardik Windlass16cdf962020-04-29 15:26:50 +0530648 END
649
Hardik Windlass91cc5292021-07-26 10:09:40 +0000650Data plane Bandwidth profile update verification for DT
651 [Documentation] Test bandwidth profile is updated for one subscriber and not changed for other subscribers.
652 ... Assumes iperf3 and jq installed on client and iperf -s running on DHCP server
653 ... Assumes Default and User_Bandwidth2 profiles are configured as bandwidth profiles
654 [Tags] non-critical dataplaneDt BandwidthProfileUpdateTCPDt VOL-2549
655 [Setup] Start Logging BandwidthProfileUpdateTCPDt
656 [Teardown] Run Keywords Collect Logs
657 ... AND Stop Logging BandwidthProfileUpdateTCPDt
658 ... AND Send File To Onos ${CURDIR}/../../tests/data/${POD_NAME}-sadis-DT.json
659 Pass Execution If '${has_dataplane}'=='False'
660 ... Bandwidth profile validation can be done only in physical pod. Skipping this test in BBSIM.
661 Delete All Devices And Verify
662 Run Keyword If ${has_dataplane} Clean Up Linux
663 Setup
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000664 Perform Sanity Test DT
Hardik Windlass91cc5292021-07-26 10:09:40 +0000665 # Update the Bandwidth Profile for the First Subscriber under test
666 ${src}= Set Variable ${hosts.src[${0}]}
667 ${dst}= Set Variable ${hosts.dst[${0}]}
668 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000669 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass91cc5292021-07-26 10:09:40 +0000670 ${onu_device_id}= Get Device ID From SN ${src['onu']}
671 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
672 ... ${of_id} ${src['uni_id']}
673 ${subscriber_id}= Set Variable ${of_id}/${onu_port}
674 ${oldBwName} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id} upstreamBandwidthProfile
675 # Delete the existing subscriber
676 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
677 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
678 # Verify VOLTHA flows for ONU under test is Zero
679 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate Device Flows
680 ... ${onu_device_id} 0
681 # Disable and Re-Enable the ONU (To replicate DT current workflow)
682 # TODO: Delete and Auto-Discovery Add of ONU (not yet supported)
683 Disable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000684 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass91cc5292021-07-26 10:09:40 +0000685 ... Validate Device DISABLED UNKNOWN
686 ... REACHABLE ${src['onu']}
687 Enable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000688 Wait Until Keyword Succeeds 360s 5s
Hardik Windlass91cc5292021-07-26 10:09:40 +0000689 ... Validate Device ENABLED ACTIVE
690 ... REACHABLE ${src['onu']}
691 # Change the bandwidth profile and load the configuration
692 ${newBwName} Set Variable If ${oldBwName} == 'Default' 'User_Bandwidth2' 'Default'
693 ${cmd} Catenate
694 ... sed 's/upstreamBandwidthProfile": "${oldBwName}"/upstreamBandwidthProfile": "${newBwName}"/g'
695 ... ${CURDIR}/../../tests/data/${POD_NAME}-sadis-DT.json > ${CURDIR}/../../tests/data/${POD_NAME}-sadis-new.json
696 ${rc} Run and Return RC ${cmd}
697 Send File To Onos ${CURDIR}/../../tests/data/${POD_NAME}-sadis-new.json
698 # Re-add the subscriber with new bandwidth profile
699 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
700 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
701 # Verify subscriber access flows are added for the ONU port
702 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
703 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
704 ... ${onu_port} ${nni_port} ${src['s_tag']}
705 # Verify ONU state in voltha
706 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
707 ... ENABLED ACTIVE REACHABLE
708 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
709 # Verify Meters in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000710 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass91cc5292021-07-26 10:09:40 +0000711 ... Verify Meters in ONOS Ietf ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} ${onu_port}
712 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
713 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
714 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
715 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
716 ... ${dst['container_name']}
717 # Verify new bandwidth profile is applied and other subscribers are working fine
718 ${checkBwName} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id} upstreamBandwidthProfile
719 Should Be Equal ${checkBwName} ${newBwName} The bandwidth profile is not updated properly
720 FOR ${I} IN RANGE 0 ${num_all_onus}
721 ${src}= Set Variable ${hosts.src[${I}]}
722 ${dst}= Set Variable ${hosts.dst[${I}]}
723 ${of_id}= Get ofID From OLT List ${src['olt']}
724 # Check for iperf3 and jq tools
725 ${stdout} ${stderr} ${rc}= Execute Remote Command which iperf3 jq
726 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
727 Pass Execution If ${rc} != 0 Skipping test: iperf3 / jq not found on the RG
728
729 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
730 ... ${of_id}
731 ${subscriber_id}= Set Variable ${of_id}/${onu_port}
732 ${bandwidth_profile_name} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id}
733 ... upstreamBandwidthProfile
734 ${us_cir} ${us_cbs} ${us_pir} ${us_pbs} ${us_gir}= Get Bandwidth Profile Details Ietf Rest
735 ... ${bandwidth_profile_name}
736 ${limiting_bw_value_upstream}= Set Variable If ${us_pir} != 0 ${us_pir} ${us_gir}
737 ${bandwidth_profile_name} Get Bandwidth Profile Name For Given Subscriber ${subscriber_id}
738 ... downstreamBandwidthProfile
739 ${ds_cir} ${ds_cbs} ${ds_pir} ${ds_pbs} ${ds_gir}= Get Bandwidth Profile Details Ietf Rest
740 ... ${bandwidth_profile_name}
741 ${limiting_bw_value_dnstream}= Set Variable If ${ds_pir} != 0 ${ds_pir} ${ds_gir}
742
743 # Stream TCP packets from RG to server
744 ${updict}= Run Iperf3 Test Client ${src} server=${dst['dp_iface_ip_qinq']}
745 ... args=-t 30
746 ${actual_upstream_bw_used}= Evaluate ${updict['end']['sum_received']['bits_per_second']}/1000
747
748 # Stream TCP packets from server to RG
749 ${dndict}= Run Iperf3 Test Client ${src} server=${dst['dp_iface_ip_qinq']}
750 ... args=-R -t 30
751 ${actual_dnstream_bw_used}= Evaluate ${dndict['end']['sum_received']['bits_per_second']}/1000
752
753 ${pct_limit_up}= Evaluate 100*${actual_upstream_bw_used}/${limiting_bw_value_upstream}
754 ${pct_limit_dn}= Evaluate 100*${actual_dnstream_bw_used}/${limiting_bw_value_dnstream}
755 Log Up: bwprof ${limiting_bw_value_upstream}Kbps, got ${actual_upstream_bw_used}Kbps (${pct_limit_up}%)
756 Log Down: bwprof ${limiting_bw_value_dnstream}Kbps, got ${actual_dnstream_bw_used}Kbps (${pct_limit_dn}%)
757
758 Should Be True ${pct_limit_up} <= ${upper_margin_pct}
759 ... The upstream bandwidth exceeded the limit (${pct_limit_up}% of limit)
760 # VOL-3125: downstream bw limit not enforced. Uncomment when fixed.
761 #Should Be True ${pct_limit_dn} <= ${upper_margin_pct}
762 #... The downstream bandwidth exceeded the limit (${pct_limit_dn}% of limit)
763 Should Be True ${pct_limit_up} >= ${lower_margin_pct}
764 ... The upstream bandwidth guarantee was not met (${pct_limit_up}% of resv)
765 Should Be True ${pct_limit_dn} >= ${lower_margin_pct}
766 ... The downstream bandwidth guarantee was not met (${pct_limit_dn}% of resv)
767 END
768
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000769*** Keywords ***
770Setup Suite
771 [Documentation] Set up the test suite
772 Common Test Suite Setup
Hung-Wei Chiucc6e6552020-06-04 12:57:38 -0700773 #Restore all ONUs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800774 #Run Keyword If ${has_dataplane} RestoreONUs ${num_all_onus}
Suchitra Vemuri1d8e8172020-06-17 17:43:56 -0700775 #power_switch.robot needs it to support different vendor's power switch
776 ${switch_type}= Get Variable Value ${web_power_switch.type}
777 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlassa3d04b92020-02-17 15:06:05 +0000778
779Clear All Devices Then Create New Device
780 [Documentation] Remove any devices from VOLTHA and ONOS
781 # Remove all devices from voltha and nos
782 Delete All Devices and Verify
783 # Execute normal test Setup Keyword
784 Setup
Girish Gowdrad2c0e9d2021-11-23 14:40:23 -0800785
786Reboot XGSPON ONU
787 [Documentation] Reboots the XGSPON ONU and verifies the ONU state after the reboot
788 [Arguments] ${olt_sn} ${onu_sn} ${reason}
789 FOR ${I} IN RANGE 0 ${num_olts}
790 ${serial_number} Evaluate ${olts}[${I}].get("serial")
791 Continue For Loop If "${serial_number}"!="${olt_sn}"
792 ${board_tech} Evaluate ${olts}[${I}].get("board_technology")
793 ${onu_device_id}= Get Device ID From SN ${onu_sn}
794 Run Keyword If "${board_tech}"=="XGS-PON" Run Keywords
795 ... Reboot Device ${onu_device_id}
796 ... AND Wait Until Keyword Succeeds 120s 5s
797 ... Validate Device ENABLED ACTIVE
798 ... REACHABLE ${onu_sn} onu=True onu_reason=${reason}
799 END