reviving att-wf "workflow" test
Change-Id: I531d6c59c123882968e0785ca0cde2eb483fac35
diff --git a/src/test/cord-api/Tests/WorkflowValidations/ATT_Workflow.robot b/src/test/cord-api/Tests/WorkflowValidations/ATT_Workflow.robot
index 5b7498f..97befea 100644
--- a/src/test/cord-api/Tests/WorkflowValidations/ATT_Workflow.robot
+++ b/src/test/cord-api/Tests/WorkflowValidations/ATT_Workflow.robot
@@ -5,97 +5,62 @@
Library Collections
Library String
Library OperatingSystem
+Resource ../../Framework/utils/utils.robot
Suite Setup Setup
Suite Teardown Teardown
+Test Template Send Event and Verify
*** Variables ***
${cord_kafka} cord-kafka
${server_ip} xos-chameleon
${server_port} 9101
${subscriber_api} /xosapi/v1/rcord/rcordsubscribers
+${att_wf_api} /xosapi/v1/att-workflow-driver/attworkflowdriverservices
+${volt_api} /xosapi/v1/volt/voltservices
${att_si_api} /xosapi/v1/att-workflow-driver/attworkflowdriverserviceinstances
+${att_whitelist_api} /xosapi/v1/att-workflow-driver/attworkflowdriverwhitelistentries
${onu_device_api} /xosapi/v1/volt/onudevices
+${olt_api} /xosapi/v1/volt/oltdevices
+${pon_ports_api} /xosapi/v1/volt/ponports
+${uni_ports_api} /xosapi/v1/volt/uniports
${onu_serial_no} onudevice123
${onu_invalid_sn} invalid_serial_no
-${mac_address} 00:AA:00:00:00:01
-${ip_address} 192.168.3.5
${deviceId} of:robot_test
${ponportno} 10
${uniportno} 100
+${mac_address} 00:AA:00:00:00:01
+${ip_address} 192.168.3.5
*** Test Cases ***
-Create Two ONU Devices
- [Documentation] Create two onu devices to be tested for valid/invalid paths
- ${resp}= CORD Get /xosapi/v1/volt/voltservices
- ${jsondata}= To Json ${resp.content}
- ${voltservice}= Get From List ${jsondata['items']} 0
- ${voltservice_id}= Get From Dictionary ${voltservice} id
- ${resp}= CORD Get /xosapi/v1/att-workflow-driver/attworkflowdriverservices
- ${jsondata}= To Json ${resp.content}
- ${attworkflowservice}= Get From List ${jsondata['items']} 0
- ${attworkflowservice_id}= Get From Dictionary ${attworkflowservice} id
- Set Suite Variable ${attworkflowservice_id}
- ${resp}= CORD Post /xosapi/v1/rcord/rcordsubscribers {"onu_device": "${onu_serial_no}", "status": "pre-provisioned"}
- ${subscriber_id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${subscriber_id}
- ${resp}= CORD Post /xosapi/v1/volt/oltdevices {"volt_service_id": ${voltservice_id}, "name": "testoltdevice1", "device_type": "ponism", "host": "172.17.0.1", "port": 50060, "switch_port": "1", "dp_id": "${deviceId}", "outer_tpid": "0x8100", "uplink": "128"}
- ${oltdevice_id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${oltdevice_id}
- ${resp}= CORD Post /xosapi/v1/volt/ponports {"olt_device_id": ${oltdevice_id}, "port_no": "${ponportno}", "name": "testponport1"}
- ${ponport_id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${ponport_id}
- ${resp}= CORD Post /xosapi/v1/volt/onudevices {"serial_number": "${onu_serial_no}", "pon_port_id": ${ponport_id}, "vendor": "abcdefg"}
- ${onu_device1_id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${onu_device1_id}
- ${resp}= CORD Post /xosapi/v1/volt/uniports {"onu_device_id": "${onu_device1_id}", "port_no": ${uniportno}, "name": "testuniport"}
- ${uni_port_id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${uni_port_id}
- ${resp}= CORD Post /xosapi/v1/volt/onudevices {"serial_number": "${onu_invalid_sn}", "pon_port_id": ${ponport_id}, "vendor": "abcdefg"}
- ${onu_device2_id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${onu_device2_id}
- ${resp}= CORD Post /xosapi/v1/att-workflow-driver/attworkflowdriverwhitelistentries {"serial_number": "${onu_serial_no}", "device_id": "${deviceId}", "pon_port_id": ${ponportno}, "owner_id": ${attworkflowservice_id}}
- ${whitelist_entry_id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${whitelist_entry_id}
-
-Activate Non-Whitelisted ONU
+Activate Non-Whitelisted ONU onu.events {'status': 'activated', 'serial_number': '${onu_invalid_sn}','uni_port_id': 100, 'of_dpid': '${deviceId}'} onu_serial_number=${onu_invalid_sn} onu_state=DISABLED
[Documentation] Validate that activating an ONU not in whitelist sets onu device to DISABLED
- Send Kafka Event onu.events {'status': 'activated','serialNumber': '${onu_invalid_sn}','portNumber': '${uniportno}', 'deviceId': '${deviceId}'}
- Wait Until Keyword Succeeds 30s 5s Validate ONU Device Status ${onu_invalid_sn} DISABLED
-Activate Whitelisted ONU in Wrong Location
+Activate Whitelisted ONU in Wrong Location onu.events {'status': 'activated', 'serial_number': '${onu_serial_no}','uni_port_id': 52, 'of_dpid': 'wrongofdpid'} onu_serial_number=${onu_serial_no} onu_state=DISABLED
[Documentation] Validate that activating an ONU in the whitelist but in the wrong location DISABLES the onu device
- Send Kafka Event onu.events {'status': 'activated', 'serialNumber': '${onu_serial_no}','portNumber': '52', 'deviceId': 'wrongofdpid'}
- Wait Until Keyword Succeeds 30s 5s Validate ONU Device Status ${onu_serial_no} DISABLED
-Activate Whitelisted ONU
+Activate Whitelisted ONU onu.events {'status': 'activated', 'serial_number': '${onu_serial_no}','uni_port_id': ${uniportno}, 'of_dpid': '${deviceId}'} onu_serial_number=${onu_serial_no} onu_state=ENABLED
[Documentation] Validate that activating an ONU in the whitelist creates a attworkflow-driver-serviceinstance
- Send Kafka Event onu.events {'status': 'activated','serialNumber': '${onu_serial_no}','portNumber': '${uniportno}', 'deviceId': '${deviceId}'}
- ${att_wf_driver_si_id}= Wait Until Keyword Succeeds 30s 5s Get ATT Service Instance ID ${onu_serial_no} AWAITING
- Wait Until Keyword Succeeds 60s 5s Validate ONU Device Status ${onu_serial_no} ENABLED
-Send Auth Request
+Send Auth Request authentication.events {'authenticationState': 'APPROVED', 'deviceId': '${deviceId}','portNumber': ${uniportno}} onu_serial_number=${onu_serial_no} subscriber_state=enabled service_instance_state=APPROVED service_instance_count=1 service_instance_dhcp_state=AWAITING
[Documentation] Validate that sending an auth request to the onu will enable the subscriber and create a service chain
- Send Kafka Event authentication.events {'authenticationState': 'APPROVED', 'deviceId': '${deviceId}','portNumber': ${uniportno}}
- Wait Until Keyword Succeeds 30s 5s Validate Subscriber Status ${onu_serial_no} enabled
- Wait Until Keyword Succeeds 30s 5s Validate Subscriber Service Chain ${onu_serial_no} 1
- ${att_wf_driver_si_id}= Wait Until Keyword Succeeds 30s 5s Get ATT Service Instance ID ${onu_serial_no} APPROVED
-Send Denied Auth Request
+Send DHCP Request dhcp.events {'macAddress': '${mac_address}','ipAddress': '${ip_address}', 'deviceId': '${deviceId}', 'portNumber': ${uniportno}, 'messageType': 'DHCPACK'} onu_serial_number=${onu_serial_no} service_instance_state=APPROVED service_instance_dhcp_state=DHCPACK
+ [Documentation] Validate that sending an dhcp request to update the subscriber's mac+ip address
+
+Send Denied Auth Request authentication.events {'authenticationState': 'DENIED', 'deviceId': '${deviceId}','portNumber': ${uniportno}} onu_serial_number=${onu_serial_no} subscriber_state=auth-failed service_instance_state=DENIED service_instance_dhcp_state=DHCPACK
[Documentation] Validate that denied auth request to the onu will disable the subscriber and remove a service chain
- Send Kafka Event authentication.events {'authenticationState': 'DENIED', 'deviceId': '${deviceId}','portNumber': ${uniportno}}
- Wait Until Keyword Succeeds 120s 5s Validate Subscriber Status ${onu_serial_no} auth-failed
- Wait Until Keyword Succeeds 120s 5s Validate Subscriber Service Chain ${onu_serial_no}
- ${att_wf_driver_si_id}= Wait Until Keyword Succeeds 30s 5s Get ATT Service Instance ID ${onu_serial_no} DENIED
Create New Whitelist Entry
[Documentation] Validate that creating a new whitelist entry for the "invalid" onu device will enable the onu
- ${resp}= CORD Post /xosapi/v1/att-workflow-driver/attworkflowdriverwhitelistentries {"serial_number": "${onu_invalid_sn}", "device_id": "${deviceId}", "pon_port_id": ${ponportno}, "owner_id": ${attworkflowservice_id}}
+ [Template] None
+ ${resp}= CORD Post ${att_whitelist_api} {"serial_number": "${onu_invalid_sn}", "device_id": "${deviceId}", "pon_port_id": ${ponportno}, "owner_id": ${attworkflowservice_id}}
${whitelist_entry2_id}= Get Json Value ${resp.content} /id
Set Suite Variable ${whitelist_entry2_id}
Wait Until Keyword Succeeds 30s 5s Validate ONU Device Status ${onu_invalid_sn} ENABLED
Remove Whitelist Entry
[Documentation] Validate that removing a whitelist entry for an onu device will disable the subscriber and remove it's service chain
+ [Template] None
CORD Delete /xosapi/v1/att-workflow-driver/attworkflowdriverwhitelistentries ${whitelist_entry_id}
Wait Until Keyword Succeeds 120s 5s Validate Subscriber Status ${onu_serial_no} auth-failed
Wait Until Keyword Succeeds 120s 5s Validate Subscriber Service Chain ${onu_serial_no}
@@ -108,13 +73,59 @@
${auth} = Create List admin@opencord.org letmein
${HEADERS} Create Dictionary Content-Type=application/json
Create Session ${server_ip} http://${server_ip}:${server_port} auth=${AUTH} headers=${HEADERS}
+ Create OLT, ONU, Subscribers, and Whitelists
Teardown
[Documentation] Delete all models created
CORD Get /xosapi/v1/rcord/rcordsubscribers
- CORD Delete /xosapi/v1/rcord/rcordsubscribers ${subscriber_id}
- Wait Until Keyword Succeeds 60s 5s CORD Delete /xosapi/v1/volt/oltdevices ${oltdevice_id}
- CORD Delete /xosapi/v1/att-workflow-driver/attworkflowdriverwhitelistentries ${whitelist_entry2_id}
+ Clean Up Objects ${subscriber_api}
+ Clean Up Objects ${att_si_api}
+ Clean Up Objects ${subscriber_api}
+ Wait Until Keyword Succeeds 60s 1s Clean Up Objects ${olt_api}
+ Clean Up Objects ${att_whitelist_api}
+
+Create OLT, ONU, Subscribers, and Whitelists
+ [Documentation] Create two onu devices to be tested for valid/invalid paths
+ ${resp}= CORD Get ${volt_api}
+ ${jsondata}= To Json ${resp.content}
+ ${voltservice}= Get From List ${jsondata['items']} 0
+ ${voltservice_id}= Get From Dictionary ${voltservice} id
+ ${resp}= CORD Get ${att_wf_api}
+ ${jsondata}= To Json ${resp.content}
+ ${attworkflowservice}= Get From List ${jsondata['items']} 0
+ ${attworkflowservice_id}= Get From Dictionary ${attworkflowservice} id
+ Set Suite Variable ${attworkflowservice_id}
+ ${resp}= CORD Post ${subscriber_api} {"onu_device": "${onu_serial_no}", "status": "pre-provisioned"}
+ ${subscriber_id}= Get Json Value ${resp.content} /id
+ Set Suite Variable ${subscriber_id}
+ ${resp}= CORD Post ${olt_api} {"volt_service_id": ${voltservice_id}, "name": "testoltdevice1", "device_type": "ponism", "host": "172.17.0.1", "port": 50060, "switch_port": "1", "dp_id": "${deviceId}", "outer_tpid": "0x8100", "uplink": "128"}
+ ${oltdevice_id}= Get Json Value ${resp.content} /id
+ Set Suite Variable ${oltdevice_id}
+ ${resp}= CORD Post ${pon_ports_api} {"olt_device_id": ${oltdevice_id}, "port_no": "${ponportno}", "name": "testponport1"}
+ ${ponport_id}= Get Json Value ${resp.content} /id
+ Set Suite Variable ${ponport_id}
+ ${resp}= CORD Post ${onu_device_api} {"serial_number": "${onu_serial_no}", "pon_port_id": ${ponport_id}, "vendor": "abcdefg"}
+ ${onu_device1_id}= Get Json Value ${resp.content} /id
+ Set Suite Variable ${onu_device1_id}
+ ${resp}= CORD Post ${uni_ports_api} {"onu_device_id": "${onu_device1_id}", "port_no": ${uniportno}, "name": "testuniport"}
+ ${uni_port_id}= Get Json Value ${resp.content} /id
+ Set Suite Variable ${uni_port_id}
+ ${resp}= CORD Post ${onu_device_api} {"serial_number": "${onu_invalid_sn}", "pon_port_id": ${ponport_id}, "vendor": "abcdefg"}
+ ${onu_device2_id}= Get Json Value ${resp.content} /id
+ Set Suite Variable ${onu_device2_id}
+ ${resp}= CORD Post ${att_whitelist_api} {"serial_number": "${onu_serial_no}", "device_id": "${deviceId}", "pon_port_id": ${ponportno}, "owner_id": ${attworkflowservice_id}}
+ ${whitelist_entry_id}= Get Json Value ${resp.content} /id
+ Set Suite Variable ${whitelist_entry_id}
+
+Send Event and Verify
+ [Arguments] ${topic} ${event} ${onu_serial_number}=${EMPTY} ${onu_state}=${EMPTY} ${subscriber_state}=${EMPTY} ${service_instance_state}=${EMPTY} ${service_instance_count}=${EMPTY} ${service_instance_dhcp_state}=${EMPTY}
+ Send Kafka Event ${topic} ${event}
+ Run Keyword If '${topic}' == 'onu.events' Wait Until Keyword Succeeds 30s 5s Validate ONU Device Status ${onu_serial_number} ${onu_state}
+ Run Keyword If '${topic}' == 'authentication.events' Wait Until Keyword Succeeds 30s 5s Validate Subscriber Status ${onu_serial_number} ${subscriber_state}
+ Run Keyword If '${topic}' == 'authentication.events' Wait Until Keyword Succeeds 30s 5s Validate Subscriber Service Chain ${onu_serial_number} ${service_instance_count}
+ Run Keyword If '${topic}' == 'authentication.events' Wait Until Keyword Succeeds 30s 5s Validate ATT Service Instance ${onu_serial_number} ${service_instance_state} ${service_instance_dhcp_state}
+ Run Keyword If '${topic}' == 'dhcp.events' Wait Until Keyword Succeeds 30s 5s Validate ATT Service Instance ${onu_serial_number} ${service_instance_state} ${service_instance_dhcp_state}
+ Run Keyword If '${topic}' == 'dhcp.events' Wait Until Keyword Succeeds 30s 5s Validate Subscriber Settings ${onu_serial_number}
Send Kafka Event
[Documentation] Send event
@@ -124,36 +135,9 @@
Send ${topic} ${event}
Flush
-CORD Get
- [Documentation] Make a GET call to XOS
- [Arguments] ${service}
- ${resp}= Get Request ${server_ip} ${service}
- Log ${resp.content}
- Should Be Equal As Strings ${resp.status_code} 200
- [Return] ${resp}
-
-CORD Post
- [Documentation] Make a POST call to XOS
- [Arguments] ${service} ${data}
- ${data}= Evaluate json.dumps(${data}) json
- ${resp}= Post Request ${server_ip} uri=${service} data=${data}
- Log ${resp.content}
- Should Be Equal As Strings ${resp.status_code} 200
- ${id}= Get Json Value ${resp.content} /id
- Set Suite Variable ${id}
- [Return] ${resp}
-
-CORD Delete
- [Documentation] Make a DELETE call to the CORD controller
- [Arguments] ${service} ${data_id}
- ${resp}= Delete Request ${SERVER_IP} uri=${service}/${data_id}
- Log ${resp.content}
- Should Be Equal As Strings ${resp.status_code} 200
- [Return] ${resp}
-
-Get ATT Service Instance ID
- [Documentation] Returns the id of the newly created onu's att workflow service instance
- [Arguments] ${serial_no} ${auth_state}
+Validate ATT Service Instance
+ [Documentation] Validates the states in the ATT-WF-SI per onu
+ [Arguments] ${serial_no} ${auth_state} ${dhcp_state}
${resp}= Get Request ${server_ip} ${att_si_api}
Log ${resp.content}
Should Be Equal As Strings ${resp.status_code} 200
@@ -167,9 +151,10 @@
\ ${id}= Get From Dictionary ${value} id
\ ${sn}= Get From Dictionary ${value} serial_number
\ ${as}= Get From Dictionary ${value} authentication_state
+ \ ${dp}= Get From Dictionary ${value} dhcp_state
\ Run Keyword If '${sn}' == '${serial_no}' Exit For Loop
+ Should Be Equal ${dp} ${dhcp_state}
Should Be Equal ${as} ${auth_state}
- [Return] ${id}
Validate Subscriber Status
[Arguments] ${serial_no} ${expected_status}
@@ -196,7 +181,8 @@
\ ${result} ${slinks}= Run Keyword And Ignore Error Get From List ${sl} 0
\ ${sn}= Get From Dictionary ${value} onu_device
\ Run Keyword If '${sn}' == '${serial_no}' Exit For Loop
- Run Keyword If '${expected_no_sc}' != '${EMPTY}' Should Be Equal As Integers ${slinks} ${expected_no_sc} ELSE Should Be Empty ${sl}
+ Run Keyword If '${expected_no_sc}' != '${EMPTY}' Should Not Be Equal As Strings ${result} FAIL
+ Run Keyword If '${expected_no_sc}' != '${EMPTY}' Should Not Be Empty ${result} ELSE Should Be Empty ${sl}
Validate ONU Device Status
[Arguments] ${serial_no} ${expected_status}
@@ -220,8 +206,6 @@
: FOR ${INDEX} IN RANGE 0 ${length}
\ ${value}= Get From List ${jsondata['items']} ${INDEX}
\ ${macAddress}= Get From Dictionary ${value} mac_address
- \ ${ipAddress}= Get From Dictionary ${value} ip_address
\ ${sn}= Get From Dictionary ${value} onu_device
\ Run Keyword If '${sn}' == '${serial_no}' Exit For Loop
- Should Be Equal ${macAddress} ${mac_address}
- Should Be Equal ${ipAddress} ${ip_address}
+ Should Be Equal ${macAddress} ${mac_address}
\ No newline at end of file
diff --git a/src/test/cord-api/Tests/WorkflowValidations/ISSU.robot b/src/test/cord-api/Tests/WorkflowValidations/ISSU.robot
new file mode 100644
index 0000000..abf61ed
--- /dev/null
+++ b/src/test/cord-api/Tests/WorkflowValidations/ISSU.robot
@@ -0,0 +1,281 @@
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation In-Service-Software-Upgrade test suite
+Suite Setup Setup Suite
+Suite Teardown Teardown Suite
+Test Setup Setup Test
+Test Teardown Teardown Test
+Library Collections
+Library String
+Library OperatingSystem
+Library XML
+Library RequestsLibrary
+Library ../../Framework/utils/utils.py
+Resource ../../Framework/utils/utils.robot
+Library ../../Framework/restApi.py
+Resource ../../Framework/Subscriber.robot
+Resource ../../Framework/ATTWorkFlowDriver.robot
+Resource ../../Framework/Kubernetes.robot
+Resource ../../Framework/ONU.robot
+Resource ../../Framework/OLT.robot
+Resource ../../Framework/DHCP.robot
+Variables ../../Properties/RestApiProperties.py
+
+*** Variables ***
+${POD_NAME} flex-pod1-olt
+${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
+${HELM_CHARTS_DIR} ~/helm-charts
+${WHITELIST_PATHFILE} ${CURDIR}/data/${POD_NAME}/ATTWhiteList.json
+${SUBSCRIBER_PATHFILE} ${CURDIR}/data/${POD_NAME}/ATTSubscriber.json
+${VOLT_DEVICE_PATHFILE} ${CURDIR}/data/${POD_NAME}/RealOLTDevice.json
+${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
+${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
+${VOLTHA_POD_NUM} 8
+
+*** Test Cases ***
+
+OLT Reboot
+ [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
+ ... Configure whitelist with correct ONU location
+ ... Validate successful authentication/DHCP/E2E ping
+ ... Reboots OLT
+ ... Validate that pings fail
+ ... Validate successful authentication/DHCP/E2E ping after OLT comes back up
+ [Tags] olt1
+ Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
+ Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
+ Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
+ # Reboot OLT
+ Login And Run Command On Remote System sudo reboot ${olt_ip} ${olt_user} ${olt_pass} prompt=#
+ Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${olt_ip}
+ Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
+ Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${olt_ip}
+ Wait Until Keyword Succeeds 120s 10s Openolt is Up ${olt_ip} ${olt_user} ${olt_pass}
+ # Validate successful pings since the OLT is Up
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Check Ping True ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
+
+Fabric Switch Reboot
+ [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
+ ... Configure whitelist with correct ONU location
+ ... Validate successful authentication/DHCP/E2E ping
+ ... Reboots Fabric Switch
+ ... Validate that pings fail
+ ... Validate successful authentication/DHCP/E2E ping after OLT comes back up
+ #[Setup] None
+ #[Teardown] None
+ [Tags] fabric1
+ Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
+ Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
+ Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
+ # Reboot Fabric Switch
+ Login And Run Command On Remote System sudo reboot ${fabric_ip} ${fabric_user} ${fabric_pass} prompt=#
+ Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${fabric_ip}
+ Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
+ #Validate DHCP and Ping False False ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
+ Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${fabric_ip}
+ # Validate successful pings since Fabric Switch is Up
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Check Ping True ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
+
+Subscriber(RG) Reboot
+ [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
+ ... Configure whitelist with correct ONU location
+ ... Validate successful authentication/DHCP/E2E ping
+ ... Reboots RG
+ ... Validate that pings fail
+ ... Validate successful authentication/DHCP/E2E ping after OLT comes back up
+ [Tags] RG-reboot
+ Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
+ Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
+ Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
+ # Reboot RG
+ Login And Run Command On Remote System sudo reboot ${src_ip} ${src_user} ${src_pass} prompt=$
+ Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${fabric_ip}
+ Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
+ Wait Until Keyword Succeeds 150s 10s Check Remote System Reachability True ${src_ip}
+ Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
+ # Perform Reauthentication/DHCP and Ping
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
+ Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
+ Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
+ Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
+ Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
+
+*** Keywords ***
+Setup Suite
+ ${auth} = Create List ${XOS_USER} ${XOS_PASSWD}
+ ${HEADERS} Create Dictionary Content-Type=application/json
+ Create Session ${server_ip} http://${server_ip}:${server_port} auth=${AUTH} headers=${HEADERS}
+ ${att_workflow_service_id}= Get Service Owner Id ${ATT_SERVICE}
+ ${volt_service_id}= Get Service Owner Id ${VOLT_SERVICE}
+ ${AttWhiteListList}= utils.jsonToList ${WHITELIST_PATHFILE} AttWhiteListInfo
+ Set Suite Variable ${AttWhiteListList}
+ ${AttWhiteListDict}= utils.listToDict ${AttWhiteListList} 0
+ ${AttWhiteListDict}= utils.setFieldValueInDict ${AttWhiteListDict} owner_id ${att_workflow_service_id}
+ ${onu_device}= Get From Dictionary ${AttWhiteListDict} serial_number
+ Set Global Variable ${onu_device}
+ ${onu_location}= Get From Dictionary ${AttWhiteListDict} pon_port_id
+ Set Global Variable ${onu_location}
+ ${SubscriberList}= utils.jsonToList ${SUBSCRIBER_PATHFILE} SubscriberInfo
+ Set Global Variable ${SubscriberList}
+ ${SubscriberDict}= utils.listToDict ${SubscriberList} 0
+ ${s_tag}= utils.getFieldValueFromDict ${SubscriberDict} s_tag
+ ${c_tag}= utils.getFieldValueFromDict ${SubscriberDict} c_tag
+ ${VoltDeviceList}= utils.jsonToList ${VOLT_DEVICE_PATHFILE} VOLTDeviceInfo
+ Set Global Variable ${VoltDeviceList}
+ Set Suite Variable ${s_tag}
+ Set Suite Variable ${c_tag}
+ Set Global Variable ${export_kubeconfig} export KUBECONFIG=${KUBERNETES_CONF}
+ # Read variables from yaml file
+ ${src_ip}= Evaluate ${hosts}.get("src").get("ip")
+ ${src_user}= Evaluate ${hosts}.get("src").get("user")
+ ${src_pass}= Evaluate ${hosts}.get("src").get("pass")
+ ${src_container_type}= Evaluate ${hosts}.get("src").get("container_type")
+ ${src_container_name}= Evaluate ${hosts}.get("src").get("container_name")
+ ${src_iface}= Evaluate ${hosts}.get("src").get("dp_iface_name")
+ ${dst_ip}= Evaluate ${hosts}.get("dst").get("ip")
+ ${dst_user} = Evaluate ${hosts}.get("dst").get("user")
+ ${dst_pass}= Evaluate ${hosts}.get("dst").get("pass")
+ ${dst_container_type}= Evaluate ${hosts}.get("dst").get("container_type")
+ ${dst_container_name}= Evaluate ${hosts}.get("dst").get("container_name")
+ ${dst_dp_iface}= Evaluate ${hosts}.get("dst").get("dp_iface_name")
+ ${dst_dp_ip}= Evaluate ${hosts}.get("dst").get("dp_iface_ip_qinq")
+ ${olt_ip}= Evaluate ${olts}[0].get("ip")
+ ${olt_user}= Evaluate ${olts}[0].get("user")
+ ${olt_pass}= Evaluate ${olts}[0].get("pass")
+ ${fabric_ip}= Evaluate ${fabric_switches}[0].get("ip")
+ ${fabric_user}= Evaluate ${fabric_switches}[0].get("user")
+ ${fabric_pass}= Evaluate ${fabric_switches}[0].get("pass")
+ ${k8s_node_ip}= Evaluate ${nodes}[0].get("ip")
+ ${k8s_node_user}= Evaluate ${nodes}[0].get("user")
+ ${k8s_node_pass}= Evaluate ${nodes}[0].get("pass")
+ Set Suite Variable ${src_ip}
+ Set Suite Variable ${src_user}
+ Set Suite Variable ${src_pass}
+ Set Suite Variable ${src_container_type}
+ Set Suite Variable ${src_container_name}
+ Set Suite Variable ${src_iface}
+ Set Suite Variable ${dst_ip}
+ Set Suite Variable ${dst_user}
+ Set Suite Variable ${dst_pass}
+ Set Suite Variable ${dst_container_type}
+ Set Suite Variable ${dst_container_name}
+ Set Suite Variable ${dst_dp_iface}
+ Set Suite Variable ${dst_dp_ip}
+ Set Suite Variable ${olt_ip}
+ Set Suite Variable ${olt_user}
+ Set Suite Variable ${olt_pass}
+ Set Suite Variable ${fabric_ip}
+ Set Suite Variable ${fabric_user}
+ Set Suite Variable ${fabric_pass}
+ Set Suite Variable ${k8s_node_ip}
+ Set Suite Variable ${k8s_node_user}
+ Set Suite Variable ${k8s_node_pass}
+ @{container_list}= Create List
+ Append To List ${container_list} att-workflow-att-workflow-driver
+ Append To List ${container_list} att-workflow-volt
+ Append To List ${container_list} onos
+ Append To List ${container_list} xos-core
+ Append To List ${container_list} vcore
+ Set Suite Variable ${container_list}
+ ${datetime}= Get Current Datetime On Kubernetes Node ${k8s_node_ip} ${k8s_node_user} ${k8s_node_pass}
+ Set Suite Variable ${datetime}
+
+Teardown Suite
+ [Documentation] Performs any additional cleanup required
+ Log Suite Teardown cleanup
+ Delete All Sessions
+
+Setup Test
+ [Documentation] Re-create Subscriber, whitelist, and olt-device models to test
+ Log Re-creating objects
+ ${datetime}= Get Current Datetime On Kubernetes Node ${k8s_node_ip} ${k8s_node_user} ${k8s_node_pass}
+ Create Whitelist
+ Create Subscriber
+ Create VOLT
+
+Teardown Test
+ [Documentation] Delete xos objects, kills processes and cleans up interfaces on src+dst servers
+ Clean Up Linux
+ Clean Up XOS
+ Log Kubernetes Containers Logs Since Time ${datetime} ${container_list}
+
+Clean Up Linux
+ [Documentation] Kill processes and clean up interfaces on src+dst servers
+ Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
+ Run Keyword And Ignore Error Kill Linux Process [d]hclient ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
+ Run Keyword If '${dst_ip}' != '${None}' Run Keyword And Ignore Error Kill Linux Process [d]hcpd ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
+ Delete IP Addresses from Interface on Remote Host ${src_iface} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
+ Run Keyword If '${dst_ip}' != '${None}' Delete Interface on Remote Host ${dst_dp_iface}.${s_tag} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
+
+Clean Up XOS
+ [Documentation] Clean up all XOS objects and reinstall voltha after OLT reboots
+ Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${VOLT_SUBSCRIBER}
+ Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${ATT_WHITELIST}
+ Sleep 20s
+ Wait Until Keyword Succeeds 30s 2s Validate Subscriber Count 0
+ Sleep 10s
+ Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${VOLT_DEVICE}
+ Sleep 20s
+ Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${ATT_SERVICEINSTANCES}
+ Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${olt_ip}
+ Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${olt_ip}
+ Wait Until Keyword Succeeds 120s 10s Openolt is Up ${olt_ip} ${olt_user} ${olt_pass}
+ Sleep 15s
+
+Create Whitelist
+ ${AttWhiteListDict}= utils.listToDict ${AttWhiteListList} 0
+ CORD Post ${ATT_WHITELIST} ${AttWhiteListDict}
+
+Remove Whitelist
+ ${whitelist_id}= Retrieve Whitelist Entry ${onu_device}
+ CORD Delete ${ATT_WHITELIST} ${whitelist_id}
+
+Update Whitelist with Wrong Location
+ ${whitelist_id}= Retrieve Whitelist Entry ${onu_device}
+ CORD Put ${ATT_WHITELIST} {"pon_port_id": 55 } ${whitelist_id}
+
+Update Whitelist with Correct Location
+ ${whitelist_id}= Retrieve Whitelist Entry ${onu_device}
+ CORD Put ${ATT_WHITELIST} {"pon_port_id": ${onu_location} } ${whitelist_id}
+
+Create Subscriber
+ ${SubscriberDict}= utils.listToDict ${SubscriberList} 0
+ Wait Until Keyword Succeeds 120s 15s CORD Post ${VOLT_SUBSCRIBER} ${SubscriberDict}
+
+Remove Subscriber
+ ${subscriber_id}= Retrieve Subscriber ${c_tag}
+ CORD Delete ${VOLT_SUBSCRIBER} ${subscriber_id}
+
+Create VOLT
+ ${VoltDeviceDict}= utils.listToDict ${VoltDeviceList} 0
+ CORD Post ${VOLT_DEVICE} ${VoltDeviceDict}