Log START and END of tests in combined container logs
Change-Id: I4aaee399d0289a497d6400466cdc193d5eaf57e4
diff --git a/Makefile b/Makefile
index 45ec35f..a7d53cf 100644
--- a/Makefile
+++ b/Makefile
@@ -152,6 +152,12 @@
source ./$</bin/activate ; set -u ;\
python -m robot.tidy --inplace $(ROBOT_FILES);
+# Install the 'kail' tool if needed: https://github.com/boz/kail
+KAIL_PATH ?= /usr/local/bin
+$(KAIL_PATH)/kail:
+ bash <( curl -sfL https://raw.githubusercontent.com/boz/kail/master/godownloader.sh) -b /tmp
+ mv /tmp/kail $(KAIL_PATH)
+
## Variables for gendocs
TEST_SOURCE := $(wildcard tests/*/*.robot)
TEST_BASENAME := $(basename $(TEST_SOURCE))
diff --git a/libraries/utils.robot b/libraries/utils.robot
index 6b8b51d..24397b4 100644
--- a/libraries/utils.robot
+++ b/libraries/utils.robot
@@ -295,3 +295,10 @@
... Check Ping ${pingStatus} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
END
+
+Announce Message
+ [Arguments] ${message}
+ [Documentation] Announce a message that will be picked up by the log aggregator
+ Run Process kubectl delete pod announcer
+ Run Process kubectl run announcer -ti --restart Never --image ubuntu
+ ... bash -- -c echo; sleep 1; echo ${message}; sleep 1; date --rfc-3339\=n ; sleep 1; echo; sleep 1
diff --git a/tests/functional/Voltha_ErrorScenarios.robot b/tests/functional/Voltha_ErrorScenarios.robot
index 8e754ca..b7cdc44 100644
--- a/tests/functional/Voltha_ErrorScenarios.robot
+++ b/tests/functional/Voltha_ErrorScenarios.robot
@@ -56,8 +56,9 @@
Adding the same OLT before and after enabling the device
[Documentation] Create OLT, Create the same OLT again and Check for the Error message
[Tags] VOL-2405 VOL-2406 AddSameOLT functional
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST AddSameOLT
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST AddSameOLT
Run Keyword If ${has_dataplane} Delete Device and Verify
${olt_device_id}= Create Device ${olt_ip} ${OLT_PORT}
Set Suite Variable ${olt_device_id}
@@ -82,8 +83,9 @@
[Documentation] Disable a device id which is not listed in the voltctl device list
... command and ensure that error message is shown.
[Tags] functional DisableInvalidDevice VOL-2412
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST DisableInvalidDevice
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DisableInvalidDevice
${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device list -o json
Should Be Equal As Integers ${rc} 0
${jsondata}= To Json ${output}
@@ -109,8 +111,9 @@
... Assuming devices are already created, up and running fine; test1 or sanity was
... executed where all the ONUs are authenticated/DHCP/pingable
[Tags] VOL-2411 DeleteBeforeDisableCheck notready
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST DeleteBeforeDisableCheck
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DeleteBeforeDisableCheck
#validate olt states
Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE REACHABLE
... ${olt_serial_number}
@@ -140,8 +143,10 @@
Check disabling of pre-provisioned OLT before enabling
[Documentation] Create OLT, disable same OLT, check error message and validates ONU
[Tags] VOL-2414 DisablePreprovisionedOLTCheck notready
- [Setup] Delete Device and Verify
- [Teardown] None
+ [Setup] Run Keywords Announce Message START TEST DisablePreprovisionedOLTCheck
+ ... AND Delete Device and Verify
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DisablePreprovisionedOLTCheck
Run Keyword If ${has_dataplane} Sleep 180s
#create/preprovision device
${olt_device_id}= Create Device ${olt_ip} ${OLT_PORT}
@@ -173,8 +178,10 @@
[Documentation] Disable and delete the logical device directly is not possible
... since it is allowed only through OLT device deletion.
[Tags] VOL-2418 DisableDelete_LogicalDevice notready
- [Setup] Delete Device and Verify
- [Teardown] None
+ [Setup] Run Keywords Announce Message START TEST DisableDelete_LogicalDevice
+ ... AND Delete Device and Verify
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DisableDelete_LogicalDevice
Run Keyword If ${has_dataplane} Sleep 180s
#create/preprovision OLT device
${olt_device_id}= Create Device ${olt_ip} ${OLT_PORT}
@@ -208,8 +215,9 @@
[Documentation] Deletes all devices, checks logical device, creates devices again and checks
... logical device, flows, ports
[Tags] VOL-2416 VOL-2417 LogicalDeviceCheck notready
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST LogicalDeviceCheck
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST LogicalDeviceCheck
Delete Device and Verify
${logical_id}= Get Logical Device ID From SN ${olt_serial_number}
Should Be Empty ${logical_id}
diff --git a/tests/functional/Voltha_FailureScenarios.robot b/tests/functional/Voltha_FailureScenarios.robot
index 0a3e305..ca3e153 100644
--- a/tests/functional/Voltha_FailureScenarios.robot
+++ b/tests/functional/Voltha_FailureScenarios.robot
@@ -59,44 +59,45 @@
... Please note this test case should be run before the restart of other containers.
... Prerequisite : ONUs are authenticated and pingable.
[Tags] functional VOL-2409 ofagentRestart notready
- [Setup] NONE
- [Teardown] NONE
+ [Setup] Announce Message START TEST ofagentRestart
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST ofagentRestart
# Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
Run Keyword If ${has_dataplane} Clean Up Linux
Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
${waitforRestart} Set Variable 120s
- ${podStatusOutput}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE}
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Log ${podStatusOutput}
- ${countBforRestart}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Restart Pod ${NAMESPACE} ofagent
Sleep 60s
Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
... Running
Repeat Sanity Test
- ${podStatusOutput}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE}
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Log ${podStatusOutput}
- ${countAfterRestart}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
- Run Keyword and Ignore Error Collect Logs
Verify restart openolt-adapter container after VOLTHA is operational
[Documentation] Restart openolt-adapter container after VOLTHA is operational.
... Prerequisite : ONUs are authenticated and pingable.
[Tags] functional VOL-1958 RestartPods
- [Setup] NONE
- [Teardown] NONE
+ [Setup] Announce Message START TEST RestartPods
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST RestartPods
${waitforRestart} Set Variable 120s
- ${podStatusOutput}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE}
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Log ${podStatusOutput}
- ${countBforRestart}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
${podName} Set Variable adapter-open-olt
Restart Pod ${NAMESPACE} ${podName}
Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${podName} ${NAMESPACE}
... Running
Repeat Sanity Test
- ${podStatusOutput}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE}
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Log ${podStatusOutput}
- ${countAfterRestart}= Run ${KUBECTL_CONFIG};kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Log to console Pod ${podName} restarted and sanity checks passed successfully
@@ -104,8 +105,10 @@
[Documentation] Test the ONU funcaionality by physically turning on/off ONU.
... Prerequisite : Subscriber are authenticated/DHCP/pingable state
[Tags] functional VOL-2488 PowerSwitch notready
- [Setup] NONE
- [Teardown] NONE
+ [Setup] Announce Message START TEST PowerSwitch
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST PowerSwitch
+ Start Log Capture PowerSwitch ${container_log_dir}
Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
@@ -128,7 +131,9 @@
ONU Reboot
[Documentation] Reboot ONU and verify that ONU comes up properly
[Tags] VOL-1957 RebootONU notready
- [Setup] NONE
+ [Setup] Announce Message START TEST RebootONU
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST RebootONU
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
${dst}= Set Variable ${hosts.dst[${I}]}
@@ -139,4 +144,3 @@
#Check after reboot that ONUs are active, authenticated/DHCP/pingable
Perform Sanity Test
END
-
diff --git a/tests/functional/Voltha_PODTests.robot b/tests/functional/Voltha_PODTests.robot
index 09e1434..b0497b0 100644
--- a/tests/functional/Voltha_PODTests.robot
+++ b/tests/functional/Voltha_PODTests.robot
@@ -57,10 +57,12 @@
[Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
... Validate successful authentication/DHCP/E2E ping for the tech profile that is used
[Tags] sanity test1
- [Teardown] NONE
+ [Setup] Run Keywords Announce Message START TEST SanityTest
+ ... AND Setup
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST SanityTest
Run Keyword If ${has_dataplane} Clean Up Linux
Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
- Run Keyword and Ignore Error Collect Logs
Test Disable and Enable OLT
[Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
@@ -68,8 +70,9 @@
... Perform disable on the OLT and validate that the pings do not succeed
... Perform enable on the OLT and validate that the pings are successful
[Tags] VOL-2410 DisableEnableOLT notready
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST DisableEnableOLT
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DisableEnableOLT
#Disable the OLT and verify the OLT/ONUs are disabled properly
${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
Should Be Equal As Integers ${rc} 0
@@ -114,8 +117,9 @@
... Perform disable on the ONUs and validate that the pings do not succeed
... Perform enable on the ONUs and validate that the pings are successful
[Tags] functional DisableEnableONU released
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST DisableEnableONU
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DisableEnableONU
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
${dst}= Set Variable ${hosts.dst[${I}]}
@@ -138,7 +142,6 @@
Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
Run Keyword and Ignore Error Collect Logs
END
- Run Keyword and Ignore Error Collect Logs
Test Subscriber Delete and Add
[Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
@@ -146,8 +149,9 @@
... Delete a subscriber and validate that the pings do not succeed
... Re-add the subscriber and validate that the pings are successful
[Tags] functional SubAddDelete released
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST SubAddDelete
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST SubAddDelete
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
${dst}= Set Variable ${hosts.dst[${I}]}
@@ -173,7 +177,6 @@
Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
Run Keyword and Ignore Error Collect Logs
END
- Run Keyword and Ignore Error Collect Logs
Check OLT/ONU Authentication After Radius Pod Restart
[Documentation] After radius restart, triggers reassociation, checks status and
@@ -182,8 +185,9 @@
... teardown from previous test or uncomment 'Teardown None'.
... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
[Tags] functional RadiusRestart released
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST RadiusRestart
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST RadiusRestart
Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
@@ -211,15 +215,15 @@
Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
Run Keyword and Ignore Error Collect Logs
END
- Run Keyword and Ignore Error Collect Logs
Check DHCP attempt fails when subscriber is not added
[Documentation] Validates when removed subscriber access, DHCP attempt, ping fails and
... when again added subscriber access, DHCP attempt, ping succeeds
... Assuming that test1 or sanity test was executed where all the ONUs are authenticated/DHCP/pingable
[Tags] functional SubsRemoveDHCP released
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST SubsRemoveDHCP
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST SubsRemoveDHCP
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
${dst}= Set Variable ${hosts.dst[${I}]}
@@ -255,15 +259,15 @@
... ${dst['container_name']}
Run Keyword and Ignore Error Collect Logs
END
- Run Keyword and Ignore Error Collect Logs
Check ONU adapter crash not forcing authentication again
[Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
... message count to make sure auth not started again and validates EAP status and ping.
... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
[Tags] functional ONUAdaptCrash notready
- [Setup] None
- [Teardown] None
+ [Setup] Announce Message START TEST ONUAdaptCrash
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST ONUAdaptCrash
@{before_list}= Create List
@{after_list}= Create List
FOR ${I} IN RANGE 0 ${num_onus}
@@ -302,7 +306,7 @@
Lists Should Be Equal ${after_list} ${before_list}
Log ${after_list}
Log ${before_list}
- Run Keyword and Ignore Error Collect Logs
+
Test Disable and Enable ONU scenario for ATT workflow
[Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
@@ -312,8 +316,9 @@
... validate that the pings are successful
... VOL-2284
[Tags] functional ATT_DisableEnableONU
- [Setup] None
- #[Teardown] None
+ [Setup] Announce Message START TEST ATT_DisableEnableONU
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST ATT_DisableEnableONU
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
${dst}= Set Variable ${hosts.dst[${I}]}
@@ -350,7 +355,6 @@
... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Run Keyword and Ignore Error Collect Logs
END
- Run Keyword and Ignore Error Collect Logs
Delete OLT, ReAdd OLT and Perform Sanity Test
[Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
@@ -358,15 +362,15 @@
... Create/Enable the same OLT again
... Validate authentication/DHCP/E2E pings succeed for all the ONUs connected to the OLT
[Tags] functional DeleteOLT
- [Setup] None
- [Teardown] NONE
+ [Setup] Announce Message START TEST DeleteOLT
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DeleteOLT
Run Keyword If ${has_dataplane} Clean Up Linux
Run Keyword If ${has_dataplane} Delete Device and Verify
Run Keyword and Ignore Error Collect Logs
# Recreate the OLT
Run Keyword If ${has_dataplane} Setup
Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
- Run Keyword and Ignore Error Collect Logs
Test disable ONUs and OLT then delete ONUs and OLT
[Documentation] On deployed POD, disable the ONU, disable the OLT and then delete ONU and OLT.
@@ -374,8 +378,9 @@
... Devices will be removed during the execution of this TC
... so calling setup at the end to add the devices back to avoid the confusion.
[Tags] functional VOL-2354 DisableDeleteONUandOLT
- [Setup] NONE
- [Teardown] None
+ [Setup] Announce Message START TEST DisableDeleteONUandOLT
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST DisableDeleteONUandOLT
${olt_device_id}= Get Device ID From SN ${olt_serial_number}
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
@@ -415,7 +420,6 @@
Run Keyword If ${has_dataplane} sleep 180s
setup
-
Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
[Documentation] Deploys an device instance and waits for it to authenticate. After
... authentication is successful the rw-core deployment is scaled to 0 instances to
@@ -423,7 +427,10 @@
... and configures ONOS for access. The test succeeds if the device is able to
... complete the DHCP sequence.
[Tags] bbsim rwcore-restart
- [Setup] Clear All Devices Then Create New Device
+ [Setup] Run Keywords Announce Message START TEST RwCoreFailAndRestart
+ ... AND Clear All Devices Then Create New Device
+ [Teardown] Run Keywords Collect Logs
+ ... AND Announce Message END TEST RwCoreFailAndRestart
${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
Set Global Variable ${of_id}
FOR ${I} IN RANGE 0 ${num_onus}
@@ -544,13 +551,6 @@
# Execute normal test Setup Keyword
Setup
-Collect Logs
- [Documentation] Collect Logs from voltha and onos cli for various commands
- Run Keyword and Ignore Error Get Device List from Voltha
- Run Keyword and Ignore Error Get Device Output from Voltha ${olt_device_id}
- Run Keyword and Ignore Error Get Logical Device Output from Voltha ${logical_id}
- Run Keyword If ${external_libs} Get ONOS Status ${k8s_node_ip}
-
Clean Up Linux
[Documentation] Kill processes and clean up interfaces on src+dst servers
FOR ${I} IN RANGE 0 ${num_onus}
@@ -569,4 +569,3 @@
... ${dst['dp_iface_name']}.${src['s_tag']} ${dst['ip']} ${dst['user']} ${dst['pass']}
... ${dst['container_type']} ${dst['container_name']}
END
-