Save per-test container logs on failure
Change-Id: I289b60e535a9c0c849785d08343c77601dca9948
diff --git a/libraries/utils.robot b/libraries/utils.robot
index 017b308..6aab95a 100644
--- a/libraries/utils.robot
+++ b/libraries/utils.robot
@@ -313,6 +313,22 @@
Run Process kubectl run announcer -ti --rm --restart Never --image ubuntu
... bash -- -c echo; sleep 1; echo ${message}; sleep 1; date --rfc-3339\=n ; sleep 1; echo; sleep 1
+Start Logging
+ [Arguments] ${label}
+ [Documentation] Start logging for test ${label}
+ ${kail_process}= Run Keyword If "${container_log_dir}" != "${None}" Start Process kail -n default
+ ... -n voltha cwd=${container_log_dir} stdout=${label}-combined.log
+ Set Test Variable ${kail_process}
+
+Stop Logging
+ [Arguments] ${label}
+ [Documentation] End logging for test; remove logfile if test passed
+ Run sync
+ Run Keyword If ${kail_process} Terminate Process ${kail_process}
+ ${test_logfile}= Run Keyword If "${container_log_dir}" != "${None}"
+ ... Join Path ${container_log_dir} ${label}-combined.log
+ Run Keyword If Test Passed Run Keyword If "${test_logfile}" != "${None}" Remove File ${test_logfile}
+
Clean Up Linux
[Documentation] Kill processes and clean up interfaces on src+dst servers
FOR ${I} IN RANGE 0 ${num_onus}
diff --git a/tests/functional/Voltha_ErrorScenarios.robot b/tests/functional/Voltha_ErrorScenarios.robot
index b7cdc44..977087c 100644
--- a/tests/functional/Voltha_ErrorScenarios.robot
+++ b/tests/functional/Voltha_ErrorScenarios.robot
@@ -52,12 +52,17 @@
${teardown_device} False
${scripts} ../../scripts
+# Per-test logging on failure is turned off by default; set this variable to enable
+${container_log_dir} ${None}
+
*** Test Cases ***
Adding the same OLT before and after enabling the device
[Documentation] Create OLT, Create the same OLT again and Check for the Error message
[Tags] VOL-2405 VOL-2406 AddSameOLT functional
- [Setup] Announce Message START TEST AddSameOLT
+ [Setup] Run Keywords Announce Message START TEST AddSameOLT
+ ... AND Start Logging AddSameOLT
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging AddSameOLT
... AND Announce Message END TEST AddSameOLT
Run Keyword If ${has_dataplane} Delete Device and Verify
${olt_device_id}= Create Device ${olt_ip} ${OLT_PORT}
@@ -83,8 +88,10 @@
[Documentation] Disable a device id which is not listed in the voltctl device list
... command and ensure that error message is shown.
[Tags] functional DisableInvalidDevice VOL-2412
- [Setup] Announce Message START TEST DisableInvalidDevice
+ [Setup] Run Keywords Announce Message START TEST DisableInvalidDevice
+ ... AND Start Logging DisableInvalidDevice
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DisableInvalidDevice
... AND Announce Message END TEST DisableInvalidDevice
${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device list -o json
Should Be Equal As Integers ${rc} 0
@@ -111,8 +118,10 @@
... Assuming devices are already created, up and running fine; test1 or sanity was
... executed where all the ONUs are authenticated/DHCP/pingable
[Tags] VOL-2411 DeleteBeforeDisableCheck notready
- [Setup] Announce Message START TEST DeleteBeforeDisableCheck
+ [Setup] Run Keywords Announce Message START TEST DeleteBeforeDisableCheck
+ ... AND Start Logging DeleteBeforeDisableCheck
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DeleteBeforeDisableCheck
... AND Announce Message END TEST DeleteBeforeDisableCheck
#validate olt states
Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE REACHABLE
@@ -144,8 +153,10 @@
[Documentation] Create OLT, disable same OLT, check error message and validates ONU
[Tags] VOL-2414 DisablePreprovisionedOLTCheck notready
[Setup] Run Keywords Announce Message START TEST DisablePreprovisionedOLTCheck
+ ... AND Start Logging DisablePreprovisionedOLTCheck
... AND Delete Device and Verify
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DisablePreprovisionedOLTCheck
... AND Announce Message END TEST DisablePreprovisionedOLTCheck
Run Keyword If ${has_dataplane} Sleep 180s
#create/preprovision device
@@ -179,8 +190,10 @@
... since it is allowed only through OLT device deletion.
[Tags] VOL-2418 DisableDelete_LogicalDevice notready
[Setup] Run Keywords Announce Message START TEST DisableDelete_LogicalDevice
+ ... AND Start Logging DisableDelete_LogicalDevice
... AND Delete Device and Verify
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DisableDelete_LogicalDevice
... AND Announce Message END TEST DisableDelete_LogicalDevice
Run Keyword If ${has_dataplane} Sleep 180s
#create/preprovision OLT device
@@ -215,8 +228,10 @@
[Documentation] Deletes all devices, checks logical device, creates devices again and checks
... logical device, flows, ports
[Tags] VOL-2416 VOL-2417 LogicalDeviceCheck notready
- [Setup] Announce Message START TEST LogicalDeviceCheck
+ [Setup] Run Keywords Announce Message START TEST LogicalDeviceCheck
+ ... AND Start Logging LogicalDeviceCheck
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging LogicalDeviceCheck
... AND Announce Message END TEST LogicalDeviceCheck
Delete Device and Verify
${logical_id}= Get Logical Device ID From SN ${olt_serial_number}
diff --git a/tests/functional/Voltha_FailureScenarios.robot b/tests/functional/Voltha_FailureScenarios.robot
index df4bde8..9250835 100644
--- a/tests/functional/Voltha_FailureScenarios.robot
+++ b/tests/functional/Voltha_FailureScenarios.robot
@@ -53,14 +53,19 @@
${teardown_device} False
${scripts} ../../scripts
+# Per-test logging on failure is turned off by default; set this variable to enable
+${container_log_dir} ${None}
+
*** Test Cases ***
Verify restart ofagent container after VOLTHA is operational
[Documentation] Restart ofagent container after VOLTHA is operational.
... Please note this test case should be run before the restart of other containers.
... Prerequisite : ONUs are authenticated and pingable.
[Tags] functional VOL-2409 ofagentRestart notready
- [Setup] Announce Message START TEST ofagentRestart
+ [Setup] Run Keywords Announce Message START TEST ofagentRestart
+ ... AND Start Logging ofagentRestart
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging ofagentRestart
... AND Announce Message END TEST ofagentRestart
# Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
Run Keyword If ${has_dataplane} Clean Up Linux
@@ -83,8 +88,10 @@
[Documentation] Restart openolt-adapter container after VOLTHA is operational.
... Prerequisite : ONUs are authenticated and pingable.
[Tags] functional VOL-1958 RestartPods
- [Setup] Announce Message START TEST RestartPods
+ [Setup] Run Keywords Announce Message START TEST RestartPods
+ ... AND Start Logging RestartPods
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RestartPods
... AND Announce Message END TEST RestartPods
${waitforRestart} Set Variable 120s
${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
@@ -106,8 +113,11 @@
... message count to make sure auth not started again and validates EAP status and ping.
... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
[Tags] functional ONUAdaptCrash notready
- [Setup] None
- [Teardown] None
+ [Setup] Run Keywords Announce Message START TEST ONUAdaptCrash
+ ... AND Start Logging ONUAdaptCrash
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging ONUAdaptCrash
+ ... AND Announce Message END TEST ONUAdaptCrash
@{before_list}= Create List
@{after_list}= Create List
FOR ${I} IN RANGE 0 ${num_onus}
@@ -152,8 +162,10 @@
[Documentation] Test the ONU funcaionality by physically turning on/off ONU.
... Prerequisite : Subscriber are authenticated/DHCP/pingable state
[Tags] functional VOL-2488 PowerSwitch notready
- [Setup] Announce Message START TEST PowerSwitch
+ [Setup] Run Keywords Announce Message START TEST PowerSwitch
+ ... AND Start Logging PowerSwitch
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging PowerSwitch
... AND Announce Message END TEST PowerSwitch
Start Log Capture PowerSwitch ${container_log_dir}
Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
@@ -178,8 +190,10 @@
ONU Reboot
[Documentation] Reboot ONU and verify that ONU comes up properly
[Tags] VOL-1957 RebootONU notready
- [Setup] Announce Message START TEST RebootONU
+ [Setup] Run Keywords Announce Message START TEST RebootONU
+ ... AND Start Logging RebootONU
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RebootONU
... AND Announce Message END TEST RebootONU
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
diff --git a/tests/functional/Voltha_PODTests.robot b/tests/functional/Voltha_PODTests.robot
index 323e979..2b93cd0 100644
--- a/tests/functional/Voltha_PODTests.robot
+++ b/tests/functional/Voltha_PODTests.robot
@@ -52,14 +52,19 @@
${teardown_device} False
${scripts} ../../scripts
+# Per-test logging on failure is turned off by default; set this variable to enable
+${container_log_dir} ${None}
+
*** Test Cases ***
Sanity E2E Test for OLT/ONU on POD
[Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
... Validate successful authentication/DHCP/E2E ping for the tech profile that is used
[Tags] sanity test1
[Setup] Run Keywords Announce Message START TEST SanityTest
+ ... AND Start Logging SanityTest
... AND Setup
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging SanityTest
... AND Announce Message END TEST SanityTest
Run Keyword If ${has_dataplane} Clean Up Linux
Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
@@ -70,8 +75,10 @@
... Perform disable on the OLT and validate that the pings do not succeed
... Perform enable on the OLT and validate that the pings are successful
[Tags] VOL-2410 DisableEnableOLT notready
- [Setup] Announce Message START TEST DisableEnableOLT
+ [Setup] Run Keywords Announce Message START TEST DisableEnableOLT
+ ... AND Start Logging DisableEnableOLT
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DisableEnableOLT
... AND Announce Message END TEST DisableEnableOLT
#Disable the OLT and verify the OLT/ONUs are disabled properly
${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
@@ -117,8 +124,10 @@
... Perform disable on the ONUs and validate that the pings do not succeed
... Perform enable on the ONUs and validate that the pings are successful
[Tags] functional DisableEnableONU released
- [Setup] Announce Message START TEST DisableEnableONU
+ [Setup] Run Keywords Announce Message START TEST DisableEnableONU
+ ... AND Start Logging DisableEnableONU
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DisableEnableONU
... AND Announce Message END TEST DisableEnableONU
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
@@ -149,8 +158,10 @@
... Delete a subscriber and validate that the pings do not succeed
... Re-add the subscriber and validate that the pings are successful
[Tags] functional SubAddDelete released
- [Setup] Announce Message START TEST SubAddDelete
+ [Setup] Run Keywords Announce Message START TEST SubAddDelete
+ ... AND Start Logging SubAddDelete
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging SubAddDelete
... AND Announce Message END TEST SubAddDelete
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
@@ -185,8 +196,10 @@
... teardown from previous test or uncomment 'Teardown None'.
... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
[Tags] functional RadiusRestart released
- [Setup] Announce Message START TEST RadiusRestart
+ [Setup] Run Keywords Announce Message START TEST RadiusRestart
+ ... AND Start Logging RadiusRestart
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RadiusRestart
... AND Announce Message END TEST RadiusRestart
Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
FOR ${I} IN RANGE 0 ${num_onus}
@@ -221,8 +234,10 @@
... when again added subscriber access, DHCP attempt, ping succeeds
... Assuming that test1 or sanity test was executed where all the ONUs are authenticated/DHCP/pingable
[Tags] functional SubsRemoveDHCP released
- [Setup] Announce Message START TEST SubsRemoveDHCP
+ [Setup] Run Keywords Announce Message START TEST SubsRemoveDHCP
+ ... AND Start Logging SubsRemoveDHCP
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging SubsRemoveDHCP
... AND Announce Message END TEST SubsRemoveDHCP
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
@@ -268,8 +283,10 @@
... validate that the pings are successful
... VOL-2284
[Tags] functional ATT_DisableEnableONU
- [Setup] Announce Message START TEST ATT_DisableEnableONU
+ [Setup] Run Keywords Announce Message START TEST ATT_DisableEnableONU
+ ... AND Start Logging ATT_DisableEnableONU
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging ATT_DisableEnableONU
... AND Announce Message END TEST ATT_DisableEnableONU
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
@@ -314,8 +331,10 @@
... Create/Enable the same OLT again
... Validate authentication/DHCP/E2E pings succeed for all the ONUs connected to the OLT
[Tags] functional DeleteOLT
- [Setup] Announce Message START TEST DeleteOLT
+ [Setup] Run Keywords Announce Message START TEST DeleteOLT
+ ... AND Start Logging DeleteOLT
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DeleteOLT
... AND Announce Message END TEST DeleteOLT
Run Keyword If ${has_dataplane} Clean Up Linux
Run Keyword If ${has_dataplane} Delete Device and Verify
@@ -330,8 +349,10 @@
... Devices will be removed during the execution of this TC
... so calling setup at the end to add the devices back to avoid the confusion.
[Tags] functional VOL-2354 DisableDeleteONUandOLT
- [Setup] Announce Message START TEST DisableDeleteONUandOLT
+ [Setup] Run Keywords Announce Message START TEST DisableDeleteONUandOLT
+ ... AND Start Logging DisableDeleteONUandOLT
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DisableDeleteONUandOLT
... AND Announce Message END TEST DisableDeleteONUandOLT
${olt_device_id}= Get Device ID From SN ${olt_serial_number}
FOR ${I} IN RANGE 0 ${num_onus}
@@ -380,8 +401,10 @@
... complete the DHCP sequence.
[Tags] bbsim rwcore-restart
[Setup] Run Keywords Announce Message START TEST RwCoreFailAndRestart
+ ... AND Start Logging RwCoreFailAndRestart
... AND Clear All Devices Then Create New Device
[Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RwCoreFailAndRestart
... AND Announce Message END TEST RwCoreFailAndRestart
${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
Set Global Variable ${of_id}
@@ -443,7 +466,12 @@
... and configures ONOS for access. The test succeeds if the device is able to
... complete the DHCP sequence.
[Tags] bbsim olt-adapter-restart
- [Setup] Clear All Devices Then Create New Device
+ [Setup] Run Keywords Announce Message START TEST OltAdapterRestart
+ ... AND Start Logging OltAdapterRestart
+ ... AND Clear All Devices Then Create New Device
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging OltAdapterRestart
+ ... AND Announce Message END TEST OltAdapterRestart
${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
Set Global Variable ${of_id}