optimize keyword Wait for ONU Adapter Reconcile
Keyword 'Wait for Onu Adapter Reconcile' was removed and new keyword
'Reconcile Onu Adapter' was introduced instead. New keyword does all
needed actions for reconcilation:
- reboot onu-adapter
- check onu-adapter was really rebooted by ready timestamp comparison
- validate connection rw-core <-> onu-adapter established
- check accessability of onu-adapter
- validate oper-state is same as before reconcile
Change-Id: Ib9e97f2e3662ef34fd658e89270e35ba44f3ee01
diff --git a/libraries/k8s.robot b/libraries/k8s.robot
index d656dfd..f9ce0f3 100755
--- a/libraries/k8s.robot
+++ b/libraries/k8s.robot
@@ -458,6 +458,14 @@
... Pods Are Ready By Label ${namespace} app ${app_name}
END
+Get Pod Ready Timestamp by Label
+ [Arguments] ${namespace} ${key} ${value}
+ [Documentation] delivers timestamp of pod was ready
+ ${cmd}= Catenate kubectl -n ${namespace} get pods -l ${key}=${value} -o=json | jq -r
+ ... ".items[].status.containerStatuses[].state.running.startedAt"
+ ${output}= Run ${cmd}
+ [Return] ${output}
+
Check Expected Running Pods Number By Label
[Arguments] ${namespace} ${key} ${value} ${number}
[Documentation] Succeeds if the desired pod has expected number replicas
@@ -479,6 +487,13 @@
... kubectl get pods -n ${namespace} | grep ${name} | awk 'NR==1{print $4}'
[Return] ${count}
+Get Pod Age
+ [Arguments] ${namespace} ${name}
+ [Documentation] Returns the age for the given Pod
+ ${rc} ${age}= Run and Return Rc and Output
+ ... kubectl get pods -n ${namespace} | grep ${name} | awk 'NR==1{print $5}'
+ [Return] ${age}
+
Verify ONOS Pod Restart
[Arguments] ${restarted}=True
[Documentation] Verifies if any of the given ONOS instances restarted
diff --git a/libraries/onu_utilities.robot b/libraries/onu_utilities.robot
index 3db563a..48316d1 100755
--- a/libraries/onu_utilities.robot
+++ b/libraries/onu_utilities.robot
@@ -114,46 +114,48 @@
# teardown is used as 'return' for result of Validate ONU Devices With Duration (used for ONUNegativeStateTests)
[Teardown] Run Keyword If "${KEYWORD STATUS}"=="FAIL" Set Suite Variable ${StateTestAllONUs} False
-Wait for ONU Adapter Reconcile
- [Documentation] After the OpenONU adapter is restarted waits for all the ONUs to go through the reconciling process
- ... till they reach the expect OpenStatus
- [Arguments] ${oper_status}
+Reconcile Onu Adapter
+ [Documentation] Restarts the openonu adapter and waits for reconciling is finished and expected oper-state is reached
+ [Arguments] ${namespace} ${usekill2restart} ${oper_status}
+ # get time of restart of openonu adapter
+ ${restart_ts}= Get Current Date
+ # restart OpenONU adapter
+ Run Keyword If ${usekill2restart} Kill And Check Onu Adaptor ${namespace}
+ ... ELSE Restart And Check Onu Adaptor ${namespace}
+ #check ready timestamp of openonu adapter, should be younger than restart timestamp
+ ${openonu_ready_ts}= Get Pod Ready Timestamp by Label ${namespace} app adapter-open-onu
+ ${restart_duration}= Subtract Date From Date ${openonu_ready_ts} ${restart_ts}
+ Should Be True ${restart_duration}>0
# wait for the reconcile to complete
- # - we check that the first ONU state is set to reconciling. All of the ONUs will be set to reconciling
- # very quickly at that point
- # we need the Ignore Error to support he kafka case, in which the reconcile happens meanwhile we're waiting for
- # the adapter to restart
- Run Keyword And Ignore Error Wait Until Keyword Succeeds ${timeout}
- ... 100ms Check ONU OperStatus ${hosts.src[0]['onu']} RECONCILING
+ # - we check that communication to openonu-adapter is established again
+ # - we check that all ONUs leave reconcile state by validate a simple voltctl request will not responds with error
+ Wait Until Keyword Succeeds ${timeout} 1s Validate Last ONU Communication
+ Wait Until Keyword Succeeds ${timeout} 1s Operation Of Onu Adaptor Finished reconciling
# - then we wait that all ONU move to the next state
+ ${list_onus} Create List
+ Build ONU SN List ${list_onus}
+ Run Keyword And Ignore Error Wait Until Keyword Succeeds ${timeout}
+ ... 1s Check all ONU OperStatus ${list_onus} ${oper_status}
+
+Operation Of Onu Adaptor Finished
+ [Documentation] This keyword checks onu adaptor finished named operation with help of a simple voltctl request.
+ ... It is an implicit check. Possible operations are (device) 'deletion' or 'reconciling'.
+ ... As long as one of the named operations still running we've got an rc!=0 and we will find our
+ ... 'operation' as part of output of get request.
+ ... As get request Onu image list is used, any other get command could be used for this check.
+ [Arguments] ${operation}
${onu_list} Create List
FOR ${I} IN RANGE 0 ${num_all_onus}
- ${onu_sn}= Set Variable ${hosts.src[${I}]['onu']}
- # skip the check if we have already performed it for this ONU
- ${onu_id}= Get Index From List ${onu_list} ${onu_sn}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${onu_device_id}= Get Device ID From SN ${src['onu']}
+ ${onu_id}= Get Index From List ${onu_list} ${onu_device_id}
Continue For Loop If -1 != ${onu_id}
- Append To List ${onu_list} ${onu_sn}
- Wait Until Keyword Succeeds ${timeout} 100ms Check ONU OperStatus ${onu_sn} ${oper_status}
+ Append To List ${onu_list} ${onu_device_id}
+ ${rc} ${output}= Get Onu Image List ${onu_device_id}
+ Run Keyword If ${rc}!=0 Should Not Contain ${output} ${operation}
+ ... Onu adaptor operation ${operation} still in progress.
END
-Check ONU OperStatus
- [Documentation] Checks that all ONUs OperStatus is ACTIVE
- [Arguments] ${sn} ${oper_status}
- ${List_ONU_Serial} Create List
- Build ONU SN List ${List_ONU_Serial}
- ${cmd}= Catenate voltctl -c ${VOLTCTL_CONFIG} device list -m 8MB -f Type=brcm_openomci_onu
- ... --format "{{.SerialNumber}}\t{{.AdminState}}\t{{.OperStatus}}\t{{.ConnectStatus}}\t{{.Reason}}"
- ... | grep -v SERIALNUMBER | grep ${sn}
- ${rc} ${output}= Run and Return Rc and Output ${cmd}
- Should Be Equal As Integers ${rc} 0
- Log ${output}
-
- @{words}= Split String ${output} \t
- ${sn}= Set Variable ${words[0]}
- ${opstatus}= Set Variable ${words[2]}
- Log OperStatus is ${opstatus} for device ${sn}
- Should Be True '${opstatus}' == '${oper_status}' OperStatus ${oper_status} not matched on ONU ${sn}
-
Log Ports
[Documentation] This keyword logs all port data available in ONOS of first port per ONU
[Arguments] ${onlyenabled}=False
diff --git a/libraries/voltctl.robot b/libraries/voltctl.robot
index f0e873a..2814990 100755
--- a/libraries/voltctl.robot
+++ b/libraries/voltctl.robot
@@ -434,6 +434,39 @@
Should Be Equal '${mib_state}' '${onu_reason}'
... Device ${sn} mib_state incorrect (${mib_state}) values=False
+Check all ONU OperStatus
+ [Documentation] Checks that all ONUs OperStatus is the required one.
+ [Arguments] ${List_ONU_Serial} ${oper_status}
+ ${cmd}= Catenate voltctl -c ${VOLTCTL_CONFIG} device list -m 8MB -f Type=brcm_openomci_onu
+ ... --format "{{.SerialNumber}}\t{{.AdminState}}\t{{.OperStatus}}\t{{.ConnectStatus}}\t{{.Reason}}"
+ ... | grep -v SERIALNUMBER | grep ${oper_status}
+ ${rc} ${output}= Run and Return Rc and Output ${cmd}
+ Should Be Equal As Integers ${rc} 0
+ @{Results}= Split String ${output} \n
+ FOR ${Line} IN @{Results}
+ @{words}= Split String ${Line} \t
+ ${sn}= Set Variable ${words[0]}
+ Remove Values From List ${List_ONU_Serial} ${sn}
+ END
+ Should Be Empty ${List_ONU_Serial} For ONUs ${List_ONU_Serial} OperStatus ${oper_status} not matched!
+
+Validate Last ONU Communication
+ [Documentation] Validates last communication with ONU.
+ [Arguments] ${validation_duration}=5s
+ ${cmd}= Catenate voltctl -c ${VOLTCTL_CONFIG} adapter list -f Type=brcm_openomci_onu
+ ... --format '{{gosince .LastCommunication}}'
+ ${rc} ${output}= Run and Return Rc and Output ${cmd}
+ Should Be Equal As Integers ${rc} 0
+ ${lastcomm}= Convert Time ${output}
+ ${validduration}= Convert Time ${validation_duration}
+ Should Be True ${lastcomm}<=${validduration}
+
+Get Onu Image List
+ [Documentation] Delivers the ONU device image list
+ [Arguments] ${dev_id}
+ ${rc} ${output}= Run and Return Rc and Output
+ ... voltctl -c ${VOLTCTL_CONFIG} device onuimage list ${dev_id}
+ [return] ${rc} ${output}
Compare Lists
[Documentation]
@@ -700,8 +733,12 @@
[Arguments] ${serial_numbers} ${olt_serial_number}=${EMPTY} ${num_onus}=${num_all_onus}
[Documentation] Appends all ONU SNs for the given OLT to the ${serial_numbers} list
FOR ${INDEX} IN RANGE 0 ${num_onus}
- Run Keyword IF "${olt_serial_number}"=="${hosts.src[${INDEX}].olt}" or "${olt_serial_number}"=="${EMPTY}"
- ... Append To List ${serial_numbers} ${hosts.src[${INDEX}].onu}
+ ${onu_sn}= Set Variable ${hosts.src[${INDEX}]['onu']}
+ # skip if we have already append this ONU
+ ${onu_id}= Get Index From List ${serial_numbers} ${onu_sn}
+ Continue For Loop If -1 != ${onu_id}
+ Run Keyword IF "${olt_serial_number}"=="${hosts.src[${INDEX}].olt}" or "${olt_serial_number}"=="${EMPTY}"
+ ... Append To List ${serial_numbers} ${onu_sn}
END
Get SN From Device ID
diff --git a/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot b/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
index 678365d..11bfdb1 100755
--- a/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
+++ b/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
@@ -263,9 +263,7 @@
Enable Device ${olt_device_id}
END
Current State Test All Onus ${expected_onu_reason}
- Run Keyword If ${usekill2restart} Kill And Check Onu Adaptor ${NAMESPACE}
- ... ELSE Restart And Check Onu Adaptor ${NAMESPACE}
- Wait for ONU Adapter Reconcile ACTIVE
+ Reconcile Onu Adapter ${NAMESPACE} ${usekill2restart} ACTIVE
Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT
... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT
... ELSE Perform Sanity Test
@@ -299,9 +297,7 @@
Run Keyword If "${workflow}"=="DT" Current State Test All Onus omci-admin-lock
... ELSE IF "${workflow}"=="TT" Current State Test All Onus omci-admin-lock
... ELSE Current State Test All Onus omci-admin-lock alternativeonustate=${alternativeonustates}
- Run Keyword If ${usekill2restart} Kill And Check Onu Adaptor ${NAMESPACE}
- ... ELSE Restart And Check Onu Adaptor ${NAMESPACE}
- Wait for ONU Adapter Reconcile UNKNOWN
+ Reconcile Onu Adapter ${NAMESPACE} ${usekill2restart} UNKNOWN
Run Keyword If "${workflow}"=="DT" Current State Test All Onus omci-admin-lock
... ELSE IF "${workflow}"=="TT" Current State Test All Onus omci-admin-lock
... ELSE Current State Test All Onus omci-admin-lock alternativeonustate=${alternativeonustates}
@@ -334,9 +330,7 @@
Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT
... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT
... ELSE Perform Sanity Test
- Run Keyword If ${usekill2restart} Kill And Check Onu Adaptor ${NAMESPACE}
- ... ELSE Restart And Check Onu Adaptor ${NAMESPACE}
- Wait for ONU Adapter Reconcile ACTIVE
+ Reconcile Onu Adapter ${NAMESPACE} ${usekill2restart} ACTIVE
Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT ${suppressaddsubscriber}
... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT ${suppressaddsubscriber}
... ELSE Perform Sanity Test ${suppressaddsubscriber}