VOL-1790 test framework and sanity test commit
Change-Id: I66b8f93927232c13c62f6c0459f1f7afacf100e6
diff --git a/.gitreview b/.gitreview
new file mode 100644
index 0000000..4e39788
--- /dev/null
+++ b/.gitreview
@@ -0,0 +1,5 @@
+[gerrit]
+host=gerrit.opencord.org
+port=29418
+project=voltha-system-tests.git
+defaultremote=origin
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..efc6c38
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,19 @@
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+LINT_ARGS ?= --verbose --configure LineTooLong:130 --ignore TooFewTestSteps --ignore TooFewKeywordSteps
+
+
+lint:
+ rflint $(LINT_ARGS) .
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..5ceb6f2
--- /dev/null
+++ b/README.md
@@ -0,0 +1,44 @@
+#System-Tests
+
+Automated test-suites to validate the stability/functionality of VOLTHA. Tests that reside in here should be written in Robot Framework and Python.
+
+Intended use includes:
+
+* Sanity testing
+* Regression testing
+* Acceptance testing
+* Functional testing
+* Scale Testing using BBSIM
+* Failure/Scenario testing
+
+##Prerequisites
+* Python Virtual-Env
+* `voltctl` - a command line tool to access VOLTHA. Reference - [voltctl](https://github.com/ciena/voltctl)
+* `kubectl` - a command line tool to access your Kubernetes Clusers. Reference - [kubectl](https://kubernetes.io/docs/reference/kubectl/kubectl/)
+ * `voltctl` and `kubectl` should be configured to your system under test prior to any test executions
+
+Directory Structures are as followed:
+```
+├── tests
+ └── sanity/ // basic tests that should always pass. Will be used as gating-patchsets
+ └── functional/ // feature/functionality tests that should be implemented as new features get developed
+└── libraries // shared test keywords (functions) across various test suites
+└── variables // shared variables across various test suites
+```
+
+
+##Getting Started
+1. Download `voltha-system-tests`
+ * `git clone https://gerrit.opencord.org/voltha-system-tests`
+
+2. Create test virtual-environment
+ * `cd voltha-system-tests/`
+ * `source setup_venv.sh`
+3. Running Test-Suites
+ * Navigate to desired test suite location
+ * `robot --exclude notready sanity.robot`
+
+This test execution will generate three report files (`output.xml`, `report.html`, `log.html`). View the `report.html` page to analyze the results.
+
+## WIP:
+* Containerizing test environment so these tests can be run independent of the system
\ No newline at end of file
diff --git a/VERSION b/VERSION
new file mode 100644
index 0000000..8acdd82
--- /dev/null
+++ b/VERSION
@@ -0,0 +1 @@
+0.0.1
diff --git a/libraries/utils.robot b/libraries/utils.robot
new file mode 100644
index 0000000..ccccf9a
--- /dev/null
+++ b/libraries/utils.robot
@@ -0,0 +1,55 @@
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# robot test functions
+
+*** Settings ***
+Documentation Library for various utilities
+Library SSHLibrary
+Library HttpLibrary.HTTP
+Library String
+Library DateTime
+Library Process
+Library Collections
+Library RequestsLibrary
+Library OperatingSystem
+
+*** Keywords ***
+Execute ONOS Command
+ [Arguments] ${cmd}
+ [Documentation] Establishes an ssh connection to the onos contoller and executes a command
+ ${conn_id}= SSHLibrary.Open Connection localhost port=8101 prompt=onos> timeout=300s
+ SSHLibrary.Login karaf karaf
+ ${output}= SSHLibrary.Execute Command ${cmd}
+ SSHLibrary.Close Connection
+ [Return] ${output}
+
+Validate Device
+ [Arguments] ${serial_number} ${admin_state} ${oper_status} ${connect_status}
+ [Documentation] Parses the output of "voltctl device list" and inspects device ${serial_number}
+ ... Arguments are matched for device states of: "admin_state", "oper_status", and "connect_status"
+ ${output}= Run voltctl device list -o json
+ ${jsondata}= To Json ${output}
+ Log ${jsondata}
+ ${length}= Get Length ${jsondata}
+ : FOR ${INDEX} IN RANGE 0 ${length}
+ \ ${value}= Get From List ${jsondata} ${INDEX}
+ \ ${astate}= Get From Dictionary ${value} adminstate
+ \ ${opstatus}= Get From Dictionary ${value} operstatus
+ \ ${cstatus}= Get From Dictionary ${value} connectstatus
+ \ ${sn}= Get From Dictionary ${value} serialnumber
+ \ Run Keyword If '${sn}' == '${serial_number}' Exit For Loop
+ Should Be Equal ${astate} ${admin_state} Device ${serial_number} admin_state != ENABLED values=False
+ Should Be Equal ${opstatus} ${oper_status} Device ${serial_number} oper_status != ACTIVE values=False
+ Should Be Equal ${cstatus} ${connect_status} Device ${serial_number} connect_status != REACHABLE values=False
\ No newline at end of file
diff --git a/setup_venv.sh b/setup_venv.sh
new file mode 100644
index 0000000..1209699
--- /dev/null
+++ b/setup_venv.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# sets up a python virtualenv for running voltha system tests
+
+WORKSPACE=${WORKSPACE:-.}
+VENVDIR="${WORKSPACE}/venv-voltha-tests"
+
+# create venv if it's not yet there
+if [ ! -x "${VENVDIR}/bin/activate" ]; then
+ echo "Setting up dev/test virtualenv in ${VENVDIR} for VOLTHA-System-Tests"
+ virtualenv -q "${VENVDIR}" --no-site-packages
+ echo "Virtualenv created."
+fi
+
+echo "Installing python requirements in virtualenv with pip"
+source "${VENVDIR}/bin/activate"
+pip3 install --upgrade pip
+pip3 install cryptography==2.4.2 robotframework robotframework-requests robotframework-sshlibrary \
+ pexpect robotframework-httplibrary robotframework-kafkalibrary pygments pyyaml \
+ robotframework-databaselibrary psycopg2==2.7.7 paramiko==2.4.2
+pip3 install requests tinydb
+
+echo "VOLTHA-System-Tests virtualenv created. Run 'source ${VENVDIR}/bin/activate'."
\ No newline at end of file
diff --git a/tests/sanity/sanity.robot b/tests/sanity/sanity.robot
new file mode 100644
index 0000000..7f6295f
--- /dev/null
+++ b/tests/sanity/sanity.robot
@@ -0,0 +1,106 @@
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+*** Settings ***
+Documentation Creates bbsim olt/onu and validates activataion
+... Assumes voltha-go, go-based onu/olt adapters, and bbsim are installed
+... voltctl and kubectl should be configured prior to running these tests
+Library OperatingSystem
+Resource ${CURDIR}/../../libraries/utils.robot
+Resource ${CURDIR}/../../variables/variables.robot
+Suite Setup Setup
+Suite Teardown Teardown
+
+*** Variables ***
+${server_ip} localhost
+${timeout} 90s
+${num_onus} 1
+
+*** Test Cases ***
+Activate Device BBSIM OLT/ONU
+ [Documentation] Validate deployment ->
+ ... create and enable bbsim device ->
+ ... re-validate deployment
+ [Tags] activate
+ #create/preprovision device
+ ${rc} ${device_id}= Run and Return Rc and Output voltctl device create -t openolt -H ${BBSIM_SERVICE}:${BBSIM_PORT}
+ Should Be Equal As Integers ${rc} 0
+ #enable device
+ ${rc} ${output}= Run and Return Rc and Output voltctl device enable ${device_id}
+ Should Be Equal As Integers ${rc} 0
+ #validate olt states
+ Wait Until Keyword Succeeds 60s 5s Validate Device ${BBSIM_OLT_SN} ENABLED ACTIVE REACHABLE
+ #validate onu states
+ Wait Until Keyword Succeeds 60s 5s Validate Device ${BBSIM_ONU_SN} ENABLED ACTIVE REACHABLE
+
+Validate OLT Connected to ONOS
+ [Documentation] Verifies the BBSIM-OLT device is activated in onos
+ [Tags] notready
+ Wait Until Keyword Succeeds ${timeout} 5s BBSIM OLT Device in ONOS
+
+Check EAPOL Flows in ONOS
+ [Documentation] Validates eapol flows for the onu are pushed from voltha
+ [Tags] notready
+ Wait Until Keyword Succeeds ${timeout} 5s Verify Eapol Flows Added
+
+Validate ONU Authenticated in ONOS
+ [Documentation] Validates onu is AUTHORIZED in ONOS as bbsim will attempt to authenticate
+ [Tags] notready
+ Wait Until Keyword Succeeds ${timeout} 5s Verify Number of AAA-Users ${number_of_onus}
+
+Provision ONU Subscriber in ONOS
+ [Documentation] Through the olt-app in ONOS, execute 'volt-add-subscriber-access' and validate IP Flows
+ [Tags] notready
+
+Validate DHCP Assignment in ONOS
+ [Documentation] After IP Flows are pushed to the device, BBSIM will start a dhclient for the ONU.
+ [Tags] notready
+
+*** Keywords ***
+Setup
+ [Documentation] Create HTTP Session with the ONOS Controller
+ ${onos_auth}= Create List karaf karaf
+ ${HEADERS} Create Dictionary Content-Type=application/json
+ Create Session ONOS http://${server_ip}:${ONOS_REST_PORT} auth=${ONOS_AUTH}
+
+Teardown
+ [Documentation] Delete all http sessions
+ Delete All Sessions
+
+BBSIM OLT Device in ONOS
+ [Documentation] Checks if bbsim olt has been connected to ONOS
+ ${resp}= Get Request ONOS onos/v1/devices
+ ${jsondata}= To Json ${resp.content}
+ Should Not Be Empty ${jsondata['devices']}
+ ${length}= Get Length ${jsondata['devices']}
+ @{serial_numbers}= Create List
+ : FOR ${INDEX} IN RANGE 0 ${length}
+ \ ${value}= Get From List ${jsondata['devices']} ${INDEX}
+ \ ${sn}= Get From Dictionary ${value} serial
+ \ ${dpid}= Get From Dictionary ${value} id
+ Should Be Equal As Strings ${dpid} ${BBSIM_DEVICE_ID}
+ Should Be Equal As Strings ${sn} ${BBSIM_OLT_SN}
+
+Verify Eapol Flows Added
+ [Arguments] ${expected_onus}
+ [Documentation] Matches for number of eapol flows based on number of onus
+ ${eapol_flows_added}= Execute ONOS Command flows -s -f ADDED | grep eapol | wc -l
+ Should Contain ${eapol_flows_added} ${expected_onus}
+
+Verify Number of AAA-Users
+ [Arguments] ${expected_onus}
+ [Documentation] Matches for number of aaa-users authorized based on number of onus
+ ${aaa_users}= Execute ONOS Command aaa-users | grep AUTHORIZED | wc -l
+ Should Contain ${aaa_users} ${expected_onus}
\ No newline at end of file
diff --git a/variables/variables.robot b/variables/variables.robot
new file mode 100644
index 0000000..927e5f6
--- /dev/null
+++ b/variables/variables.robot
@@ -0,0 +1,21 @@
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Variables ***
+${BBSIM_SERVICE} bbsim.voltha.svc
+${BBSIM_PORT} 50060
+${BBSIM_DEVICE_ID} of:0000626273696d76
+${BBSIM_OLT_SN} BBSIMOLT000
+${BBSIM_ONU_SN} BBSM00000001
+${ONOS_REST_PORT} 8181
\ No newline at end of file