blob: 75790c55df5711a8148e6ba92bb55466f3121a36 [file] [log] [blame]
Hardik Windlass1808a712022-03-10 11:23:30 +00001Voltha Test Automation
2======================
3
4This page shows how the Voltha tests are automated to run on `OpenCORD Jenkins <https://jenkins.opencord.org/>`_.
5
6Tests of different workflows, i.e. ATT, DT and TT, with various configurations based on either BBSim deployment or
7actual Physical POD run on Jenkins periodically.
8
9.. figure:: ../_static/Voltha_Test_Automation_Jenkins.png
10 :alt: Voltha OpenCord Jenkins
11 :width: 70%
12 :align: center
13
14 Voltha OpenCord Jenkins
15
16Similar set of jobs are kept under similar tabs. Like,
17all the master build jobs are placed under `VOLTHA-2.X-Tests <https://jenkins.opencord.org/view/VOLTHA-2.X-Tests/>`_ tab,
18jobs related to the voltha-2.8 LTS release are in VOLTHA-2.8 tab.
19
20Likewise, tests that collect measurements on Voltha performance at scale are placed under `voltha-scale-measurements <https://jenkins.opencord.org/view/voltha-scale-measurements/>`_,
21and the Voltha Soak testing jobs are kept under the tab named `voltha-soak <https://jenkins.opencord.org/view/voltha-soak/>`_.
22
23Jenkins also host `verification jobs <https://jenkins.opencord.org/view/VOLTHA%202.x%20verify/>`_ that trigger automatically whenever a patch is pushed or updated on Gerrit for the review.
24
25
26Job Templates
27^^^^^^^^^^^^^
28
29Templates for the jobs are defined in `ci-management <https://github.com/opencord/ci-management>`__ repo.
30
31
32BBSim Jobs
33^^^^^^^^^^
34
35OpenCORD Jenkins host a large number of BBSim based regression jobs that run periodically
36with different deployment configurations to verify Voltha control plane functionality
37for different operator workflows. Such as,
38
39- `periodic-voltha-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-test-bbsim/>`_ & `periodic-voltha-multiple-olts-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multiple-olts-test-bbsim/>`_ : tests Voltha functional scenarios, recovery and error scenarios, ONOS HA scenarios etc. for ATT
40- `periodic-voltha-multi-uni-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multi-uni-test-bbsim/>`_ & `periodic-voltha-multi-uni-multiple-olts-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multi-uni-multiple-olts-test-bbsim/>`_ : tests Voltha functionality with multiple UNIs enabled, and with single & multiple OLT (BBSim) deployments
Joey Armstrongc5427b22023-08-15 08:55:33 -040041- `periodic-voltha-dt-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-dt-test-bbsim-master>`_ : tests functional and failure scenarios for DT workflow
Joey Armstrong449ce7a2023-07-18 18:32:24 -040042- `periodic-voltha-dt-fttb-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-dt-fttb-test-bbsim-master/>`_ : tests functional scenarios for DT FTTB workflow
Hardik Windlass1808a712022-03-10 11:23:30 +000043- `periodic-voltha-unitag-subscriber-tt-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-unitag-subscriber-tt-test-bbsim/>`_ : tests Voltha functionality with UNI-Tag Subscriber provisioning enabled for TT workflow
44- `periodic-voltha-tt-maclearner-sanity-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-tt-maclearner-sanity-test-bbsim/>`_ : tests Voltha functionality with MAC Learner enabled for TT workflow
45- `periodic-software-upgrade-test-bbsim <https://jenkins.opencord.org/job/periodic-software-upgrade-test-bbsim/>`_ : tests ONOS Applications, Voltha Components and ONU Software Upgrades
46- `periodic-voltha-openonu-go-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-openonu-go-test-bbsim/>`_ & `periodic-voltha-multiple-olts-openonu-go-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multiple-olts-openonu-go-test-bbsim/>`_ : tests state for ONU Go adapter with different operator workflows, different constellations and tech profiles
47- `periodic-voltha-pm-data-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-pm-data-test-bbsim/>`_ & `periodic-voltha-multiple-olts-pm-data-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multiple-olts-pm-data-test-bbsim/>`_ : tests Open ONU Go adapter PM data
48- `periodic-voltha-etcd-test <https://jenkins.opencord.org/job/periodic-voltha-etcd-test/>`_ : tests Voltha etcd functionality
49- `periodic-voltha-test-DMI <https://jenkins.opencord.org/job/periodic-voltha-test-DMI/>`_ : tests device management interface functionality
Hardik Windlass9d8e6c82022-03-23 08:28:43 +000050- `periodic-voltha-memory-leak-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-memory-leak-test-bbsim/>`_ : tests and catches memory leak in Voltha components
Hardik Windlass1808a712022-03-10 11:23:30 +000051
52Similar to the above mentioned jobs which are based on **master**,
53Jenkins has regression jobs for other Voltha releases as well, like votlha-2.8 and voltha-2.9.
54
55The template for the jobs that run on BBSim based deployment is defined `here <https://github.com/opencord/ci-management/blob/master/jjb/voltha-e2e.yaml>`__.
56
57BBSim based deployment is also used to test Voltha at scale.
58The scale jobs run the tests that collect measurements on Voltha performance.
59
60Scale jobs do the deployment with various combinations
61like single or multiple voltha stacks with one or more BBSim based OLTs,
62multiple PONs and ONUs thus testing 4000+ subscribers for different workflows.
63More details about scale jobs can be found `here <https://jenkins.opencord.org/view/voltha-scale-measurements/>`__.
64
65
66Hardware (or Physical POD) Jobs
67^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
68
69OpenCord Jenkins host various regression jobs that runs on the ONF physical pods
70situated at different geographical locations.
71Hardware jobs test both control and dataplane functionality of Voltha.
72
73These hardware jobs are mostly defined in a combination of a build job and a test job.
74The template for the jobs that run on physical pod is defined `here <https://github.com/opencord/ci-management/blob/master/jjb/voltha-test/voltha-nightly-jobs.yaml>`__.
75
76Some of the configurations that are currently being tested on ONF's Physical Pods are:
77
78- Default Technology Profile with master or LTS release build for ATT
79- 1T4GEM profile with master or LTS release build for ATT
80- 1T4GEM-unencrypted profile with master or LTS release build for ATT
81- 1T8GEM Technology Profile with master or LTS release build for DT
82- 1T8GEM Technology Profile with master or LTS release build for DT with GPON based OLT setup
83- 1T8GEM Technology Profile with master or LTS release build for DT with multiple OLTs setup
84- Different service based Technology Profiles with master or LTS release build for TT
85- Different service based profiles with master or LTS release build for TT with multiple UNIs enabled
86
87
88Per-Patchset Verification Jobs
89^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
90
91The verification jobs gets triggered automatically whenever a patch is pushed or updated on Gerrit, e.g.
92
93- verify_{project}_licensed : This job verifies that there is valid license/copyright on files.
94- verify_{project}_tag-collision : This job checks for changes to version files,
95 and that they don't duplicate tags already in the git repo.
96- verify_{project}_unit-test : This job runs the Makefile based unit tests.
97- verify_{project}_sanity-test : This job verifies that basic Voltha functionality is not broken
98 for any of the operator workflows.
99
100Apart from system triggered jobs, there are some verification jobs which can be triggered by the user
101by providing apt comment in the Gerrit patch, e.g.
102
Hardik Windlass9d8e6c82022-03-23 08:28:43 +0000103- *hardware test functional tests* : This comment triggers the run on a dedicated physical pod
Hardik Windlass1808a712022-03-10 11:23:30 +0000104 and executes the entire functional test suite for ATT.
Hardik Windlass9d8e6c82022-03-23 08:28:43 +0000105- *hardware test functional tests DT* : This comment triggers the functional suite tests for DT.
106- *voltha test openonu singleolt* : Triggers patchset-voltha-openonu-go-test-bbsim job.
107- *voltha 2.9 test openonu singleolt* : Triggers patchset-voltha-2.9-openonu-go-test-bbsim job.
108- *voltha 2.8 test openonu singleolt* : Triggers patchset-voltha-2.8-openonu-go-test-bbsim job.
109- *voltha test openonu multiolt* : Triggers patchset-voltha-multiple-olts-openonu-go-test-bbsim job.
110- *voltha 2.9 test openonu multiolt* : Triggers patchset-voltha-2.9-multiple-olts-openonu-go-test-bbsim job.
111- *voltha 2.8 test openonu multiolt* : Triggers patchset-voltha-2.8-multiple-olts-openonu-go-test-bbsim job.
112- *voltha test pm data singleolt* : Triggers patchset-voltha-pm-data-test-bbsim job.
113- *voltha test pm data multiolt* : Triggers patchset-voltha-multiple-olts-pm-data-test-bbsim job.
Hardik Windlass1808a712022-03-10 11:23:30 +0000114
115The templates for the above mentioned and other similar jobs can be found in the
116`ci-management <https://github.com/opencord/ci-management>`__ repo.