blob: fc7bc698a74123f8ce1ddd73091fe555375e84fc [file] [log] [blame]
Charles Chanfcfe8902022-02-02 17:06:27 -08001.. SPDX-FileCopyrightText: 2021 Open Networking Foundation <info@opennetworking.org>
2.. SPDX-License-Identifier: Apache-2.0
3
Charles Chancaebcf32021-09-20 22:17:52 -07004System Integration Test (TestON)
5================================
Jon Hallced4be72021-10-04 10:19:04 -07006
7Each component of SD-Fabric has it's own testing infrastructure for component level tests. However, we need a way to
8test all the components together. For SD-Fabric, we are using a framework called `TestON`_ to write these tests. We
9use TestON to manipulate the components via CLI sessions over ssh or rest APIs. More information on how TestON works
10can be found in the `TestON Guide`_. The dataplane for these tests are either an emulated network running in Mininet
11using software switches, or hardware pods located in our lab. We use a variety of traffic generation tools running
12on the hosts to create traffic in the fabric.
13
14The current :ref:`topology<Topology>` for the hardware pod used for integration test is a set of leaf pair switches,
15a dual homed management server, and 3 compute nodes. One compute node is dual homed, one is dual homed with a second
16link to the second leaf switch, and one single homed to the second leaf switch.
17
18.. image:: ../images/paired-leaves-pod.svg
19 :scale: 100
20
21We also run soak testing and failure/recovery test in a 2x2 hardware pod.
22
23.. image:: ../images/topology-2x2.png
24 :width: 300px
25
26Test Suites
27-----------
28
29 * Functionality Tests
30
31 * `Paired Leaves <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/SRStaging/SRpairedLeaves>`_
32
33 * Basic Connectivity - All hosts can ping each other and their fabric interfaces.
34
35 * Link Failure - Verify a traffic stream fails over when there is a link failure. We test both failure of both
36 the source and destination links for dual and single homed hosts.
37
38 * Switch Failure - Verify a traffic stream fails over when there is a switch failure. We test several device
39 failure modes for both the source and destination leaves for dual and single homed hosts.
40
41 * `Bridging <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/SRBridging>`_ - Verify bridging works between different host VLAN configurations.
42
43 * `Routing <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/SRRouting>`_ - Verify routing works with different route configurations.
44
45 * `INT <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/INT>`_ - Verify INT reports for different packet drop reasons.
46
47 * `UPF <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/UP4>`_ - Tests ONOS UP4 APIs, such as attachment and detachment of UEs as well as verify encapsulation of both
48 upstream and downstream traffic.
49
50 * QOS
51
52 * `QOS <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/QOS>`_ - Generate traffic flows with different QFIs and verify QOS using port statistics.
53
54 * `QOSNonMobile <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/QOSNonMobile>`_ - Generate traffic flows with non mobile Traffic Classification and verify QOS using port statistics.
55
56 * Failure/Recovery Tests
57
58 * Link Failure Tests - Setup a flow from between hosts connected to different leaves, disable a link between a
59 leaf and a spine used by the flow, verify and measure how fast the flow is rerouted to another link.
60
61 * `Access Leaf <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/SRStaging/SReNBLeafSpinePortstateFailure>`_ - Disable a link between the access leaf and a spine.
62
63 * `Upstream Leaf <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/SRStaging/SRupstreamLeafSpinePortstateFailure>`_ - Disable a link between the upstream lead and a spine.
64
65 * Switch Failure Tests - Setup a flow from between hosts connected to different leaves, disable a spine used by
66 the flow, verify and measure how fast the flow is rerouted to another spine.
67
68 * `ONL Reboot <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/SRStaging/SRONLReboot>`_ - Simulate a switch failure by restarting ONL on the switch.
69
70 * `Stratum Restart <https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON/tests/USECASE/SegmentRouting/SRStaging/SRstratumRestart>`_ - Simulate a switch failure by killing the stratum agent on the switch.
71
72
73Test Results
74------------
75
76The nightly test results can be found by looking at the `SD-Fabric Nightly Tests`_ view on the `Aether Jenkins`_
77website. The functionality test suites are run on the Paired Leaves Pod and the failure/recovery jobs are run on the
78Staging Pod.
79
80.. _SD-Fabric Nightly Tests: https://jenkins.aetherproject.org/view/SD-Fabric%20Nightly%20Tests/
81.. _Aether Jenkins: https://jenkins.aetherproject.org/login
82.. _TestON: https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON
83.. _TestON Guide: https://wiki.onosproject.org/display/ONOS/System+Testing+Guide