testing/pdp_testing.rst

..
   SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org>
   SPDX-License-Identifier: Apache-2.0

PDP Testing
==============


Test Framework
--------------

We use `TestVectors`_ to connect to Stratum hardware switches using gRPC and execute gNMI and P4Runtime tests.
For Aether, we convert existing ptf unit tests written for fabric-tna to TestVectors and execute them on Stratum
hardware switches in loopback mode using `TestVectors-Runner`_.

We use Jenkins to schedule and trigger our tests which run against a set of hardware switches.

Test Scenarios
-------------------

fabric-tna is a P4 program based on the Tofino Native Architecture(TNA). Currently 4 profiles are supported for
compiling the fabric-tna P4 program.

1. fabric
2. fabric-spgw
3. fabric-int
4. fabric-spgw-int

Based on the ptf unit tests for fabric-tna, we generate TestVectors for each profile to run on Stratum hardware
switches. The names of generated tests can be found in `Test List`_.

Prerequisites to Generate, Run Tests
--------------------------------------
1. Stratum running on a hardware switch with 4 ports running in loopback mode.
2. Create a port-map similar to `port-map`_. The three fields are ptf_port, p4_port and iface_name.
   ptf_port is the port id used in the test, p4_port is a valid port id from Stratum hardware.
   iface_name is ignored in TestVector generation, this can be any value.

How to Generate TestVectors
--------------------------------------
1. Checkout fabric-tna repo.

.. code-block:: shell

   $git clone https://github.com/stratum/fabric-tna.git
   $cd fabric-tna

2. Compile the P4 program for specific profile.

.. code-block:: shell

   $make <profile>

Supported profiles are: fabric, fabric-spgw, fabric-int, fabric-spgw-int.

3. Generate TestVectors.

.. code-block:: shell

   $cd ptf
   $run/tv/run <profile> PORTMAP=port_map.json GRPCADDR=<switch_ip>:<switch_port> CPUPORT=<cpu_port>

switch_ip and switch_port are IP and port where Stratum is running.
cpu_port is 192 for dual pipe switch and 320 for quad pipe switch.
Generated TestVectors are stored under 'fabric-tna/ptf/TestVectors'.

How to Run TestVectors
--------------------------------------
1. Checkout `TestVectors-Runner`_ repo.

.. code-block:: shell

   $git clone https://github.com/stratum/testvectors-runner -b support-fabric-tna
   $cd testvectors-runner

2. Build tv-runner docker image.

.. code-block:: shell

   $docker build -t tvrunner:fabric-tna-binary -f build/test/Dockerfile .

3. Push PipelineConfig.

.. code-block:: shell

   $IMAGE_NAME=tvrunner:fabric-tna-binary ./tvrunner.sh --target ${tv_dir}/target.pb.txt --portmap ${tv_dir}/portmap.pb.txt --tv-dir ${tv_dir} --dp-mode loopback --tv-name PipelineConfig

4. Run Setup.

.. code-block:: shell

   $IMAGE_NAME=tvrunner:fabric-tna-binary ./tvrunner.sh --dp-mode loopback --match-type in --target ${tv_dir}/target.pb.txt --portmap ${tv_dir}/portmap.pb.txt --tv-dir ${tv_dir}/${test_name}/setup

5. Run TestVector.

.. code-block:: shell

   $IMAGE_NAME=tvrunner:fabric-tna-binary ./tvrunner.sh --dp-mode loopback --match-type in --target ${tv_dir}/target.pb.txt --portmap ${tv_dir}/portmap.pb.txt --tv-dir ${tv_dir}/${test_name} --tv-name ${test_name}.* --result-dir ./results --result-file ${test_name}

6. Run Teardown.

.. code-block:: shell

   $IMAGE_NAME=tvrunner:fabric-tna-binary ./tvrunner.sh --dp-mode loopback --match-type in --target ${tv_dir}/target.pb.txt --portmap ${tv_dir}/portmap.pb.txt --tv-dir ${tv_dir}/${test_name}/teardown

tv_dir is the directory where TestVectors are stored. In this case, tv_dir is 'fabric-tna/ptf/TestVectors'.
tv_name is the name of the test case. It's also the directory name of the test under 'fabric-tna/ptf/TestVectors'.

Results for each test are generated under 'testvectors-runner/results' directory in csv format.

.. _TestVectors: https://github.com/stratum/testvectors
.. _TestVectors-Runner: https://github.com/stratum/testvectors-runner/tree/support-fabric-tna
.. _Test List: https://github.com/stratum/stratum-ci/blob/master/ptf_tv_resources/converted-tests.yaml
.. _port-map: https://github.com/stratum/stratum-ci/blob/master/ptf_configs/x86-64-stordis-bf2556x-1t-r0/port_map.json