.. SPDX-FileCopyrightText: 2021 Open Networking Foundation <info@opennetworking.org>
.. SPDX-License-Identifier: Apache-2.0

SD-Fabric 1.1
=============

After one year of incubation under the ONF's member-only license,
we are excited to present the first `open-source release of SD-Fabric under the Apache 2.0 license
<https://opennetworking.org/news-and-events/press-releases/onfs-leading-private-5g-connected-edge-platform-aether-now-released-to-open-source/>`_

SD-Fabric 1.1 comes with numerous new features and improvements.
The main focus for this release is full integration with Aether,
ONF’s private-5G edge connectivity solution for enterprises.
SD-Fabric brings the many benefits of network programmability to Aether,
from a hardware-accelerated highly-available P4-UPF, to fabric-wide slicing and QoS,
and per-packet visibility via Inband-Network Telemetry (INT).

In this release, we provide two UPF options: the switch-based P4-UPF, offering the highest performance;
and the CPU-based BESS-UPF, tailored for deployment flexibility and horizontal scalability.
Both UPFs come with numerous bug fixes as well as integration with Aether 2.0 features,
such as support for 5G base stations, application filtering, QoS via multi-level rate-limiting,
and enhanced visibility with per-flow metrics and INT integration.

Moreover, we provide many operational improvements such as better support for
enterprise networks, simplified configuration, improved stability, and reduced resource usage.

We hope the transition to open source will catalyze the adoption of SD-Fabric and
attract more users and contributors from the community.
Please visit the :ref:`SD-Fabric documentation website <index>` to learn more about
all the features SD-Fabric offers and how to get started.

Feature Highlights
------------------

Open Source Release
^^^^^^^^^^^^^^^^^^^
All SD-Fabric components are now completely open source under the Apache 2.0 license.
We have published all source code, docker images, helm charts, documentation, and mailing lists.

Aether Integration
^^^^^^^^^^^^^^^^^^
This release provides the following integrations with Aether 2.0:

- **4G/5G slice based on P4-UPF**:
  which can now be instantiated using the Aether management portal or APIs.
  In this release, we support creating only one P4-UPF slice,
  creation of more P4-UPF slices will come in future releases.
  As before, more than one slice can be created when using BESS-UPF.
- **Initial INT integration**:
  this release adds di-metrics-exporter,
  a new service which fetches and aggregates real-time INT-based metrics and anomalies from Intel\ |reg| DeepInsight,
  to produce per-UE health status reports to be consumed by the Aether Analytics Engine (work-in-progress).
  SD-fabric supports monitoring of UE traffic when using both P4-UPF and BESS-UPF.

P4-UPF
^^^^^^
This release aims at bringing feature parity with BESS-UPF. To this end, P4-UPF now supports
the following new features:

- **Application filtering**:
  where the UPF can drop or forward traffic based on application endpoints
  (IPv4 prefix and port range) configured using PFCP SDF filters received from SD-Core.
  We support all ROC policies: ALLOW-ALL, DENY-ALL, ALLOW-PUBLIC.
- **Multi-level rate-limiting**:
  where traffic can be policed at the application level, UE level, or slice level.
  Per-application and per-UE rate limits can be configured by means of PFCP QERs received from SD-Core.

BESS-UPF
^^^^^^^^
- **Port ranges in app filtering rules**:
  BESS-UPF now supports installing PDRs with SDF filters
  matching on more than one transport-layer port.
- **Arbitrary IPv4 prefix lengths in app filtering rules**:
  including Aether's **allow-public** rule set (which blocks traffic to RFC 1918 private subnets).
- **Per-flow metrics (experimental)**:
  when enabled, the BESS pipeline collects metrics such as throughput, latency, jitter, and packet loss.
  Metrics are collected for each PFCP PDR rule,
  allowing performance monitoring for multiple application flows for the same UE.
  Metrics can be exported via the Prometheus endpoint in PFCP-Agent.

Slicing & QoS
^^^^^^^^^^^^^
We continue to make slicing and QoS more configurable to adapt to different use cases.

- **ONOS now reads slicing parameters via netcfg**:
  to support static configuration of arbitrary slices and traffic classes.
- **Updated config generation scripts**:
  to automatically generate both Stratum chassis_config and ONOS netcfg file using high-level parameters.

Operations
^^^^^^^^^^
Numerous improvements to make it easier to deploy and operate SD-Fabric.

- **Support for P4Runtime port translation**:
  which allows hiding ASIC-specific internal port numbers from user-facing configuration files.
  Now, when configuring switch interfaces in both the ONOS netcfg and Stratum chassis config,
  users can specify front panel port/channel numbers instead of SDK port numbers.
- **Remove the management server from the data plane**:
  user traffic is now processed by P4-UPF or BESS-UPF and then
  forwarded upstream (or to edge-applications) directly by the fabric switches.
- **Support base stations behind L3-routed network**:
  supporting real-world enterprise deployments.
- **Improve support for paired-leaves topology and distributed UPF**:
  by enabling phased-recovery by default to ensure
  traffic is not sent to a recovering switch until it is fully programmed, avoiding unnecessary packet drops.

Bug fixes and other improvements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Made primary controller election in ONOS persistent during device and cluster events,
  ensuring flow/group operations to always be successful.
- Improved the UP4's distributed UPF replication protocol,
  fixing a known issue that was causing missing or stale flows in some switches.
- Implemented a reconciliation mechanisms in the ONOS device manager to
  facilitate recovery from network partition scenarios
- Fixed wrong flow handling during network partition scenarios
- Enabled ZGC garbage collector in ONOS to reduce garbage collector pauses and their effects on the cluster.
- Upgraded Atomix to 3.1.12 and Karaf to 4.2.14

Testing
^^^^^^^
- Built a new Go-based integration test infrastructure for PFCP-Agent,
  including ``pfcpsim``, a CLI-based tool to emulate SMF interactions in tests.
  The new infrastructure allows testing PFCP-Agent interaction for both P4-UPF and BESS-UPF.
- Nightly SD-Fabric integration tests now include PFCP-Agent
- Added new nightly test cases for new P4-UPF features and persistent controller election

Development
^^^^^^^^^^^
- Created a v1model version of the fabric-tna P4 program.
  The new P4 program allows sharing the same pipeconf and PTF tests as the TNA version of the program,
  facilitating development and testing in bmv2-based emulated environments.

Upgrade Notes
-------------
If you are upgrading from SD-Fabric 1.0, there are several substantial changes that require additional attention.

**ONOS netcfg**

- We renamed all pipeconf names and replaced ``spgw`` with ``upf``
  (e.g., ``fabric-spgw-int`` is now named ``fabric-upf-int``)
- We bumped the supported Intel\ |reg| P4 Studio SDE from ``9_5_0`` to ``9_7_0``
- S1U/N3 address and UE pool configuration for the UP4 app is no longer necessary.
  Please use the PFCP-Agent ``upf.json`` or helm values to specify them.
- When using 5G base stations, make sure to set the ``pscEncapEnabled`` flag to true in the UP4's netcfg
- Because of P4Runtime translation,
  all port numbers in the ONOS netcfg should match the **singleton port id** used in the Stratum chassis config
  (instead of the ASIC-specific P4 port number).
- See :ref:`ONOS Network Config <onos_network_config>` for more details

**PFCP agent**

- We made UE pool configuration in the PFCP Agent mandatory.
  Make sure to configure ``ue_ip_pool``, under the ``cpiface`` group, in the ``upf.json`` file (or Helm values).
- Access IP configuration parameter is now mandatory.
  Make sure to configure ``access_ip`` in the ``upf.json`` file (or Helm values).
- ``p4rtciface.ue_ip_pool`` has been removed. Use ``cpiface.ue_ip_pool`` instead.
- Introduced ``slice_id`` field. It defaults to 0 if not specified.
- See :ref:`PFCP agent config <pfcp_agent_config>` for more details

**Stratum chassis config**

- Since the Stratum's singleton port ID can now be used in the ONOS netcfg for all port numbers
  (thanks to P4Runtime translation), we recommend using the following easy-to-understand convention:

    - For unchannelized ports: use the same front panel port number,
      e.g., for port “9/0” use singleton port ID “9”
    - For channelized ports: use the formula “port number x 100 + channel”,
      e.g., when configuring channels 9/0, 9/1, 9/2, and 9/3,
      the corresponding singleton port IDs should be 900, 901, 902, and 903
- See `Stratum port number <https://github.com/stratum/stratum/blob/main/stratum/hal/bin/barefoot/README.run.md#user-content-port-numbers>`_ for more details

**Docker image**

- All our docker images are now hosted on DockerHub.
  Remember to update image registry, repo and tag if you were pulling images from the Aether Registry previously
- We renamed the ``tost`` image to ``sdfabric-onos``

**Helm value**

- ``logging.karafVersion`` should be updated from ``4.2.9`` to ``4.2.14``


Known Issues and Limitations
----------------------------
- Some INT collectors might not support topologies with dual-homed hosts.
- In P4-UPF, configuration of slice rate limits is currently not exposed to Aether.
  We plan to add this in the next minor release (SD-Fabric v1.1.1).
- In BESS-UPF, when using application filtering with port ranges, we support only ranges up to 100 ports.

Component Versions
------------------

SD-Fabric ONOS
^^^^^^^^^^^^^^
- Image

  - `opennetworking/sdfabric-onos <https://hub.docker.com/r/opennetworking/sdfabric-onos>`_: ``1.1.0``

- Source code

  - `onos <https://github.com/opennetworkinglab/onos>`_: ``2.5.7-rc1``
  - `trellis-control <https://github.com/opennetworkinglab/trellis-control>`_: ``3.1.0``
  - `fabric-tna <https://github.com/stratum/fabric-tna>`_: ``1.0.0``
  - `up4 <https://github.com/omec-project/up4>`_: ``1.0.0``

Stratum
^^^^^^^
- Image: `stratumproject/stratum-bfrt <https://hub.docker.com/r/stratumproject/stratum-bfrt>`_: ``22.03.01-9.7.0``
- Repo: `stratum <https://github.com/stratum/stratum>`_: ``29f391580de8d5ee2d7a207a9a4261fda4e54a6f``

DBUF
^^^^
- Image: `opennetworking/dbuf <https://hub.docker.com/r/opennetworking/dbuf>`_: ``1.0.0``
- Repo: `dbuf <https://github.com/omec-project/dbuf>`_: ``v1.0.0``

PFCP Agent
^^^^^^^^^^
- Image: `omecproject/upf-epc-pfcpiface <https://hub.docker.com/r/omecproject/upf-epc-pfcpiface>`_

    - For P4-UPF: ``master-fcdbc95``
    - For BESS-UPF: ``master-9cb736e``

- Repo: `pfcpiface <https://github.com/omec-project/upf/tree/master/pfcpiface>`_

    - For P4-UPF: ``to-be-updated``
    - For BESS-UPF: ``to-be-updated``

BESS UPF
^^^^^^^^
- Image: `omecproject/upf-epc-bess <https://github.com/omec-project/upf>`_: ``master-9cb736e``
- Repo: `upf <https://github.com/omec-project/upf>`_: ``to-be-updated``

DeepInsight Utility
^^^^^^^^^^^^^^^^^^^

- Image: `registry.opennetworking.org/deepinsight/utility <https://registry.opennetworking.org/harbor/projects/38/repositories/utility>`_: ``4.2.0-ONF``
- Repo: `bf-di-scripts <https://github.com/opennetworkinglab/bf-di-scripts/blob/master/4/utility/>`_: ``fbc6999d4dc92d8f9909df1c9f2e7b66b0edbf2e``

.. note::
    DeepInsight is a 3rd party licensed software from Intel\ |reg|.
    It is integrated with SD-Fabric but the image is not distributed as part of the SD-Fabric open source release.

Helm Chart Versions
-------------------
- `sdfabric <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/refs/heads/master/sdfabric>`_: ``1.1.0``

  - `onos-classic chart <https://github.com/onosproject/onos-helm-charts/tree/master/onos-classic>`_: ``0.1.28``
  - `stratum chart <https://github.com/stratum/stratum-helm-charts/tree/master/stratum>`_: ``0.1.18``
  - `pfcp-agent chart <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/refs/heads/master/pfcp-agent>`_: ``0.0.12``
  - `dbuf chart <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/refs/heads/master/dbuf/>`_: ``0.0.4``
  - `int-host-reporter chart <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/refs/heads/master/int-host-reporter>`_: ``0.0.4``
  - `di-topology-watchdog chart <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/refs/heads/master/di-topology-watchdog>`_: ``0.0.5``
  - `di-metrics-exporter chart <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/refs/heads/master/di-metrics-exporter>`_: ``0.0.3``

- `bess-upf chart <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/refs/heads/master/bess-upf>`_: ``0.0.4``

.. |reg|    unicode:: U+000AE .. REGISTERED SIGN