First pass up to configuration
Change-Id: Ifd82c38aa18d5216c5772a64d9d77046def005fe
diff --git a/advanced/connectivity/external-connectivity.rst b/advanced/connectivity/external-connectivity.rst
index 618ddf3..5231df7 100644
--- a/advanced/connectivity/external-connectivity.rst
+++ b/advanced/connectivity/external-connectivity.rst
@@ -68,7 +68,7 @@
(10.0.1.2). Note that we use the same IP address on both the Quagga and
upstream interfaces.
-- ``vlan-untagged``: Configure the same VLAN ID on both interfaces. It doesn't
+- ``vlan-untagged``: Configure the same VLAN ID on both interfaces. It doesn't
matter exactly what the VLAN ID is, but it must be the same on both the
Quagga-facing and upstream-facing interfaces.
diff --git a/architecture.rst b/architecture.rst
index ff078a4..4fa4cb1 100644
--- a/architecture.rst
+++ b/architecture.rst
@@ -84,13 +84,13 @@
Policy Based Routing. But SDN control greatly simplifies the software running on each switch,
and control is moved into SDN applications running in the edge cloud.
-While these traditional switching/routing features are not particularly novel, SD-Fabric’s
+While these traditional switching/routing features are not particularly novel, SD-Fabric's
fundamental embrace of programmable silicon offers advantages that go far beyond traditional
fabrics.
Programmable Data Planes & P4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-SD-Fabric’s data plane is fully programmable. In marked contrast to traditional fabrics, features
+SD-Fabric's data plane is fully programmable. In marked contrast to traditional fabrics, features
are not prescribed by switch vendors. This is made possible by P4, a high-level programming
language used to define the switch packet processing pipeline, which can be compiled to run at
line-rate on programmable ASICs like Intel Tofino (see https://opennetworking.org/p4/). P4
@@ -115,7 +115,7 @@
With integrated UPF processing, SD-Fabric can implement a 4G/5G local breakout for edge
applications that is multi-terabit and low-latency, without taking away CPU processing power for
containers or VMs. In contrast to UPF solutions based on full or partial smartNIC offload,
-SDFabric’s embedded UPF does not require additional hardware other than the same leaf and spine
+SD-Fabric's embedded UPF does not require additional hardware other than the same leaf and spine
switches used to interconnect servers and base stations. At the same time, SD-Fabric can be
integrated with both CPU-based or smartNIC-based UPFs to improve scale while supporting
differentiated services on a hardware-based fast-path at line rate for mission critical 4G/5G
@@ -138,10 +138,10 @@
- For GTP-U tunnels, it produces reports about the inner flow, thus monitoring the
forwarding behavior and perceived QoS for individual UE flows.
-SD-Fabric’s INT implementation is compliant with the open source INT specification, and it has
-been validated to work with Intel’s DeepInsight performance monitoring solution, which acts as
+SD-Fabric's INT implementation is compliant with the open source INT specification, and it has
+been validated to work with Intel's DeepInsight performance monitoring solution, which acts as
the collector of INT reports generated by switches. Moreover, to avoid overloading the INT
-collector and to minimize the overhead of INT reports in the fabric, SD-Fabric’s data plane uses
+collector and to minimize the overhead of INT reports in the fabric, SD-Fabric's data plane uses
P4 to implement smart filters and triggers that drastically reduce the number of reports
generated, for example, by filtering out duplicates and by triggering report generation only in
case of meaningful anomalies (e.g., spikes in hop latency, path changes, drops, queue congestion,
@@ -150,7 +150,7 @@
Flexible ASIC Resource Allocation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The P4 program at the base of SD-Fabric’s software stack defines match-action tables for
+The P4 program at the base of SD-Fabric's software stack defines match-action tables for
common L2/L3 features such as bridging, IPv4/IPv6 routing, MPLS termination, and ACL, as well
as specialized features like UPF, with tables that store GTP-U tunnel information and more. In
contrast to fixed-function ASICs used in traditional fabrics, table sizes are not fixed. The use of
@@ -211,7 +211,7 @@
a critical role in closed loop control by offering a programmatic way to dynamically change the
packet forwarding behavior.
-At a high level, SD-Fabric’s APIs fall into four major categories: configuration, information,
+At a high level, SD-Fabric's APIs fall into four major categories: configuration, information,
control, and OAM.
- Configuration: APIs let users set up SD-Fabric features such as VLAN information for
@@ -251,7 +251,7 @@
and spine switches. Similarly, logs of each component in SD-Fabric are streamed to an EFK stack
(ElasticSearch, Fluentbit, Kibana) for log preservation, filtering and analysis. SD-Fabric offers a
single-pane-of-glass for logging and troubleshooting network state, which can further be
-integrated with operator’s backend systems
+integrated with operator's backend systems
.. image:: images/arch-logging.png
:width: 1000px
@@ -318,7 +318,7 @@
Open Network Operating System (ONOS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-SD-Fabric uses ONF’s Open Network Operating System (ONOS) as the SDN controller. ONOS is
+SD-Fabric uses ONF's Open Network Operating System (ONOS) as the SDN controller. ONOS is
designed as a distributed system, composed of multiple instances operating in a cluster, with all
instances actively operating on the network while being functionally identical. This unique
capability of ONOS simultaneously affords high availability and horizontal scaling of the control
@@ -339,7 +339,7 @@
API, which allows applications to program switching devices in a pipeline-agnostic
way. By using Flow-Objectives, applications can be written without worrying about low-level
pipeline details of various switching chips. The API is implemented by specific device drivers
-that are aware of the pipelines they serve and can thus convert the application’s API calls to
+that are aware of the pipelines they serve and can thus convert the application's API calls to
device-specific rules. In this way, the application can be written once, and adapted to pipelines
from different ASIC vendors.
diff --git a/configuration.rst b/configuration.rst
index 6119f31..1089b21 100644
--- a/configuration.rst
+++ b/configuration.rst
@@ -16,14 +16,14 @@
Data plane configurations, including port speed, breakout channel and queue configurations
are done in Stratum **chassis config**.
-We have implemented config pusher in both ONOS and Stratum helm chart,
+We have implemented config loader in both ONOS and Stratum helm chart,
fetching and applying configurations automatically once the system is ready.
.. toctree::
:maxdepth: 2
:glob:
- configuration/config-pusher
configuration/network
configuration/component
configuration/chassis
+ configuration/config-loader
diff --git a/configuration/chassis.rst b/configuration/chassis.rst
index 8d75245..9181a6a 100644
--- a/configuration/chassis.rst
+++ b/configuration/chassis.rst
@@ -3,37 +3,44 @@
Stratum Chassis Configuration
=============================
+Chassis config is an internal data structure used by Stratum. The term "chassis"
+refers to a switching box with one or more switching nodes (e.g., chips). The
+chassis config file is usually placed on disk and loaded when starting Stratum.
+This config includes all the not-so-frequent settings that are required before
+the switch can accept flow programming requests from the controller.
+
+In the following, we provide a **non-exhaustive overview of the configurations
+relevant to SD-Fabric for Tofino-based devices**. For a complete reference check
+out the `ChassisConfig Protobuf message format
+<https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L824-L833>`_
+and the `guide for running Stratum on Tofino-based switches
+<https://github.com/stratum/stratum/blob/main/stratum/hal/bin/barefoot/README.run.md>`_
+in the Stratum repository.
+
.. tip::
- Check out `examples <https://github.com/stratum/stratum/tree/main/stratum/hal/config>`_
- for every platform supported by Stratum.
+ Check out the `Stratum repository
+ <https://github.com/stratum/stratum/tree/main/stratum/hal/config>`_ for
+ examples of chassis config files for different switch platforms.
- See the `Deployment Guide <../deployment.rst>`_ to learn about how to deploy Stratum
+ See the :ref:`deployment_guide` to learn about how to deploy Stratum
with a custom chassis config.
-The Stratum chassis config is the internal data structure that encapsulates the so called
-"config" pushed to the entire chassis. The term "chassis" refers to the a switching box
-with one or more switching nodes managed by a management interface.
-
-The chassis config file will be placed on the disk of the device and loaded when starting the Stratum agent.
-The config includes all the not-so-frequent settings that are required before the switch
-can accept flow programming requests from the controller.
-
-A valid chassis config includes a ``chassis`` and a ``node`` field, and may includes one
-or more ``singleton_port`` field. See each section below for more detail.
-
.. note::
- In Stratum the external interface for pushing config is gNMI.
- The protobuf realization of the YANG models for the config is internally converted
- to a chassis config before it is consumed by the internal stack components.
+ In Stratum the external interface for pushing config is gNMI. The protobuf
+ realization of the YANG models for the config is internally converted to a
+ chassis config before it is consumed by the internal stack components. Not
+ all configurations are available through gNMI, for this reason we recommend
+ using the chassis config file.
- The `format of chassis config <https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L824-L833>`_
- is based on protobuf text format, check out
+ The file uses the protobuf text format, check out
`the protobuf language guide for more info <https://developers.google.com/protocol-buffers/docs/overview?hl=en>`_
Chassis
-------
+A valid chassis config includes a ``chassis``, a ``node`` field, and one
+or more ``singleton_port`` fields.
A ``chassis`` uniquely identifies a switch with a single management interface.
Each chassis may contain one or more slots (aka linecards),
@@ -41,13 +48,10 @@
A chassis contains the following fields:
-* ``platform``: The chassis platform. **Required**
+* ``platform``: The chassis platform, e.g., ``PLT_GENERIC_BAREFOOT_TOFINO`` for
+ Tofino-based devices (check out `here <https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L33-L47>`_ for other platforms). **Required**
* ``name``: An arbitrary name for the chassis. **Optional**
-* ``config_params``: Parameters configured for the entire chassis when config is pushed to the the switch. **Optional**
-
-.. tip::
-
- Check out the list of platforms `here <https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L33-L47>`_
+* ``config_params``: Parameters for the entire chassis, usually empty for Tofino-based devices. **Optional**
Node
----
@@ -57,68 +61,57 @@
A node contains the following fields:
-* ``id``: The unique ID of the switch node on the chassis as used by the P4Runtime controller. **Required**
+* ``id``: An arbitrary ID of the switch node on the chassis (corresponds to the P4Runtime device ID). **Required**
* ``name``: An arbitrary name for the switching node. **Optional**
-* ``slot``: The 1-base index of the slot (aka linecard) which this node belongs. **Required**
-* ``index``: The optional 1-base index of the node within the chassis. **Optional**
+* ``slot``: The 1-base index of the slot (aka linecard) to which this node belongs. **Required**
+* ``index``: An optional 1-based index of the node within the chassis. **Optional**
.. _singleton_port:
Singleton Port
--------------
-A ``singleton port`` in the chassis configuration uniquely identifies a single physical port on
-a single chassis.
+A ``singleton port`` uniquely identifies a switch port and it contains the following fields:
-A singleton port contains the following fields:
+* ``id``: An arbitrary ID, unique in the scope of a chassis. **Required**
+* ``name``: An optional user friendly name for the port (e.g., ``10/1`` to identify channel
+ 1 of front-panel port 10). **Required**
-* ``id``: The unique ID of the singleton port. **Required**
-* ``name``: An optional arbitrary name for the singleton port. **Required**
-
- The control plan (e.g., ONOS) can use this name to query port information as a gNMI path key.
+ ONOS obtains this name via gNMI and uses it internally to represent ports.
* ``slot``: The 1-base index of the slot (aka linecard) of the port. **Required**
* ``port``: The 1-base index of the singleton port on the slot. **Required**
* ``channel``: The 1-base channel index. Absence or zero means non-channelized. **Optional**
* ``speed_bps``: The speed of the port in bps. **Required**
* ``node``: The id of the corresponding node that the port belongs to. **Required**
-* ``config_params``: Parameters configured for this port. **Optional**
+* ``config_params``: **Optional**
- * ``admin_state``: The initial admin state for this port, port will be disabled by default.
+ * ``admin_state``: The initial admin state for this port, disabled by default.
* Choose from ``ADMIN_STATE_DISABLED`` or ``ADMIN_STATE_ENABLED``.
- * ``mtu``: The maximum transmission unit for this port.
* ``autoneg``: Whether auto-negotiation is enabled or not for this port.
* Choose from ``TRI_STATE_FALSE`` or ``TRI_STATE_TRUE``
* The initial configuration might be different if this field is empty, based on the
platform or the connector you are using.
- * ``fec_mode``: The
+ * ``fec_mode``: Forward Error Correction (FEC) mode
* Choose from ``FEC_MODE_ON``, ``FEC_MODE_OFF``, or ``FEC_MODE_AUTO``
- * ``loopback_mode``:
-
- * Choose from ``LOOPBACK_STATE_NONE``, ``LOOPBACK_STATE_MAC``, or ``LOOPBACK_STATE_PHY``
-
Example chassis config
----------------------
-In this example, we want to set up a **Tofino-based switch** with only one node/slot.
+In this example, we want to set up a **Tofino-based switch** with only one node, one slot,
+and two ports:
-And we want to set up two ports:
-
-* Port 1: 100G port without channelization, enabled by default, and disable auto auto-negotiation.
-* Port 2: break out to four 10G ports, uses different channels for each port and enable
- them by default with auto-negotiation.
+* Port 1: 100G port without channelization, enabled by default, without auto-negotiation.
+* Port 2: break out port with four 10G channels, enabled by default, with auto-negotiation.
.. image:: ../images/chassis-config-example.svg
:width: 500px
-Below is an example of a chassis configuration with a list of singleton port:
-
.. code-block::
description: "A chassis config example."
diff --git a/configuration/component.rst b/configuration/component.rst
index 88349b3..6c911c8 100644
--- a/configuration/component.rst
+++ b/configuration/component.rst
@@ -4,12 +4,13 @@
Different from network configurations,
component configurations are attributes and feature flags specific to ONOS components.
-For more details, please read ONOS Wiki `Component Configuration <https://wiki.onosproject.org/display/ONOS/Component+Configuration>`_
+For more details, please check the ONOS Wiki `Component Configuration <https://wiki.onosproject.org/display/ONOS/Component+Configuration>`_
Enable/Disable Phased Recovery
------------------------------
-Phased recovery is an experimental feature that ensure a
-recovering switch are programmed before pointing traffic from other switches towards it.
+Phased recovery is an experimental feature that aims at minimizing traffic
+disruption when recovering from a switch failure. When enabled, a recovering
+switch is programmed before pointing traffic from other switches towards it.
To check current status:
@@ -30,11 +31,11 @@
onos > cfg get org.onosproject.segmentrouting.phasedrecovery.impl.PhasedRecoveryManager phasedRecovery
name=phasedRecovery, type=boolean, value=true, defaultValue=false, description=Enabling phased recovery.
-Now the value is true and we have successfully enable phased recovery.
+Now the value is true and we have successfully enabled phased recovery.
To disable phased recovery, change the value to ``false``.
.. tip::
- Simply run ``cfg get`` in ONOS CLI to view all available component configurations supported by ONOS.
+ Simply run ``cfg get`` in the ONOS CLI to view all available component configurations supported by ONOS.
The default values are fine-tuned for SD-Fabric, and therefore we use them for most of the cases.
diff --git a/configuration/config-loader.rst b/configuration/config-loader.rst
new file mode 100644
index 0000000..51d8fbd
--- /dev/null
+++ b/configuration/config-loader.rst
@@ -0,0 +1,66 @@
+ONOS Configuration Loader
+=========================
+
+Config loader is a Kubernetes pod installed by the SD-Fabric Helm
+chart which implements a reconciliation loop to update the ONOS network
+configuration and component configuration.
+
+Config loader will periodically read the running configurations from ONOS and eventually
+push updates to ONOS if the running state isn't same as the desired state.
+
+.. image:: ../images/config-loader.svg
+
+.. note::
+ We recommend using the Helm values below to update the ONOS configuration instead
+ of manually changing it via ONOS REST APIs or CLI, otherwise, manual changes
+ will be overridden by config loader. If you need to apply config changes
+ during debugging or testing we recommend disabling the reconciliation loop
+ (``reconcile_mode: false``).
+
+Below is a reference of the available configuration parameters and their default values:
+
+.. code-block::
+
+ onos:
+ config:
+ reconcile_mode: true
+ reconcile_interval: 30s
+ image:
+ registry: ""
+ repository: opencord/onos-classic-helm-utils
+ tag: 0.1.0
+ pullPolicy: "IfNotPresent"
+ service_account_name: onos-config-loader
+ # NOTE these values won't change the credentials inside ONOS, they
+ # are only used by config loader for authenticating REST API calls.
+ username: karaf
+ password: karaf
+ netcfg: >
+ # {
+ # "devices" : {
+ # "device:leaf1" : {
+ # ...
+ # }
+ componentConfig:
+ # "org.onosproject.provider.lldp.impl.LldpLinkProvider": >
+ # {
+ # "enabled": "false"
+ # }
+
+
+* ``reconcile_mode``: If disable, config loader will only update the configuration once. Useful for debugging/testing.
+* ``reconcile_interval``: The interval in seconds between reconciliation checks.
+* ``image``: to pull the container image of the config loader.
+ Use the default values unless you have special requirements.
+
+ * ``registry``: If empty, the config loader image will be downloaded from Docker Hub.
+ * ``repository``: The repository of config push.
+ * ``tag``: The image tag of config loader.
+ * ``pullPolicy``: How Kubernetes pulls the image, available options are `Never, Always, IfNotPresent`.
+
+* ``service_account_name``: Name of the Kubernetes Service Account used by the config loader pods to read
+ information from Kubernetes.
+* ``username``: Username used in ONOS REST API calls.
+* ``password``: Password used in ONOS REST API calls.
+* ``netcfg``: ONOS network configuration as JSON blob.
+* ``componentConfig``: A map of ONOS component IDs and corresponding list of configurations as a JSON map.
diff --git a/configuration/config-pusher.rst b/configuration/config-pusher.rst
deleted file mode 100644
index 194c219..0000000
--- a/configuration/config-pusher.rst
+++ /dev/null
@@ -1,60 +0,0 @@
-Configuration Pusher
-====================
-
-Once the SD-Fabric has been deployed to the Kubernetes, a special Kubernetes
-pod, config pusher will enter to a reconciliation loop to update both
-network configuration and component configuration to ONOS controller.
-
-Config pusher will read running state from ONOS controller and
-then update the ONOS with desired state if running state isn't same as desired state.
-
-.. image:: ../images/config-pusher.svg
-
-.. note::
- We highly recommend using the Helm Chart to update the configuration instead of manually changing the ONOS unless you're working on debugging or testing.
-
-Config Pusher supports variety options to adjust its behavior and you can change it via Helm Chart's value file.
-
-Below is the default values of config pusher and please change it to meet your environment.
-
-
-.. code-block::
-
- onos:
- config:
- reconcile_mode: true
- reconcile_interval: 30s
- image:
- registry: ""
- repository: opencord/onos-classic-helm-utils
- tag: 0.1.0
- pullPolicy: "IfNotPresent"
- service_account_name: onos-config-loader
- # NOTE that these values won't change the configuration in the ONOS container, they are only used in the script that loads the config
- username: karaf
- password: karaf
- # netcfg: >
- # {}
- # componentConfig:
- # "org.onosproject.provider.lldp.impl.LldpLinkProvider": >
- # {
- # "enabled": "false"
- # }
-
-
-* ``reconcile_mode``: If disable, config pusher will only update the configuration once, it's useful for debugging.
-* ``reconcile_interval``: The interval(seconds) config pusher used to perform reconciliation loop.
-* ``image``: About how to pull config pusher's container image, keep it default unless you have special requirement.
-
- * ``registry``: If empty, the config pusher image will be downloaded from Docker Hub.
- * ``repository``: The repository of config push.
- * ``tag``: The image tag of config pusher.
- * ``pullPolicy``: How Kubernetes pull the image, available options are `Never, Always, IfNotPresent`.
-
-* ``service_account_name``: Name of the Kubernetes Service Account, config pusher pods needs the permission to read
- information from Kubernetes.
-* ``username``: Username is used to access ONOS,
-* ``password``: Password is used to access ONOS.
-* ``netcfg``: Json data for network configuration.
-* ``componentConfig``: A map contains "component-name": "json config" that will be iterated
- over and sent via POST to ONOS.
diff --git a/configuration/network.rst b/configuration/network.rst
index dbe6426..5bf70aa 100644
--- a/configuration/network.rst
+++ b/configuration/network.rst
@@ -3,8 +3,8 @@
Network Configuration
=====================
SD-Fabric uses several different types of network configurations.
-We only focus on ``devices`` and ``ports`` configuration in this section.
-With these configured properly, SD-Fabric can provide basic L2/L3 connectivity.
+We only focus on ``devices`` and ``ports`` configuration in this section, which
+are used to provide basic L2/L3 connectivity.
See :ref:`advanced-features` for advanced feature configurations.
@@ -36,7 +36,7 @@
}
}
-- ``device:leaf1``: DPID of the device.
+- ``device:leaf1``: arbitrary globally unique identifier of the device, must always be prefixed with ``device:``.
- ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in
forwarding IPv4 traffic. Can be arbitrary and should be globally unique.
@@ -61,38 +61,80 @@
- ``isEdgeRouter``: True for leaf switches. False for spine switches.
-- ``adjacencySids``: Deprecated. Just put an empty array for now.
+- ``adjacencySids``: Deprecated. Always use an empty array.
-- ``name``: Name of the device. It is an arbitrary name to identify the device easily.
+- ``name``: Human friendly name used in the ONOS UI.
-- ``managementAddress``: gRPC endpoint of the Stratum device and a numerical device ID.
- The IP address can be replaced by domain name as well.
+- ``managementAddress``: gRPC endpoint of the Stratum device and the P4Runtime
+ internal device ID associated to the ASIC (usually ``1``),
+ in the format of ``grpc://[device_addr]?device_id=[P4Runtime device ID]``
+ The IP address can be replaced by a domain name.
- ``driver``: ``stratum-bmv2`` or ``stratum-tofino``, depending on which switch this is.
-- ``pipeconf``: A list of available pipeconfs can be dumped by running ``pipeconfs`` in ONOS CLI.
- Select the pipeconf you would like to use for this device.
+- ``pipeconf``: the P4 program to deploy on this switch. A list of available
+ pipeconfs can be dumped by running ``pipeconfs`` in the ONOS CLI. When running
+ with Tofino-based devices, we provide pre-installed pipeconfs with ID
+ ``org.stratumproject.<profile>.<device-type>_<bf-sde-version>``:
+
+ - The available *profiles* are:
+
+ - ``fabric``: for basic L2/L3 capabilities
+ - ``fabric-spgw``: with 4G/5G mobile user plane support
+ - ``fabric-int``: with INT support
+ - ``fabric-spgw-int``: with SPGW and INT support
+
+ - The supported *device-types*:
+
+ - ``montara``: for dual-pipe Tofino ASIC SKUs
+ - ``mavericks``: for quad-pipe Tofino ASIC SKUs
+
+ - The Intel/Barefoot SDE version used in Stratum, e.g., ``sde_9_5_0``
.. caution::
- We should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
+ You should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
``ipv6NodeSid``. Please check here for the reserved values:
http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml
.. note::
- Most of the SD-Fabric configurations support dynamic configuration updates.
- Unfortunately, SD-Fabric currently **do not support dynamic device
- configuration updates**. You will have to restart the device when if
- corresponding device configuration changes.
+ Most of the SD-Fabric configurations support dynamic updates. Unfortunately,
+ we currently **do not support dynamic device configuration updates**. You
+ will have to restart (reboot) the switch if the corresponding device
+ configuration changes.
Having said that, when introducing a completely new device in the network,
- the device configurations pushed before the device's connection should
- apply correctly.
+ the device configurations pushed before ONOS connects to the switch for the
+ first time should be applied correctly.
+
+Port IDs for Tofino-based devices
+---------------------------------
+
+Before describing the ONOS netcfg, it is worth nothing how we refer to ports for
+Tofino-based devices. Netcfg uses the format ``device:<name>/<port-number>``.
+
+``<port-number>`` is a special value that is usually different than the number
+shown in the switch front panel. It is the same number used for P4 table
+programming and depends on the specific Tofino ASIC SKU (e.g., dual-pipe vs.
+quad-pipe) and switch vendor/platform. In Stratum this is often referred to as
+the *SDK port ID*, as this is the number used for all Tofino SDK calls. In Intel
+documentation this is referred to as the ``DP_ID``. We plan to remove this
+dependency on such a low level detail in future releases, but for now, to find
+out the mapping between front-panel ports and ``DP_ID`` you have the following
+options:
+
+- Ask your switch vendor
+- Use the command `pm.show` on the BF shell of a running Stratum instance
+ (see :ref:`troubleshooting_guide`)
Bridging and Unicast Routing
----------------------------
+In the following we illustrate how to enable basic bridging and routing on a
+per-port basis.
+
.. attention::
- VLAN **4094** is reserved for unconfigured ports (e.g. spine facing ports)
+ - VLAN **4090** is reserved for pseudowire transport flow rules on the spines
Access Ports
^^^^^^^^^^^^
@@ -103,13 +145,13 @@
{
"ports" : {
- "of:0000000000000204/12" : {
+ "device:leaf1/12" : {
"interfaces" : [{
"name" : "serverA-intf",
"vlan-untagged": 10
}]
},
- "of:0000000000000204/16" : {
+ "device:leaf1/16" : {
"interfaces" : [{
"name" : "serverB-intf",
"vlan-untagged": 10
@@ -118,7 +160,7 @@
}
}
-The example above shows two ports (12 and 16) on switch ``of:204`` that have
+The example above shows two ports (12 and 16) on switch ``leaf1`` that have
been assigned to VLAN 10 using the ``vlan-untagged`` keyword.
It simply means that packets come in and leave out of these switches untagged,
@@ -138,14 +180,14 @@
{
"ports" : {
- "of:0000000000000204/12" : {
+ "device:leaf1/12" : {
"interfaces" : [{
"name" : "serverA-intf",
"ips" : [ "10.0.1.254/24"],
"vlan-untagged": 10
}]
},
- "of:0000000000000204/16" : {
+ "device:leaf1/16" : {
"interfaces" : [{
"name" : "serverB-intf",
"ips" : [ "10.0.1.254/24"],
@@ -170,8 +212,8 @@
the same VLAN. This is possible by adding more subnet/gateway IPs in the
``ips`` array.
-.. tip::
- One subnet cannot be configured on multiple leaf switches.
+.. attention::
+ The same subnet cannot be configured on multiple leaf switches.
We usually configure one subnet for all the ports on the same leaf switch.
@@ -183,7 +225,7 @@
{
"ports" : {
- "of:0000000000000204/24" : {
+ "device:leaf1/24" : {
"interfaces" : [{
"name" : "serverA-intf",
"ips" : [ "10.0.2.254/24", "10.0.4.254/24" ],
@@ -193,8 +235,8 @@
}
}
-The configuration above for port 24 on switch of:204 shows two VLANs 20 and 40
-configured on that port, with corresponding subnets and gateway IPs.
+The configuration above for port 24 on switch ``leaf1`` shows two VLANs 20 and
+40 configured on that port, with corresponding subnets and gateway IPs.
Note that there is no specific ordering required in the ``ips`` or
``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets.
@@ -216,7 +258,7 @@
{
"ports" : {
- "of:0000000000000204/24" : {
+ "device:leaf1/24" : {
"interfaces" : [ {
"name" : "serverA-intf",
"ips" : [ "10.0.2.254/24", "10.0.4.254/24", "10.0.1.254/24" ],
@@ -230,8 +272,8 @@
Note that it is also necessary to configure the subnet/gateway IP corresponding
to the native VLAN if you wish to route out of that VLAN.
-Configuring interface for IPv6
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Configuring interface for IPv6 [#f1]_
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is similar to configure IPv6 routing. Simply replace the addresses in
``ips`` with IPv6 addresses. For example:
@@ -240,7 +282,7 @@
{
"ports" : {
- "of:0000000000000204/24" : {
+ "device:leaf1/24" : {
"interfaces" : [ {
"name" : "serverA-intf",
"ips" : [ "10.0.2.254/24", "2000::1ff/120" ],
@@ -254,8 +296,8 @@
There is a known issue that breaks dynamic VLAN configuration.
Until the issue get resolved, you need to restart the switch agent to reinstall the flows.
-IPv6 Router Advertisement
-^^^^^^^^^^^^^^^^^^^^^^^^^
+IPv6 Router Advertisement [#f1]_
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Router Advertisement overview
"""""""""""""""""""""""""""""
@@ -326,7 +368,7 @@
{
"ports": {
- "of:0000000000000018/16": {
+ "device:leaf2/16": {
"interfaces": [{
"ips": [ "192.168.114.1/24", "2001:0558:FF10:04C9::2:1ff/120", "FE80::4EA8:2AFF:FE24:8E5F/120" ],
"vlan-untagged": "11",
@@ -349,7 +391,7 @@
{
"devices": {
- "of:0000000000000018": {
+ "device:leaf2": {
"routeradvertisement" : {
"prefixes": [ "2001:0558:FF10:04C9::3:1ff/120"]
}
@@ -388,7 +430,7 @@
shown in the example below.
The fabric is not designed to be one big Ethernet fabric. The bridging domain
-is restricted to within one ToR.
+is restricted to one ToR.
If the bridging domain is extended across two ToRs directly linked to each
other, there is a chance of loops.
@@ -397,3 +439,7 @@
not be used as such.
.. image:: ../images/config-vlan-invalid.png
+
+.. rubric:: Footnotes
+
+.. [#f1] IPv6 support on the data plane (P4 program) is still work-in-progress.
diff --git a/deployment.rst b/deployment.rst
index a0eda5b..12a0d98 100644
--- a/deployment.rst
+++ b/deployment.rst
@@ -2,6 +2,8 @@
SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
SPDX-License-Identifier: Apache-2.0
+.. _deployment_guide:
+
Deployment Guide
================
diff --git a/dict.txt b/dict.txt
index e493cee..c839c43 100644
--- a/dict.txt
+++ b/dict.txt
@@ -104,6 +104,7 @@
preprocessor
programmability
protobuf
+pseudowire
reStructuredText
recirculation
routable
diff --git a/images/config-pusher.svg b/images/config-loader.svg
similarity index 100%
rename from images/config-pusher.svg
rename to images/config-loader.svg
diff --git a/index.rst b/index.rst
index 1f2cfc1..7024d10 100644
--- a/index.rst
+++ b/index.rst
@@ -13,19 +13,19 @@
- SD-Fabric is a **natively disaggregated solution using bare metal switches with merchant silicon ASICs**.
Instead of using OEM networking hardware, SD-Fabric uses hardware directly from ODMs.
The trend of using bare metal (white box) switches is unmistakable in the networking industry today,
- spurred by the massive bandwidthdensity and growing sophistication of merchant silicon ASICs.
+ spurred by the massive bandwidth density and growing sophistication of merchant silicon ASICs.
- SD-Fabric is **based on SDN principles using P4-enabled programmable data planes**
(such as the Intel® Tofino® ASICs).
P4 allows for the introduction of new features that cannot be found in traditional fabrics built on
fixed-function ASICs.
- For example, SDFabric makes possible a 5G mobile core User Plane Function (UPF)
- (this UPF is part of the SD-Core™ project, which runs on SD-Fabric).
- And by externalizing the network’s control, management functions, and policy decisions in the ONOS™ SDN controller,
+ For example, SD-Fabric provides a high performance 4G/5G mobile core User Plane Function (UPF) integrated
+ with SD-Core™ project.
+ And by externalizing the network's control, management functions, and policy decisions in the ONOS™ SDN controller,
SD-Fabric provides network operators with a number of SDN benefits
- (compared to traditional embedded network control like BGP) that include customizability,
+ when compared to a traditional embedded network control like BGP, such as enhanced customizability,
centralized configuration, automation, and simplified operations and troubleshooting.
- As a result, SDFabric is truly software defined in both the control and data planes.
+ As a result, SD-Fabric is truly software defined in both the control and data planes.
- SD-Fabric is **optimized for application developers**,
exposing APIs to enable developers to get better visibility and control while enabling custom P4 forwarding logic
@@ -38,9 +38,9 @@
The programmability of SD-Fabric has enabled the implementation of fine-grained measurement
(via In-band Network Telemetry (INT)), network verification, and closed loop control (see https://prontoproject.org/).
Furthermore, as open source,
- SD-Fabric is further secured through the benefit of open inspection by a broad community.
+ SD-Fabric is further secured through the benefit of open inspection by a broader community.
-- SD-Fabric can be **deployed as-a-Service from the public cloud**. When deployed as-aService,
+- SD-Fabric can be **deployed as-a-Service from the public cloud**. When deployed as-a-Service,
SD-Fabric provides a full stack implementation designed to run on white box switches at the network edge,
with centralized CI/CD and operations running from the public cloud controlling multiple edges.
@@ -51,15 +51,16 @@
.. image:: images/overview-topo.png
:width: 500px
-- **Right-sized Topology**: Scale from smallest HA setup of a pair-of-ToRs (sub/single
- rack) to a full leaf-spine fabric (multiple racks) as needed in edge or DC deployments.
-- **API Driven**: Programmable API with ability to drop or reroute traffic plus obtain
- telemetry and program application workloads across switches, CPU and NICs.
-- **Cloud Managed**: Fully integrated and configured by Aether™ Management platform.
-- **5G as a Workload**: Tofino + BESS UPF scalable on demand, Smart NIC + BESS UPF
- extensions, 5G slicing as primary construct.
-- **Visibility**: Throughout the entire network enabling closed loop control.
-- **Integration**: With K8s CNI and overlay enabling true end-to-end programmability and visibility.
+- **Right-sized Topology**: Scale from a single Top-of-Rack (ToR) switch, to an
+ Highly-Available (HA) setup with paired ToRs, to a full leaf-spine fabric
+ for multi-rack deployments.
+- **API Driven**: Well-defined APIs provide the ability to drop or reroute traffic,
+ create slices with configurable QoS, obtain telemetry, and more.
+- **Cloud Managed**: Fully integrated and configured by Aether™ Management Platform.
+- **5G as a Workload**: Reduce CPU load and increase performance by terminating GTP-U tunnels
+ directly in the fabric switches, with QoS and slicing support.
+- **End-to-End Visibility**: With Inband Network Telemetry (INT) on switches and end hosts,
+ enabling closed loop control.
Get Started
-----------
diff --git a/specification.rst b/specification.rst
index f59ccc0..b2c502f 100644
--- a/specification.rst
+++ b/specification.rst
@@ -1,6 +1,8 @@
Specification
=============
+In the following we provide an exhaustive list of all features supported.
+
SDN Features
------------
- ONOS cluster of all-active N instances affording N-way redundancy and scale, where N = 3 or N = 5
@@ -33,8 +35,9 @@
-----------
IP connectivity
- - IPv4 and IPv6 unicast routing (internal use of MPLS Segment Routing)
+ - IPv4 and IPv6 [#f1]_ unicast routing (internal use of MPLS Segment Routing)
- Subnetting configuration on all non-spine facing leaf ports; no configuration required on any spine port
+ - Equal Cost Multi-Path (ECMP) for traffic across spine switches
- IPv6 router advertisement
- ARP, NDP, IGMP handling
- Number of flows in spines greatly simplified by MPLS Segment Routing
@@ -73,17 +76,20 @@
- Provide easy access for 3rd party edge application developers and for the Aether centralized management platform
- Support for traffic redirecting, dropping, network slicing and QoS
-Programmability
----------------
-- Support for Stratum, P4Runtime and gNMI and P4 programs
-- Innovative services enabled by programmable pipeline
- - 4G/5G UPF - GTP encap/decap, idle-mode buffering, QoS and more
- - BNG - PPPoE, anti-spoofing, accounting and more
+Data Plane Programmability
+--------------------------
+- Support for Stratum, P4Runtime/gNMI, and P4 programs
+- Innovative services enabled by programmable data plane:
+
+ - 4G/5G User Plane Function (UPF): GTP encap/decap, usage reporting, downlink buffering and data notifications,
+ QoS and more, with integration with mobile core control plane via PFCP protocol (3GPP standard interface).
+ - Inband Network Telemetry (INT): INT-XD mode with support for flow reports, drop reports,
+ queue congestion reports, with smart filters to reduce volume of reports ingested by the INT collector.
Troubleshooting & Diagnostics
-----------------------------
-- T3: Troubleshooting tool to diagnose broken forwarding paths fabric wide
-- ONOS-diags: One-click Diagnostics collection tool
+- T3: Troubleshooting tool to diagnose broken forwarding paths fabric wide (work in progress)
+- ONOS-diags: One-click diagnostics collection tool for issue reporting
.. _Topology:
@@ -104,14 +110,14 @@
Single Leaf Pair (Dual-Homing)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Compared to single switch, it provides more redundancy in terms of server NIC failure and link failure.
+Compared to a single switch, it provides redundancy in terms of server NIC failure and link failure.
.. image:: images/topology-pair.png
:width: 225px
Leaf-Spine (without pairing)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Compared to single switch, it offers more redundancy in terms of switch failure and provides better scalability.
+Provide horizontal-scalability for multi-rack deployments, with redundancy for spine switch failures:
.. image:: images/topology-2x2.png
:width: 300px
@@ -135,22 +141,21 @@
Resiliency
----------
-Provides HA in following scenarios
+Provides HA in the following scenarios:
- Controller instance failure (requires 3 or 5 node ONOS cluster)
- - Link failures
- - Spine failure
+ - Leaf-spine link failures
+ - Spine switch failure
-Further HA support in following failure scenarios with dual-homing enabled
+Further HA support in following failure scenarios with dual-homing enabled:
- - Leaf failure
+ - Leaf switch failure
- Upstream router failure
- Host NIC failure
Scalability
-----------
In Production
-
- Up to 80k routes (with route optimization)
- 170k Flows
- 600 direct-attached hosts
@@ -158,22 +163,22 @@
- 2 spine switches
In Pre-Production
-
- Up to 120k routes (with route optimization)
- 250k flows
- 600 direct-attached hosts
- 8 leaf switches
- 2 spine switches
- - 5000 active UEs, 10 call per second
+
+4G/5G specific
+ - 5000 active UEs, 10 calls per second
Security
--------
- - TLS-secured connection between controllers and switches (premium feature)
- - AAA 802.1x authentication
+ - TLS-secured gRPC connection between controllers and switches (work-in-progress)
Aether-ready
------------
-Fully integrated with Aether (5G/LTE private enterprise edge cloud solution)
+Fully integrated with Aether (5G/4G private enterprise edge cloud solution)
including deployment automation, CI/CD, logging, monitoring, and alerting.
Overlay Support
@@ -193,16 +198,26 @@
- CPU: 32 Cores
- RAM: 128GB RAM. 64GB dedicated to ONOS JVM heap
+Recommendation (per ONOS instance) for 5K UEs when enabling UPF:
+
+ - CPU: 1 Cores
+ - RAM: 4GB RAM
+
+
White Box Switch Hardware
-------------------------
- Multi-vendor: APS Networks™, Dell™, Delta Networks™, Edgecore Networks™, Inventec™, Netburg™, QCT™
- Multi-chipset:
- - Intel Tofino (supports all features, including programmability, UPF & INT)
+ - Intel Tofino (supports all features, including UPF & INT)
- Broadcom Tomahawk®, Tomahawk+®, Trident2 (traditional fabric features only)
-- 1/10G, 25G, 40G, 100G
+- 1/10G, 25G, 40G, and 100G ports
- Refer to Supported Devices list in https://github.com/stratum/stratum for the most up-to-date hardware list
White Box Switch Software
-------------------------
- Open source ONL, ONIE, Docker, Kubernetes
- Stratum available from ONF
+
+.. rubric:: Footnotes
+
+.. [#f1] IPv6 support on the data plane (P4 program) is still work-in-progress.
diff --git a/troubleshooting.rst b/troubleshooting.rst
index 222c934..6a64d34 100644
--- a/troubleshooting.rst
+++ b/troubleshooting.rst
@@ -44,7 +44,7 @@
Stratum
^^^^^^^
-To execute following BF SDE commands,
+To execute following BF Shell commands,
- Login to Stratum switch by `ssh root@<switch_ip>`. Default password is `onl`
- Attach to Stratum docker container by `docker attach \`docker ps | grep stratum-bfrt | awk \'{print $1}\'\``
@@ -52,8 +52,8 @@
- Hit `enter` for the prompt
- Use `<Ctrl-P><Ctrl-Q>` to exit the container. Do not use `<Ctrl-C>` since it will terminate the process.
-BF SDE
-""""""
+BF Shell
+""""""""
- `pm.show`: List port configurations. `-a` to list all ports.
K8s troubleshooting
