First pass up to configuration
Change-Id: Ifd82c38aa18d5216c5772a64d9d77046def005fe
diff --git a/configuration/chassis.rst b/configuration/chassis.rst
index 8d75245..9181a6a 100644
--- a/configuration/chassis.rst
+++ b/configuration/chassis.rst
@@ -3,37 +3,44 @@
Stratum Chassis Configuration
=============================
+Chassis config is an internal data structure used by Stratum. The term "chassis"
+refers to a switching box with one or more switching nodes (e.g., chips). The
+chassis config file is usually placed on disk and loaded when starting Stratum.
+This config includes all the not-so-frequent settings that are required before
+the switch can accept flow programming requests from the controller.
+
+In the following, we provide a **non-exhaustive overview of the configurations
+relevant to SD-Fabric for Tofino-based devices**. For a complete reference check
+out the `ChassisConfig Protobuf message format
+<https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L824-L833>`_
+and the `guide for running Stratum on Tofino-based switches
+<https://github.com/stratum/stratum/blob/main/stratum/hal/bin/barefoot/README.run.md>`_
+in the Stratum repository.
+
.. tip::
- Check out `examples <https://github.com/stratum/stratum/tree/main/stratum/hal/config>`_
- for every platform supported by Stratum.
+ Check out the `Stratum repository
+ <https://github.com/stratum/stratum/tree/main/stratum/hal/config>`_ for
+ examples of chassis config files for different switch platforms.
- See the `Deployment Guide <../deployment.rst>`_ to learn about how to deploy Stratum
+ See the :ref:`deployment_guide` to learn about how to deploy Stratum
with a custom chassis config.
-The Stratum chassis config is the internal data structure that encapsulates the so called
-"config" pushed to the entire chassis. The term "chassis" refers to the a switching box
-with one or more switching nodes managed by a management interface.
-
-The chassis config file will be placed on the disk of the device and loaded when starting the Stratum agent.
-The config includes all the not-so-frequent settings that are required before the switch
-can accept flow programming requests from the controller.
-
-A valid chassis config includes a ``chassis`` and a ``node`` field, and may includes one
-or more ``singleton_port`` field. See each section below for more detail.
-
.. note::
- In Stratum the external interface for pushing config is gNMI.
- The protobuf realization of the YANG models for the config is internally converted
- to a chassis config before it is consumed by the internal stack components.
+ In Stratum the external interface for pushing config is gNMI. The protobuf
+ realization of the YANG models for the config is internally converted to a
+ chassis config before it is consumed by the internal stack components. Not
+ all configurations are available through gNMI, for this reason we recommend
+ using the chassis config file.
- The `format of chassis config <https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L824-L833>`_
- is based on protobuf text format, check out
+ The file uses the protobuf text format, check out
`the protobuf language guide for more info <https://developers.google.com/protocol-buffers/docs/overview?hl=en>`_
Chassis
-------
+A valid chassis config includes a ``chassis``, a ``node`` field, and one
+or more ``singleton_port`` fields.
A ``chassis`` uniquely identifies a switch with a single management interface.
Each chassis may contain one or more slots (aka linecards),
@@ -41,13 +48,10 @@
A chassis contains the following fields:
-* ``platform``: The chassis platform. **Required**
+* ``platform``: The chassis platform, e.g., ``PLT_GENERIC_BAREFOOT_TOFINO`` for
+ Tofino-based devices (check out `here <https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L33-L47>`_ for other platforms). **Required**
* ``name``: An arbitrary name for the chassis. **Optional**
-* ``config_params``: Parameters configured for the entire chassis when config is pushed to the the switch. **Optional**
-
-.. tip::
-
- Check out the list of platforms `here <https://github.com/stratum/stratum/blob/main/stratum/hal/lib/common/common.proto#L33-L47>`_
+* ``config_params``: Parameters for the entire chassis, usually empty for Tofino-based devices. **Optional**
Node
----
@@ -57,68 +61,57 @@
A node contains the following fields:
-* ``id``: The unique ID of the switch node on the chassis as used by the P4Runtime controller. **Required**
+* ``id``: An arbitrary ID of the switch node on the chassis (corresponds to the P4Runtime device ID). **Required**
* ``name``: An arbitrary name for the switching node. **Optional**
-* ``slot``: The 1-base index of the slot (aka linecard) which this node belongs. **Required**
-* ``index``: The optional 1-base index of the node within the chassis. **Optional**
+* ``slot``: The 1-base index of the slot (aka linecard) to which this node belongs. **Required**
+* ``index``: An optional 1-based index of the node within the chassis. **Optional**
.. _singleton_port:
Singleton Port
--------------
-A ``singleton port`` in the chassis configuration uniquely identifies a single physical port on
-a single chassis.
+A ``singleton port`` uniquely identifies a switch port and it contains the following fields:
-A singleton port contains the following fields:
+* ``id``: An arbitrary ID, unique in the scope of a chassis. **Required**
+* ``name``: An optional user friendly name for the port (e.g., ``10/1`` to identify channel
+ 1 of front-panel port 10). **Required**
-* ``id``: The unique ID of the singleton port. **Required**
-* ``name``: An optional arbitrary name for the singleton port. **Required**
-
- The control plan (e.g., ONOS) can use this name to query port information as a gNMI path key.
+ ONOS obtains this name via gNMI and uses it internally to represent ports.
* ``slot``: The 1-base index of the slot (aka linecard) of the port. **Required**
* ``port``: The 1-base index of the singleton port on the slot. **Required**
* ``channel``: The 1-base channel index. Absence or zero means non-channelized. **Optional**
* ``speed_bps``: The speed of the port in bps. **Required**
* ``node``: The id of the corresponding node that the port belongs to. **Required**
-* ``config_params``: Parameters configured for this port. **Optional**
+* ``config_params``: **Optional**
- * ``admin_state``: The initial admin state for this port, port will be disabled by default.
+ * ``admin_state``: The initial admin state for this port, disabled by default.
* Choose from ``ADMIN_STATE_DISABLED`` or ``ADMIN_STATE_ENABLED``.
- * ``mtu``: The maximum transmission unit for this port.
* ``autoneg``: Whether auto-negotiation is enabled or not for this port.
* Choose from ``TRI_STATE_FALSE`` or ``TRI_STATE_TRUE``
* The initial configuration might be different if this field is empty, based on the
platform or the connector you are using.
- * ``fec_mode``: The
+ * ``fec_mode``: Forward Error Correction (FEC) mode
* Choose from ``FEC_MODE_ON``, ``FEC_MODE_OFF``, or ``FEC_MODE_AUTO``
- * ``loopback_mode``:
-
- * Choose from ``LOOPBACK_STATE_NONE``, ``LOOPBACK_STATE_MAC``, or ``LOOPBACK_STATE_PHY``
-
Example chassis config
----------------------
-In this example, we want to set up a **Tofino-based switch** with only one node/slot.
+In this example, we want to set up a **Tofino-based switch** with only one node, one slot,
+and two ports:
-And we want to set up two ports:
-
-* Port 1: 100G port without channelization, enabled by default, and disable auto auto-negotiation.
-* Port 2: break out to four 10G ports, uses different channels for each port and enable
- them by default with auto-negotiation.
+* Port 1: 100G port without channelization, enabled by default, without auto-negotiation.
+* Port 2: break out port with four 10G channels, enabled by default, with auto-negotiation.
.. image:: ../images/chassis-config-example.svg
:width: 500px
-Below is an example of a chassis configuration with a list of singleton port:
-
.. code-block::
description: "A chassis config example."
diff --git a/configuration/component.rst b/configuration/component.rst
index 88349b3..6c911c8 100644
--- a/configuration/component.rst
+++ b/configuration/component.rst
@@ -4,12 +4,13 @@
Different from network configurations,
component configurations are attributes and feature flags specific to ONOS components.
-For more details, please read ONOS Wiki `Component Configuration <https://wiki.onosproject.org/display/ONOS/Component+Configuration>`_
+For more details, please check the ONOS Wiki `Component Configuration <https://wiki.onosproject.org/display/ONOS/Component+Configuration>`_
Enable/Disable Phased Recovery
------------------------------
-Phased recovery is an experimental feature that ensure a
-recovering switch are programmed before pointing traffic from other switches towards it.
+Phased recovery is an experimental feature that aims at minimizing traffic
+disruption when recovering from a switch failure. When enabled, a recovering
+switch is programmed before pointing traffic from other switches towards it.
To check current status:
@@ -30,11 +31,11 @@
onos > cfg get org.onosproject.segmentrouting.phasedrecovery.impl.PhasedRecoveryManager phasedRecovery
name=phasedRecovery, type=boolean, value=true, defaultValue=false, description=Enabling phased recovery.
-Now the value is true and we have successfully enable phased recovery.
+Now the value is true and we have successfully enabled phased recovery.
To disable phased recovery, change the value to ``false``.
.. tip::
- Simply run ``cfg get`` in ONOS CLI to view all available component configurations supported by ONOS.
+ Simply run ``cfg get`` in the ONOS CLI to view all available component configurations supported by ONOS.
The default values are fine-tuned for SD-Fabric, and therefore we use them for most of the cases.
diff --git a/configuration/config-loader.rst b/configuration/config-loader.rst
new file mode 100644
index 0000000..51d8fbd
--- /dev/null
+++ b/configuration/config-loader.rst
@@ -0,0 +1,66 @@
+ONOS Configuration Loader
+=========================
+
+Config loader is a Kubernetes pod installed by the SD-Fabric Helm
+chart which implements a reconciliation loop to update the ONOS network
+configuration and component configuration.
+
+Config loader will periodically read the running configurations from ONOS and eventually
+push updates to ONOS if the running state isn't same as the desired state.
+
+.. image:: ../images/config-loader.svg
+
+.. note::
+ We recommend using the Helm values below to update the ONOS configuration instead
+ of manually changing it via ONOS REST APIs or CLI, otherwise, manual changes
+ will be overridden by config loader. If you need to apply config changes
+ during debugging or testing we recommend disabling the reconciliation loop
+ (``reconcile_mode: false``).
+
+Below is a reference of the available configuration parameters and their default values:
+
+.. code-block::
+
+ onos:
+ config:
+ reconcile_mode: true
+ reconcile_interval: 30s
+ image:
+ registry: ""
+ repository: opencord/onos-classic-helm-utils
+ tag: 0.1.0
+ pullPolicy: "IfNotPresent"
+ service_account_name: onos-config-loader
+ # NOTE these values won't change the credentials inside ONOS, they
+ # are only used by config loader for authenticating REST API calls.
+ username: karaf
+ password: karaf
+ netcfg: >
+ # {
+ # "devices" : {
+ # "device:leaf1" : {
+ # ...
+ # }
+ componentConfig:
+ # "org.onosproject.provider.lldp.impl.LldpLinkProvider": >
+ # {
+ # "enabled": "false"
+ # }
+
+
+* ``reconcile_mode``: If disable, config loader will only update the configuration once. Useful for debugging/testing.
+* ``reconcile_interval``: The interval in seconds between reconciliation checks.
+* ``image``: to pull the container image of the config loader.
+ Use the default values unless you have special requirements.
+
+ * ``registry``: If empty, the config loader image will be downloaded from Docker Hub.
+ * ``repository``: The repository of config push.
+ * ``tag``: The image tag of config loader.
+ * ``pullPolicy``: How Kubernetes pulls the image, available options are `Never, Always, IfNotPresent`.
+
+* ``service_account_name``: Name of the Kubernetes Service Account used by the config loader pods to read
+ information from Kubernetes.
+* ``username``: Username used in ONOS REST API calls.
+* ``password``: Password used in ONOS REST API calls.
+* ``netcfg``: ONOS network configuration as JSON blob.
+* ``componentConfig``: A map of ONOS component IDs and corresponding list of configurations as a JSON map.
diff --git a/configuration/config-pusher.rst b/configuration/config-pusher.rst
deleted file mode 100644
index 194c219..0000000
--- a/configuration/config-pusher.rst
+++ /dev/null
@@ -1,60 +0,0 @@
-Configuration Pusher
-====================
-
-Once the SD-Fabric has been deployed to the Kubernetes, a special Kubernetes
-pod, config pusher will enter to a reconciliation loop to update both
-network configuration and component configuration to ONOS controller.
-
-Config pusher will read running state from ONOS controller and
-then update the ONOS with desired state if running state isn't same as desired state.
-
-.. image:: ../images/config-pusher.svg
-
-.. note::
- We highly recommend using the Helm Chart to update the configuration instead of manually changing the ONOS unless you're working on debugging or testing.
-
-Config Pusher supports variety options to adjust its behavior and you can change it via Helm Chart's value file.
-
-Below is the default values of config pusher and please change it to meet your environment.
-
-
-.. code-block::
-
- onos:
- config:
- reconcile_mode: true
- reconcile_interval: 30s
- image:
- registry: ""
- repository: opencord/onos-classic-helm-utils
- tag: 0.1.0
- pullPolicy: "IfNotPresent"
- service_account_name: onos-config-loader
- # NOTE that these values won't change the configuration in the ONOS container, they are only used in the script that loads the config
- username: karaf
- password: karaf
- # netcfg: >
- # {}
- # componentConfig:
- # "org.onosproject.provider.lldp.impl.LldpLinkProvider": >
- # {
- # "enabled": "false"
- # }
-
-
-* ``reconcile_mode``: If disable, config pusher will only update the configuration once, it's useful for debugging.
-* ``reconcile_interval``: The interval(seconds) config pusher used to perform reconciliation loop.
-* ``image``: About how to pull config pusher's container image, keep it default unless you have special requirement.
-
- * ``registry``: If empty, the config pusher image will be downloaded from Docker Hub.
- * ``repository``: The repository of config push.
- * ``tag``: The image tag of config pusher.
- * ``pullPolicy``: How Kubernetes pull the image, available options are `Never, Always, IfNotPresent`.
-
-* ``service_account_name``: Name of the Kubernetes Service Account, config pusher pods needs the permission to read
- information from Kubernetes.
-* ``username``: Username is used to access ONOS,
-* ``password``: Password is used to access ONOS.
-* ``netcfg``: Json data for network configuration.
-* ``componentConfig``: A map contains "component-name": "json config" that will be iterated
- over and sent via POST to ONOS.
diff --git a/configuration/network.rst b/configuration/network.rst
index dbe6426..5bf70aa 100644
--- a/configuration/network.rst
+++ b/configuration/network.rst
@@ -3,8 +3,8 @@
Network Configuration
=====================
SD-Fabric uses several different types of network configurations.
-We only focus on ``devices`` and ``ports`` configuration in this section.
-With these configured properly, SD-Fabric can provide basic L2/L3 connectivity.
+We only focus on ``devices`` and ``ports`` configuration in this section, which
+are used to provide basic L2/L3 connectivity.
See :ref:`advanced-features` for advanced feature configurations.
@@ -36,7 +36,7 @@
}
}
-- ``device:leaf1``: DPID of the device.
+- ``device:leaf1``: arbitrary globally unique identifier of the device, must always be prefixed with ``device:``.
- ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in
forwarding IPv4 traffic. Can be arbitrary and should be globally unique.
@@ -61,38 +61,80 @@
- ``isEdgeRouter``: True for leaf switches. False for spine switches.
-- ``adjacencySids``: Deprecated. Just put an empty array for now.
+- ``adjacencySids``: Deprecated. Always use an empty array.
-- ``name``: Name of the device. It is an arbitrary name to identify the device easily.
+- ``name``: Human friendly name used in the ONOS UI.
-- ``managementAddress``: gRPC endpoint of the Stratum device and a numerical device ID.
- The IP address can be replaced by domain name as well.
+- ``managementAddress``: gRPC endpoint of the Stratum device and the P4Runtime
+ internal device ID associated to the ASIC (usually ``1``),
+ in the format of ``grpc://[device_addr]?device_id=[P4Runtime device ID]``
+ The IP address can be replaced by a domain name.
- ``driver``: ``stratum-bmv2`` or ``stratum-tofino``, depending on which switch this is.
-- ``pipeconf``: A list of available pipeconfs can be dumped by running ``pipeconfs`` in ONOS CLI.
- Select the pipeconf you would like to use for this device.
+- ``pipeconf``: the P4 program to deploy on this switch. A list of available
+ pipeconfs can be dumped by running ``pipeconfs`` in the ONOS CLI. When running
+ with Tofino-based devices, we provide pre-installed pipeconfs with ID
+ ``org.stratumproject.<profile>.<device-type>_<bf-sde-version>``:
+
+ - The available *profiles* are:
+
+ - ``fabric``: for basic L2/L3 capabilities
+ - ``fabric-spgw``: with 4G/5G mobile user plane support
+ - ``fabric-int``: with INT support
+ - ``fabric-spgw-int``: with SPGW and INT support
+
+ - The supported *device-types*:
+
+ - ``montara``: for dual-pipe Tofino ASIC SKUs
+ - ``mavericks``: for quad-pipe Tofino ASIC SKUs
+
+ - The Intel/Barefoot SDE version used in Stratum, e.g., ``sde_9_5_0``
.. caution::
- We should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
+ You should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
``ipv6NodeSid``. Please check here for the reserved values:
http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml
.. note::
- Most of the SD-Fabric configurations support dynamic configuration updates.
- Unfortunately, SD-Fabric currently **do not support dynamic device
- configuration updates**. You will have to restart the device when if
- corresponding device configuration changes.
+ Most of the SD-Fabric configurations support dynamic updates. Unfortunately,
+ we currently **do not support dynamic device configuration updates**. You
+ will have to restart (reboot) the switch if the corresponding device
+ configuration changes.
Having said that, when introducing a completely new device in the network,
- the device configurations pushed before the device's connection should
- apply correctly.
+ the device configurations pushed before ONOS connects to the switch for the
+ first time should be applied correctly.
+
+Port IDs for Tofino-based devices
+---------------------------------
+
+Before describing the ONOS netcfg, it is worth nothing how we refer to ports for
+Tofino-based devices. Netcfg uses the format ``device:<name>/<port-number>``.
+
+``<port-number>`` is a special value that is usually different than the number
+shown in the switch front panel. It is the same number used for P4 table
+programming and depends on the specific Tofino ASIC SKU (e.g., dual-pipe vs.
+quad-pipe) and switch vendor/platform. In Stratum this is often referred to as
+the *SDK port ID*, as this is the number used for all Tofino SDK calls. In Intel
+documentation this is referred to as the ``DP_ID``. We plan to remove this
+dependency on such a low level detail in future releases, but for now, to find
+out the mapping between front-panel ports and ``DP_ID`` you have the following
+options:
+
+- Ask your switch vendor
+- Use the command `pm.show` on the BF shell of a running Stratum instance
+ (see :ref:`troubleshooting_guide`)
Bridging and Unicast Routing
----------------------------
+In the following we illustrate how to enable basic bridging and routing on a
+per-port basis.
+
.. attention::
- VLAN **4094** is reserved for unconfigured ports (e.g. spine facing ports)
+ - VLAN **4090** is reserved for pseudowire transport flow rules on the spines
Access Ports
^^^^^^^^^^^^
@@ -103,13 +145,13 @@
{
"ports" : {
- "of:0000000000000204/12" : {
+ "device:leaf1/12" : {
"interfaces" : [{
"name" : "serverA-intf",
"vlan-untagged": 10
}]
},
- "of:0000000000000204/16" : {
+ "device:leaf1/16" : {
"interfaces" : [{
"name" : "serverB-intf",
"vlan-untagged": 10
@@ -118,7 +160,7 @@
}
}
-The example above shows two ports (12 and 16) on switch ``of:204`` that have
+The example above shows two ports (12 and 16) on switch ``leaf1`` that have
been assigned to VLAN 10 using the ``vlan-untagged`` keyword.
It simply means that packets come in and leave out of these switches untagged,
@@ -138,14 +180,14 @@
{
"ports" : {
- "of:0000000000000204/12" : {
+ "device:leaf1/12" : {
"interfaces" : [{
"name" : "serverA-intf",
"ips" : [ "10.0.1.254/24"],
"vlan-untagged": 10
}]
},
- "of:0000000000000204/16" : {
+ "device:leaf1/16" : {
"interfaces" : [{
"name" : "serverB-intf",
"ips" : [ "10.0.1.254/24"],
@@ -170,8 +212,8 @@
the same VLAN. This is possible by adding more subnet/gateway IPs in the
``ips`` array.
-.. tip::
- One subnet cannot be configured on multiple leaf switches.
+.. attention::
+ The same subnet cannot be configured on multiple leaf switches.
We usually configure one subnet for all the ports on the same leaf switch.
@@ -183,7 +225,7 @@
{
"ports" : {
- "of:0000000000000204/24" : {
+ "device:leaf1/24" : {
"interfaces" : [{
"name" : "serverA-intf",
"ips" : [ "10.0.2.254/24", "10.0.4.254/24" ],
@@ -193,8 +235,8 @@
}
}
-The configuration above for port 24 on switch of:204 shows two VLANs 20 and 40
-configured on that port, with corresponding subnets and gateway IPs.
+The configuration above for port 24 on switch ``leaf1`` shows two VLANs 20 and
+40 configured on that port, with corresponding subnets and gateway IPs.
Note that there is no specific ordering required in the ``ips`` or
``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets.
@@ -216,7 +258,7 @@
{
"ports" : {
- "of:0000000000000204/24" : {
+ "device:leaf1/24" : {
"interfaces" : [ {
"name" : "serverA-intf",
"ips" : [ "10.0.2.254/24", "10.0.4.254/24", "10.0.1.254/24" ],
@@ -230,8 +272,8 @@
Note that it is also necessary to configure the subnet/gateway IP corresponding
to the native VLAN if you wish to route out of that VLAN.
-Configuring interface for IPv6
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Configuring interface for IPv6 [#f1]_
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is similar to configure IPv6 routing. Simply replace the addresses in
``ips`` with IPv6 addresses. For example:
@@ -240,7 +282,7 @@
{
"ports" : {
- "of:0000000000000204/24" : {
+ "device:leaf1/24" : {
"interfaces" : [ {
"name" : "serverA-intf",
"ips" : [ "10.0.2.254/24", "2000::1ff/120" ],
@@ -254,8 +296,8 @@
There is a known issue that breaks dynamic VLAN configuration.
Until the issue get resolved, you need to restart the switch agent to reinstall the flows.
-IPv6 Router Advertisement
-^^^^^^^^^^^^^^^^^^^^^^^^^
+IPv6 Router Advertisement [#f1]_
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Router Advertisement overview
"""""""""""""""""""""""""""""
@@ -326,7 +368,7 @@
{
"ports": {
- "of:0000000000000018/16": {
+ "device:leaf2/16": {
"interfaces": [{
"ips": [ "192.168.114.1/24", "2001:0558:FF10:04C9::2:1ff/120", "FE80::4EA8:2AFF:FE24:8E5F/120" ],
"vlan-untagged": "11",
@@ -349,7 +391,7 @@
{
"devices": {
- "of:0000000000000018": {
+ "device:leaf2": {
"routeradvertisement" : {
"prefixes": [ "2001:0558:FF10:04C9::3:1ff/120"]
}
@@ -388,7 +430,7 @@
shown in the example below.
The fabric is not designed to be one big Ethernet fabric. The bridging domain
-is restricted to within one ToR.
+is restricted to one ToR.
If the bridging domain is extended across two ToRs directly linked to each
other, there is a chance of loops.
@@ -397,3 +439,7 @@
not be used as such.
.. image:: ../images/config-vlan-invalid.png
+
+.. rubric:: Footnotes
+
+.. [#f1] IPv6 support on the data plane (P4 program) is still work-in-progress.