Network Configuration
=====================
SD-Fabric uses several different types of network configurations.
We only focus on ``devices`` and ``ports`` configuration in this section.
With these configured properly, SD-Fabric can provide basic L2/L3 connectivity.

See :ref:`advanced-features` for advanced feature configurations.

Device Configuration
--------------------
Each switch in SD-Fabric requires a device config.

.. code-block:: json

    {
      "devices" : {
        "device:leaf1" : {
          "segmentrouting" : {
            "ipv4NodeSid" : 101,
            "ipv4Loopback" : "192.168.0.201",
            "ipv6NodeSid" : 111,
            "ipv6Loopback" : "2000::c0a8:0201",
            "routerMac" : "00:00:00:00:02:01",
            "isEdgeRouter" : true,
            "adjacencySids" : []
          },
          "basic" : {
            "name": "Leaf1",
            "managementAddress": "grpc://10.128.100.51:9339?device_id=1",
            "driver": "stratum-tofino",
            "pipeconf": "org.stratumproject.fabric-spgw-int.montara_sde_9_5_0"
          }
        }
      }
    }

- ``device:leaf1``: DPID of the device.

- ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in
  forwarding IPv4 traffic. Can be arbitrary and should be globally unique.

- ``ipv4Loopback``: IPv4 loopback address. Can be arbitrary, should be globally
  unique and should not be part of the same subnet(s) defined on the data plane
  ports (see port config).

- ``ipv6NodeSid``: IPv6 node segment ID, which is used as an MPLS label in
  forwarding IPv6 traffic. Can be arbitrary and should be globally unique. Only
  required when using IPv6.

- ``ipv6Loopback``: IPv6 loopback address. Can be arbitrary, should be globally
  unique and should not be part of the same subnet(s) defined on the data plane
  ports (see port config).  Only required when using IPv6.

- ``routerMac``: Router MAC address. Can be arbitrary and should be globally
  unique.  This MAC address will be used to reply the ARP request for the
  loopback IP or the Interface IP that will be introduced later.  (We recommend
  using the MAC address of the device's management interface as the router
  MAC.)

- ``isEdgeRouter``: True for leaf switches. False for spine switches.

- ``adjacencySids``: Deprecated.  Just put an empty array for now.

- ``name``: Name of the device. It is an arbitrary name to identify the device easily.

- ``managementAddress``: gRPC endpoint of the Stratum device and a numerical device ID.
  The IP address can be replaced by domain name as well.

- ``driver``: ``stratum-bmv2`` or ``stratum-tofino``, depending on which switch this is.

- ``pipeconf``: A list of available pipeconfs can be dumped by running ``pipeconfs`` in ONOS CLI.
  Select the pipeconf you would like to use for this device.

.. caution::
    We should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
    ``ipv6NodeSid``.  Please check here for the reserved values:
    http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml

.. note::
    Most of the SD-Fabric configurations support dynamic configuration updates.
    Unfortunately, SD-Fabric currently **do not support dynamic device
    configuration updates**.  You will have to restart the device when if
    corresponding device configuration changes.

    Having said that, when introducing a completely new device in the network,
    the device configurations pushed before the device's connection should
    apply correctly.

Bridging and Unicast Routing
----------------------------

.. attention::
    - VLAN **4094** is reserved for unconfigured ports (e.g. spine facing ports)

Access Ports
^^^^^^^^^^^^

The necessary but minimum configuration for an access port is simply a VLAN.

.. code-block:: json

    {
      "ports" : {
        "of:0000000000000204/12" : {
          "interfaces" : [{
            "name" : "serverA-intf",
            "vlan-untagged": 10
          }]
        },
        "of:0000000000000204/16" : {
          "interfaces" : [{
            "name" : "serverB-intf",
            "vlan-untagged": 10
          }]
        }
      }
    }

The example above shows two ports (12 and 16) on switch ``of:204`` that have
been assigned to VLAN 10 using the ``vlan-untagged`` keyword.

It simply means that packets come in and leave out of these switches untagged,
but internally they are assigned VLAN 10 and they belong to the bridging domain
defined for VLAN 10.

``name`` is used to associate the interface with a globally unique, user
friendly name. It can be omitted.

With the configuration shown above, the packets will always be bridged, but
they cannot be routed out of the VLAN (e.g. to other subnets).  To add the
capability to route out of VLAN 10, we need to add a subnet/gateway IP (similar
to `interface-vlans or SVIs in traditional networks
<https://www.youtube.com/watch?v=bUXpmiJpGb0>`_).

.. code-block:: json

    {
      "ports" : {
        "of:0000000000000204/12" : {
          "interfaces" : [{
            "name" : "serverA-intf",
            "ips" : [ "10.0.1.254/24"],
            "vlan-untagged": 10
          }]
        },
        "of:0000000000000204/16" : {
          "interfaces" : [{
            "name" : "serverB-intf",
            "ips" : [ "10.0.1.254/24"],
            "vlan-untagged": 10
          }]
        }
      }
    }

In this example, VLAN 10 is associated with subnet ``10.0.1.0/24``, and the
gateway IP for hosts in this subnet is ``10.0.1.254/32``.

When the desire is to route out of a VLAN, this assignment is currently
necessary on all ports configured in the same VLAN.

.. note::
    Typically we only expect a single subnet for a VLAN. Similar to traditional
    networks, for us, a subnet == VLAN. Different VLANs should be configured in
    different subnets.

    In certain use-cases, it may be necessary to configure multiple subnets in
    the same VLAN. This is possible by adding more subnet/gateway IPs in the
    ``ips`` array.

.. tip::
    One subnet cannot be configured on multiple leaf switches.

    We usually configure one subnet for all the ports on the same leaf switch.

Tagged Ports
^^^^^^^^^^^^
Tagged port configuration is similar.

.. code-block:: json

    {
      "ports" : {
        "of:0000000000000204/24" : {
          "interfaces" : [{
            "name" : "serverA-intf",
            "ips" : [ "10.0.2.254/24", "10.0.4.254/24" ],
            "vlan-tagged" : [ 20, 40 ]
          }]
        }
      }
    }

The configuration above for port 24 on switch of:204 shows two VLANs 20 and 40
configured on that port, with corresponding subnets and gateway IPs.

Note that there is no specific ordering required in the ``ips`` or
``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets.

In a future release, we will correlate VLAN and subnets configuration in a more
readable way.

Native VLAN on Tagged Ports
^^^^^^^^^^^^^^^^^^^^^^^^^^^

An additional configuration ``vlan-native`` possible on tagged ports includes
the ability to specify a VLAN (and thus a bridging domain) for incoming
untagged packets.

Typically, such configuration in trunk ports in traditional networks is
referred to a native VLAN.

.. code-block:: json

    {
      "ports" : {
        "of:0000000000000204/24" : {
          "interfaces" : [ {
            "name" : "serverA-intf",
            "ips" : [ "10.0.2.254/24", "10.0.4.254/24", "10.0.1.254/24" ],
            "vlan-tagged" : [ 20, 40 ],
            "vlan-native" : 10
          }]
        }
      }
    }

Note that it is also necessary to configure the subnet/gateway IP corresponding
to the native VLAN if you wish to route out of that VLAN.

Configuring interface for IPv6
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is similar to configure IPv6 routing. Simply replace the addresses in
``ips`` with IPv6 addresses. For example:

.. code-block:: json

    {
      "ports" : {
        "of:0000000000000204/24" : {
          "interfaces" : [ {
            "name" : "serverA-intf",
            "ips" : [ "10.0.2.254/24", "2000::1ff/120" ],
            "vlan-tagged" : [ 20, 40 ]
          }]
        }
      }
    }

.. note::
    There is a known issue that breaks dynamic VLAN configuration.
    Until the issue get resolved, you need to restart the switch agent to reinstall the flows.

IPv6 Router Advertisement
^^^^^^^^^^^^^^^^^^^^^^^^^

Router Advertisement overview
"""""""""""""""""""""""""""""

Router advertisement application is for enabling **Router Advertisement** and
**Router Solicitation** functionalities supported by IPv6 routers.

More details are available in `RFC 4861 <https://tools.ietf.org/html/rfc4861>`_.

Application identifies which IPv6 interfaces are currently configured in the
system and it will try to send out **unsolicited Router Advertisement** (RA)
messages from these interfaces.

Each such RA message will have two mandatory options named **Source link-layer
address** and **MTU**.

Additional RA option **prefix** can be enabled using component configuration
**raGlobalPrefixConfStatus**.

Application also processes **Router Solicitations** (RS) sent from hosts. Upon
receiving RS on a particular interface application stops RA transmission in
that interface and immediately sends RA targeted to the solicited host. After
that application continues unsolicited RA transmission on that interface.

Activate and configure RA
"""""""""""""""""""""""""

RA application can be activated from CLI by running

.. code-block:: console

  onos> app activate routeradvertisement

Behavior of RA application is controlled by ONOS component configuration
subsystem and following are possible configuration options.

- ``raThreadDelay``: Delay between consecutive RA transmissions

- ``raPoolSize``: Capacity of thread pool to be used for RA transmissions

- ``raFlagMbitStatus``: RA flag “Managed address configuration”
  enabled/disabled

- ``raFlagObitStatus``: RA flag “Other configuration” enabled/disabled

- ``raOptionPrefixStatus``: RA Option “prefix” is enabled/disabled. Router
  prefixes will be available in RA only if this flag is “true”

- ``raGlobalPrefixConfStatus``: Enable switch level global prefix
  configuration.
  Once ``raGlobalPrefixConfStatus`` is enabled, RA prefix option is generated
  from port configuration of device, see for more details.

To set the options, following the command (example for ``raOptionPrefixStatus``)

.. code-block:: console

  onos> cfg set org.onosproject.ra.RouterAdvertisementManager raOptionPrefixStatus true

Prefix details are picked up from network interface configuration.

RA app will filter out link-local IPs while preparing prefixes.

For example, in following configuration, Prefix will include only
**2001:0558:FF10:04C9::2:1ff/120**.

.. code-block:: json

    {
      "ports": {
        "of:0000000000000018/16": {
          "interfaces": [{
            "ips": [ "192.168.114.1/24", "2001:0558:FF10:04C9::2:1ff/120", "FE80::4EA8:2AFF:FE24:8E5F/120" ],
            "vlan-untagged": "11",
            "name": "18-15"
          }]
        }
      }
    }

Global prefix configuration
"""""""""""""""""""""""""""

In some cases, users may want to have a set of global prefix **advertised on
all edge interfaces**.

Such prefixes can be configured in **devices** section of network configuration
in the following way.

.. code-block:: json

    {
      "devices": {
        "of:0000000000000018": {
          "routeradvertisement" : {
            "prefixes": [ "2001:0558:FF10:04C9::3:1ff/120"]
          }
        }
      }
    }

.. note::
    When global prefix is configured, RA app will ignore any prefixes
    configured on switch interfaces.

Notes about interface config
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There is no need to configure ports on switches that are meant to connect to
other switches.

The VLAN (untagged or tagged) configuration is only meant for ports that are
connected to hosts (edge ports).

.. image:: ../images/config-vlan.png

Furthermore, note that the same VLAN can be configured on multiple ToRs - e.g.
VLAN 20 in the figure above.

However this does not mean that the ports are in the same bridging domain,
because in the fabric, the communication between ToRs is through a routed
network.

In other words, a host on VLAN 20 (untagged or tagged) connected to one ToR can
communicate with another host on VLAN 20 (untagged or tagged) connected to a
different ToR, but the MAC addresses will change as the traffic goes through a
routed network.

Please do not use this feature to connect switches in unsupported topologies as
shown in the example below.

The fabric is not designed to be one big Ethernet fabric. The bridging domain
is restricted to within one ToR.

If the bridging domain is extended across two ToRs directly linked to each
other, there is a chance of loops.

In other words, the ToRs/Leafs are not standalone 802.1Q bridges, and should
not be used as such.

.. image:: ../images/config-vlan-invalid.png