blob: 44f46006693ffdfe7043fdd2241f205970718690 [file] [log] [blame]
Charles Chanfcfe8902022-02-02 17:06:27 -08001.. SPDX-FileCopyrightText: 2021 Open Networking Foundation <info@opennetworking.org>
2.. SPDX-License-Identifier: Apache-2.0
3
Charles Chan2caff7b2021-10-11 20:25:16 -07004.. _onos_network_config:
5
Charles Chancaebcf32021-09-20 22:17:52 -07006Network Configuration
7=====================
Wailok Shum2f05cd32021-09-30 22:18:19 +08008SD-Fabric uses several different types of network configurations.
Carmelo Cascone43989982021-10-12 00:01:19 -07009We only focus on ``devices`` and ``ports`` configuration in this section, which
10are used to provide basic L2/L3 connectivity.
Wailok Shum2f05cd32021-09-30 22:18:19 +080011
12See :ref:`advanced-features` for advanced feature configurations.
13
14Device Configuration
15--------------------
16Each switch in SD-Fabric requires a device config.
17
18.. code-block:: json
19
20 {
21 "devices" : {
22 "device:leaf1" : {
23 "segmentrouting" : {
24 "ipv4NodeSid" : 101,
25 "ipv4Loopback" : "192.168.0.201",
26 "ipv6NodeSid" : 111,
27 "ipv6Loopback" : "2000::c0a8:0201",
28 "routerMac" : "00:00:00:00:02:01",
29 "isEdgeRouter" : true,
30 "adjacencySids" : []
31 },
32 "basic" : {
33 "name": "Leaf1",
34 "managementAddress": "grpc://10.128.100.51:9339?device_id=1",
35 "driver": "stratum-tofino",
Carmelo Casconef2b17912022-02-04 09:27:27 -080036 "pipeconf": "org.stratumproject.fabric-upf-int.montara_sde_9_5_0"
Wailok Shum2f05cd32021-09-30 22:18:19 +080037 }
38 }
39 }
40 }
41
Carmelo Cascone43989982021-10-12 00:01:19 -070042- ``device:leaf1``: arbitrary globally unique identifier of the device, must always be prefixed with ``device:``.
Wailok Shum2f05cd32021-09-30 22:18:19 +080043
44- ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in
45 forwarding IPv4 traffic. Can be arbitrary and should be globally unique.
46
47- ``ipv4Loopback``: IPv4 loopback address. Can be arbitrary, should be globally
48 unique and should not be part of the same subnet(s) defined on the data plane
49 ports (see port config).
50
51- ``ipv6NodeSid``: IPv6 node segment ID, which is used as an MPLS label in
52 forwarding IPv6 traffic. Can be arbitrary and should be globally unique. Only
53 required when using IPv6.
54
55- ``ipv6Loopback``: IPv6 loopback address. Can be arbitrary, should be globally
56 unique and should not be part of the same subnet(s) defined on the data plane
57 ports (see port config). Only required when using IPv6.
58
59- ``routerMac``: Router MAC address. Can be arbitrary and should be globally
60 unique. This MAC address will be used to reply the ARP request for the
61 loopback IP or the Interface IP that will be introduced later. (We recommend
62 using the MAC address of the device's management interface as the router
63 MAC.)
64
65- ``isEdgeRouter``: True for leaf switches. False for spine switches.
66
Carmelo Cascone43989982021-10-12 00:01:19 -070067- ``adjacencySids``: Deprecated. Always use an empty array.
Wailok Shum2f05cd32021-09-30 22:18:19 +080068
Carmelo Cascone43989982021-10-12 00:01:19 -070069- ``name``: Human friendly name used in the ONOS UI.
Wailok Shum2f05cd32021-09-30 22:18:19 +080070
Carmelo Cascone43989982021-10-12 00:01:19 -070071- ``managementAddress``: gRPC endpoint of the Stratum device and the P4Runtime
72 internal device ID associated to the ASIC (usually ``1``),
73 in the format of ``grpc://[device_addr]?device_id=[P4Runtime device ID]``
74 The IP address can be replaced by a domain name.
Wailok Shum2f05cd32021-09-30 22:18:19 +080075
76- ``driver``: ``stratum-bmv2`` or ``stratum-tofino``, depending on which switch this is.
77
Carmelo Cascone43989982021-10-12 00:01:19 -070078- ``pipeconf``: the P4 program to deploy on this switch. A list of available
79 pipeconfs can be dumped by running ``pipeconfs`` in the ONOS CLI. When running
80 with Tofino-based devices, we provide pre-installed pipeconfs with ID
81 ``org.stratumproject.<profile>.<device-type>_<bf-sde-version>``:
82
83 - The available *profiles* are:
84
85 - ``fabric``: for basic L2/L3 capabilities
Carmelo Casconef2b17912022-02-04 09:27:27 -080086 - ``fabric-upf``: with 4G/5G mobile user plane (UPF) support
Carmelo Cascone43989982021-10-12 00:01:19 -070087 - ``fabric-int``: with INT support
Carmelo Casconef2b17912022-02-04 09:27:27 -080088 - ``fabric-upf-int``: with UPF and INT support
Carmelo Cascone43989982021-10-12 00:01:19 -070089
Hung-Wei Chiu554a9d02021-11-22 14:10:19 -080090 - The supported *device-types*: Use the command ``weutil`` in the BMC mode to get the product name.
Carmelo Cascone43989982021-10-12 00:01:19 -070091
Hung-Wei Chiu554a9d02021-11-22 14:10:19 -080092 - ``montara``: for dual-pipe Tofino ASIC SKUs (Product name is 100BF-32X)
93 - ``mavericks``: for quad-pipe Tofino ASIC SKUs (Product name is 100BF-32QS)
Carmelo Cascone43989982021-10-12 00:01:19 -070094
95 - The Intel/Barefoot SDE version used in Stratum, e.g., ``sde_9_5_0``
Wailok Shum2f05cd32021-09-30 22:18:19 +080096
97.. caution::
Carmelo Cascone43989982021-10-12 00:01:19 -070098 You should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
Wailok Shum2f05cd32021-09-30 22:18:19 +080099 ``ipv6NodeSid``. Please check here for the reserved values:
100 http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml
101
102.. note::
Carmelo Cascone43989982021-10-12 00:01:19 -0700103 Most of the SD-Fabric configurations support dynamic updates. Unfortunately,
104 we currently **do not support dynamic device configuration updates**. You
105 will have to restart (reboot) the switch if the corresponding device
106 configuration changes.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800107
108 Having said that, when introducing a completely new device in the network,
Carmelo Cascone43989982021-10-12 00:01:19 -0700109 the device configurations pushed before ONOS connects to the switch for the
110 first time should be applied correctly.
111
112Port IDs for Tofino-based devices
113---------------------------------
114
115Before describing the ONOS netcfg, it is worth nothing how we refer to ports for
Carmelo Cascone46d63032022-03-07 00:22:30 -0800116Tofino-based devices. netcfg uses the format ``device:<name>/<port-number>``.
Carmelo Cascone43989982021-10-12 00:01:19 -0700117
Charles Chana937f772022-02-23 16:24:35 -0800118``<port-number>`` is a unique, arbitrary value that should be consistent
119with the ``id`` field defined in Stratum chassis config.
120See :ref:`singleton_port` for more details.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800121
122Bridging and Unicast Routing
123----------------------------
124
Carmelo Cascone43989982021-10-12 00:01:19 -0700125In the following we illustrate how to enable basic bridging and routing on a
126per-port basis.
127
Wailok Shum2f05cd32021-09-30 22:18:19 +0800128.. attention::
129 - VLAN **4094** is reserved for unconfigured ports (e.g. spine facing ports)
Carmelo Cascone43989982021-10-12 00:01:19 -0700130 - VLAN **4090** is reserved for pseudowire transport flow rules on the spines
Wailok Shum2f05cd32021-09-30 22:18:19 +0800131
132Access Ports
133^^^^^^^^^^^^
134
135The necessary but minimum configuration for an access port is simply a VLAN.
136
137.. code-block:: json
138
139 {
140 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700141 "device:leaf1/12" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800142 "interfaces" : [{
143 "name" : "serverA-intf",
144 "vlan-untagged": 10
145 }]
146 },
Carmelo Cascone43989982021-10-12 00:01:19 -0700147 "device:leaf1/16" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800148 "interfaces" : [{
149 "name" : "serverB-intf",
150 "vlan-untagged": 10
151 }]
152 }
153 }
154 }
155
Carmelo Cascone43989982021-10-12 00:01:19 -0700156The example above shows two ports (12 and 16) on switch ``leaf1`` that have
Wailok Shum2f05cd32021-09-30 22:18:19 +0800157been assigned to VLAN 10 using the ``vlan-untagged`` keyword.
158
159It simply means that packets come in and leave out of these switches untagged,
160but internally they are assigned VLAN 10 and they belong to the bridging domain
161defined for VLAN 10.
162
163``name`` is used to associate the interface with a globally unique, user
164friendly name. It can be omitted.
165
166With the configuration shown above, the packets will always be bridged, but
167they cannot be routed out of the VLAN (e.g. to other subnets). To add the
168capability to route out of VLAN 10, we need to add a subnet/gateway IP (similar
169to `interface-vlans or SVIs in traditional networks
170<https://www.youtube.com/watch?v=bUXpmiJpGb0>`_).
171
172.. code-block:: json
173
174 {
175 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700176 "device:leaf1/12" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800177 "interfaces" : [{
178 "name" : "serverA-intf",
179 "ips" : [ "10.0.1.254/24"],
180 "vlan-untagged": 10
181 }]
182 },
Carmelo Cascone43989982021-10-12 00:01:19 -0700183 "device:leaf1/16" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800184 "interfaces" : [{
185 "name" : "serverB-intf",
186 "ips" : [ "10.0.1.254/24"],
187 "vlan-untagged": 10
188 }]
189 }
190 }
191 }
192
193In this example, VLAN 10 is associated with subnet ``10.0.1.0/24``, and the
194gateway IP for hosts in this subnet is ``10.0.1.254/32``.
195
196When the desire is to route out of a VLAN, this assignment is currently
197necessary on all ports configured in the same VLAN.
198
199.. note::
200 Typically we only expect a single subnet for a VLAN. Similar to traditional
201 networks, for us, a subnet == VLAN. Different VLANs should be configured in
202 different subnets.
203
204 In certain use-cases, it may be necessary to configure multiple subnets in
205 the same VLAN. This is possible by adding more subnet/gateway IPs in the
206 ``ips`` array.
207
Carmelo Cascone43989982021-10-12 00:01:19 -0700208.. attention::
209 The same subnet cannot be configured on multiple leaf switches.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800210
211 We usually configure one subnet for all the ports on the same leaf switch.
212
213Tagged Ports
214^^^^^^^^^^^^
215Tagged port configuration is similar.
216
217.. code-block:: json
218
219 {
220 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700221 "device:leaf1/24" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800222 "interfaces" : [{
223 "name" : "serverA-intf",
224 "ips" : [ "10.0.2.254/24", "10.0.4.254/24" ],
225 "vlan-tagged" : [ 20, 40 ]
226 }]
227 }
228 }
229 }
230
Carmelo Cascone43989982021-10-12 00:01:19 -0700231The configuration above for port 24 on switch ``leaf1`` shows two VLANs 20 and
23240 configured on that port, with corresponding subnets and gateway IPs.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800233
234Note that there is no specific ordering required in the ``ips`` or
235``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets.
236
237In a future release, we will correlate VLAN and subnets configuration in a more
238readable way.
239
240Native VLAN on Tagged Ports
241^^^^^^^^^^^^^^^^^^^^^^^^^^^
242
243An additional configuration ``vlan-native`` possible on tagged ports includes
244the ability to specify a VLAN (and thus a bridging domain) for incoming
245untagged packets.
246
247Typically, such configuration in trunk ports in traditional networks is
248referred to a native VLAN.
249
250.. code-block:: json
251
252 {
253 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700254 "device:leaf1/24" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800255 "interfaces" : [ {
256 "name" : "serverA-intf",
257 "ips" : [ "10.0.2.254/24", "10.0.4.254/24", "10.0.1.254/24" ],
258 "vlan-tagged" : [ 20, 40 ],
259 "vlan-native" : 10
260 }]
261 }
262 }
263 }
264
265Note that it is also necessary to configure the subnet/gateway IP corresponding
266to the native VLAN if you wish to route out of that VLAN.
267
Carmelo Cascone43989982021-10-12 00:01:19 -0700268Configuring interface for IPv6 [#f1]_
269^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Wailok Shum2f05cd32021-09-30 22:18:19 +0800270
271It is similar to configure IPv6 routing. Simply replace the addresses in
272``ips`` with IPv6 addresses. For example:
273
274.. code-block:: json
275
276 {
277 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700278 "device:leaf1/24" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800279 "interfaces" : [ {
280 "name" : "serverA-intf",
281 "ips" : [ "10.0.2.254/24", "2000::1ff/120" ],
282 "vlan-tagged" : [ 20, 40 ]
283 }]
284 }
285 }
286 }
287
288.. note::
289 There is a known issue that breaks dynamic VLAN configuration.
290 Until the issue get resolved, you need to restart the switch agent to reinstall the flows.
291
Carmelo Cascone43989982021-10-12 00:01:19 -0700292IPv6 Router Advertisement [#f1]_
293^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Wailok Shum2f05cd32021-09-30 22:18:19 +0800294
295Router Advertisement overview
296"""""""""""""""""""""""""""""
297
298Router advertisement application is for enabling **Router Advertisement** and
299**Router Solicitation** functionalities supported by IPv6 routers.
300
301More details are available in `RFC 4861 <https://tools.ietf.org/html/rfc4861>`_.
302
303Application identifies which IPv6 interfaces are currently configured in the
304system and it will try to send out **unsolicited Router Advertisement** (RA)
305messages from these interfaces.
306
307Each such RA message will have two mandatory options named **Source link-layer
308address** and **MTU**.
309
310Additional RA option **prefix** can be enabled using component configuration
311**raGlobalPrefixConfStatus**.
312
313Application also processes **Router Solicitations** (RS) sent from hosts. Upon
314receiving RS on a particular interface application stops RA transmission in
315that interface and immediately sends RA targeted to the solicited host. After
316that application continues unsolicited RA transmission on that interface.
317
318Activate and configure RA
319"""""""""""""""""""""""""
320
321RA application can be activated from CLI by running
322
323.. code-block:: console
324
325 onos> app activate routeradvertisement
326
327Behavior of RA application is controlled by ONOS component configuration
328subsystem and following are possible configuration options.
329
330- ``raThreadDelay``: Delay between consecutive RA transmissions
331
332- ``raPoolSize``: Capacity of thread pool to be used for RA transmissions
333
334- ``raFlagMbitStatus``: RA flag “Managed address configuration”
335 enabled/disabled
336
337- ``raFlagObitStatus``: RA flag “Other configuration” enabled/disabled
338
339- ``raOptionPrefixStatus``: RA Option “prefix” is enabled/disabled. Router
340 prefixes will be available in RA only if this flag is “true”
341
342- ``raGlobalPrefixConfStatus``: Enable switch level global prefix
343 configuration.
344 Once ``raGlobalPrefixConfStatus`` is enabled, RA prefix option is generated
345 from port configuration of device, see for more details.
346
347To set the options, following the command (example for ``raOptionPrefixStatus``)
348
349.. code-block:: console
350
351 onos> cfg set org.onosproject.ra.RouterAdvertisementManager raOptionPrefixStatus true
352
353Prefix details are picked up from network interface configuration.
354
355RA app will filter out link-local IPs while preparing prefixes.
356
357For example, in following configuration, Prefix will include only
358**2001:0558:FF10:04C9::2:1ff/120**.
359
360.. code-block:: json
361
362 {
363 "ports": {
Carmelo Cascone43989982021-10-12 00:01:19 -0700364 "device:leaf2/16": {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800365 "interfaces": [{
366 "ips": [ "192.168.114.1/24", "2001:0558:FF10:04C9::2:1ff/120", "FE80::4EA8:2AFF:FE24:8E5F/120" ],
367 "vlan-untagged": "11",
368 "name": "18-15"
369 }]
370 }
371 }
372 }
373
374Global prefix configuration
375"""""""""""""""""""""""""""
376
377In some cases, users may want to have a set of global prefix **advertised on
378all edge interfaces**.
379
380Such prefixes can be configured in **devices** section of network configuration
381in the following way.
382
383.. code-block:: json
384
385 {
386 "devices": {
Carmelo Cascone43989982021-10-12 00:01:19 -0700387 "device:leaf2": {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800388 "routeradvertisement" : {
389 "prefixes": [ "2001:0558:FF10:04C9::3:1ff/120"]
390 }
391 }
392 }
393 }
394
395.. note::
396 When global prefix is configured, RA app will ignore any prefixes
397 configured on switch interfaces.
398
399Notes about interface config
400^^^^^^^^^^^^^^^^^^^^^^^^^^^^
401
402There is no need to configure ports on switches that are meant to connect to
403other switches.
404
405The VLAN (untagged or tagged) configuration is only meant for ports that are
406connected to hosts (edge ports).
407
408.. image:: ../images/config-vlan.png
409
410Furthermore, note that the same VLAN can be configured on multiple ToRs - e.g.
411VLAN 20 in the figure above.
412
413However this does not mean that the ports are in the same bridging domain,
414because in the fabric, the communication between ToRs is through a routed
415network.
416
417In other words, a host on VLAN 20 (untagged or tagged) connected to one ToR can
418communicate with another host on VLAN 20 (untagged or tagged) connected to a
419different ToR, but the MAC addresses will change as the traffic goes through a
420routed network.
421
422Please do not use this feature to connect switches in unsupported topologies as
423shown in the example below.
424
425The fabric is not designed to be one big Ethernet fabric. The bridging domain
Carmelo Cascone43989982021-10-12 00:01:19 -0700426is restricted to one ToR.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800427
428If the bridging domain is extended across two ToRs directly linked to each
429other, there is a chance of loops.
430
431In other words, the ToRs/Leafs are not standalone 802.1Q bridges, and should
432not be used as such.
433
434.. image:: ../images/config-vlan-invalid.png
Carmelo Cascone43989982021-10-12 00:01:19 -0700435
436.. rubric:: Footnotes
437
438.. [#f1] IPv6 support on the data plane (P4 program) is still work-in-progress.