blob: 5bf70aac8f343e7f1bfd9ddfd8e3cea5925b74df [file] [log] [blame]
Charles Chan2caff7b2021-10-11 20:25:16 -07001.. _onos_network_config:
2
Charles Chancaebcf32021-09-20 22:17:52 -07003Network Configuration
4=====================
Wailok Shum2f05cd32021-09-30 22:18:19 +08005SD-Fabric uses several different types of network configurations.
Carmelo Cascone43989982021-10-12 00:01:19 -07006We only focus on ``devices`` and ``ports`` configuration in this section, which
7are used to provide basic L2/L3 connectivity.
Wailok Shum2f05cd32021-09-30 22:18:19 +08008
9See :ref:`advanced-features` for advanced feature configurations.
10
11Device Configuration
12--------------------
13Each switch in SD-Fabric requires a device config.
14
15.. code-block:: json
16
17 {
18 "devices" : {
19 "device:leaf1" : {
20 "segmentrouting" : {
21 "ipv4NodeSid" : 101,
22 "ipv4Loopback" : "192.168.0.201",
23 "ipv6NodeSid" : 111,
24 "ipv6Loopback" : "2000::c0a8:0201",
25 "routerMac" : "00:00:00:00:02:01",
26 "isEdgeRouter" : true,
27 "adjacencySids" : []
28 },
29 "basic" : {
30 "name": "Leaf1",
31 "managementAddress": "grpc://10.128.100.51:9339?device_id=1",
32 "driver": "stratum-tofino",
33 "pipeconf": "org.stratumproject.fabric-spgw-int.montara_sde_9_5_0"
34 }
35 }
36 }
37 }
38
Carmelo Cascone43989982021-10-12 00:01:19 -070039- ``device:leaf1``: arbitrary globally unique identifier of the device, must always be prefixed with ``device:``.
Wailok Shum2f05cd32021-09-30 22:18:19 +080040
41- ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in
42 forwarding IPv4 traffic. Can be arbitrary and should be globally unique.
43
44- ``ipv4Loopback``: IPv4 loopback address. Can be arbitrary, should be globally
45 unique and should not be part of the same subnet(s) defined on the data plane
46 ports (see port config).
47
48- ``ipv6NodeSid``: IPv6 node segment ID, which is used as an MPLS label in
49 forwarding IPv6 traffic. Can be arbitrary and should be globally unique. Only
50 required when using IPv6.
51
52- ``ipv6Loopback``: IPv6 loopback address. Can be arbitrary, should be globally
53 unique and should not be part of the same subnet(s) defined on the data plane
54 ports (see port config). Only required when using IPv6.
55
56- ``routerMac``: Router MAC address. Can be arbitrary and should be globally
57 unique. This MAC address will be used to reply the ARP request for the
58 loopback IP or the Interface IP that will be introduced later. (We recommend
59 using the MAC address of the device's management interface as the router
60 MAC.)
61
62- ``isEdgeRouter``: True for leaf switches. False for spine switches.
63
Carmelo Cascone43989982021-10-12 00:01:19 -070064- ``adjacencySids``: Deprecated. Always use an empty array.
Wailok Shum2f05cd32021-09-30 22:18:19 +080065
Carmelo Cascone43989982021-10-12 00:01:19 -070066- ``name``: Human friendly name used in the ONOS UI.
Wailok Shum2f05cd32021-09-30 22:18:19 +080067
Carmelo Cascone43989982021-10-12 00:01:19 -070068- ``managementAddress``: gRPC endpoint of the Stratum device and the P4Runtime
69 internal device ID associated to the ASIC (usually ``1``),
70 in the format of ``grpc://[device_addr]?device_id=[P4Runtime device ID]``
71 The IP address can be replaced by a domain name.
Wailok Shum2f05cd32021-09-30 22:18:19 +080072
73- ``driver``: ``stratum-bmv2`` or ``stratum-tofino``, depending on which switch this is.
74
Carmelo Cascone43989982021-10-12 00:01:19 -070075- ``pipeconf``: the P4 program to deploy on this switch. A list of available
76 pipeconfs can be dumped by running ``pipeconfs`` in the ONOS CLI. When running
77 with Tofino-based devices, we provide pre-installed pipeconfs with ID
78 ``org.stratumproject.<profile>.<device-type>_<bf-sde-version>``:
79
80 - The available *profiles* are:
81
82 - ``fabric``: for basic L2/L3 capabilities
83 - ``fabric-spgw``: with 4G/5G mobile user plane support
84 - ``fabric-int``: with INT support
85 - ``fabric-spgw-int``: with SPGW and INT support
86
87 - The supported *device-types*:
88
89 - ``montara``: for dual-pipe Tofino ASIC SKUs
90 - ``mavericks``: for quad-pipe Tofino ASIC SKUs
91
92 - The Intel/Barefoot SDE version used in Stratum, e.g., ``sde_9_5_0``
Wailok Shum2f05cd32021-09-30 22:18:19 +080093
94.. caution::
Carmelo Cascone43989982021-10-12 00:01:19 -070095 You should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
Wailok Shum2f05cd32021-09-30 22:18:19 +080096 ``ipv6NodeSid``. Please check here for the reserved values:
97 http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml
98
99.. note::
Carmelo Cascone43989982021-10-12 00:01:19 -0700100 Most of the SD-Fabric configurations support dynamic updates. Unfortunately,
101 we currently **do not support dynamic device configuration updates**. You
102 will have to restart (reboot) the switch if the corresponding device
103 configuration changes.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800104
105 Having said that, when introducing a completely new device in the network,
Carmelo Cascone43989982021-10-12 00:01:19 -0700106 the device configurations pushed before ONOS connects to the switch for the
107 first time should be applied correctly.
108
109Port IDs for Tofino-based devices
110---------------------------------
111
112Before describing the ONOS netcfg, it is worth nothing how we refer to ports for
113Tofino-based devices. Netcfg uses the format ``device:<name>/<port-number>``.
114
115``<port-number>`` is a special value that is usually different than the number
116shown in the switch front panel. It is the same number used for P4 table
117programming and depends on the specific Tofino ASIC SKU (e.g., dual-pipe vs.
118quad-pipe) and switch vendor/platform. In Stratum this is often referred to as
119the *SDK port ID*, as this is the number used for all Tofino SDK calls. In Intel
120documentation this is referred to as the ``DP_ID``. We plan to remove this
121dependency on such a low level detail in future releases, but for now, to find
122out the mapping between front-panel ports and ``DP_ID`` you have the following
123options:
124
125- Ask your switch vendor
126- Use the command `pm.show` on the BF shell of a running Stratum instance
127 (see :ref:`troubleshooting_guide`)
Wailok Shum2f05cd32021-09-30 22:18:19 +0800128
129Bridging and Unicast Routing
130----------------------------
131
Carmelo Cascone43989982021-10-12 00:01:19 -0700132In the following we illustrate how to enable basic bridging and routing on a
133per-port basis.
134
Wailok Shum2f05cd32021-09-30 22:18:19 +0800135.. attention::
136 - VLAN **4094** is reserved for unconfigured ports (e.g. spine facing ports)
Carmelo Cascone43989982021-10-12 00:01:19 -0700137 - VLAN **4090** is reserved for pseudowire transport flow rules on the spines
Wailok Shum2f05cd32021-09-30 22:18:19 +0800138
139Access Ports
140^^^^^^^^^^^^
141
142The necessary but minimum configuration for an access port is simply a VLAN.
143
144.. code-block:: json
145
146 {
147 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700148 "device:leaf1/12" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800149 "interfaces" : [{
150 "name" : "serverA-intf",
151 "vlan-untagged": 10
152 }]
153 },
Carmelo Cascone43989982021-10-12 00:01:19 -0700154 "device:leaf1/16" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800155 "interfaces" : [{
156 "name" : "serverB-intf",
157 "vlan-untagged": 10
158 }]
159 }
160 }
161 }
162
Carmelo Cascone43989982021-10-12 00:01:19 -0700163The example above shows two ports (12 and 16) on switch ``leaf1`` that have
Wailok Shum2f05cd32021-09-30 22:18:19 +0800164been assigned to VLAN 10 using the ``vlan-untagged`` keyword.
165
166It simply means that packets come in and leave out of these switches untagged,
167but internally they are assigned VLAN 10 and they belong to the bridging domain
168defined for VLAN 10.
169
170``name`` is used to associate the interface with a globally unique, user
171friendly name. It can be omitted.
172
173With the configuration shown above, the packets will always be bridged, but
174they cannot be routed out of the VLAN (e.g. to other subnets). To add the
175capability to route out of VLAN 10, we need to add a subnet/gateway IP (similar
176to `interface-vlans or SVIs in traditional networks
177<https://www.youtube.com/watch?v=bUXpmiJpGb0>`_).
178
179.. code-block:: json
180
181 {
182 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700183 "device:leaf1/12" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800184 "interfaces" : [{
185 "name" : "serverA-intf",
186 "ips" : [ "10.0.1.254/24"],
187 "vlan-untagged": 10
188 }]
189 },
Carmelo Cascone43989982021-10-12 00:01:19 -0700190 "device:leaf1/16" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800191 "interfaces" : [{
192 "name" : "serverB-intf",
193 "ips" : [ "10.0.1.254/24"],
194 "vlan-untagged": 10
195 }]
196 }
197 }
198 }
199
200In this example, VLAN 10 is associated with subnet ``10.0.1.0/24``, and the
201gateway IP for hosts in this subnet is ``10.0.1.254/32``.
202
203When the desire is to route out of a VLAN, this assignment is currently
204necessary on all ports configured in the same VLAN.
205
206.. note::
207 Typically we only expect a single subnet for a VLAN. Similar to traditional
208 networks, for us, a subnet == VLAN. Different VLANs should be configured in
209 different subnets.
210
211 In certain use-cases, it may be necessary to configure multiple subnets in
212 the same VLAN. This is possible by adding more subnet/gateway IPs in the
213 ``ips`` array.
214
Carmelo Cascone43989982021-10-12 00:01:19 -0700215.. attention::
216 The same subnet cannot be configured on multiple leaf switches.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800217
218 We usually configure one subnet for all the ports on the same leaf switch.
219
220Tagged Ports
221^^^^^^^^^^^^
222Tagged port configuration is similar.
223
224.. code-block:: json
225
226 {
227 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700228 "device:leaf1/24" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800229 "interfaces" : [{
230 "name" : "serverA-intf",
231 "ips" : [ "10.0.2.254/24", "10.0.4.254/24" ],
232 "vlan-tagged" : [ 20, 40 ]
233 }]
234 }
235 }
236 }
237
Carmelo Cascone43989982021-10-12 00:01:19 -0700238The configuration above for port 24 on switch ``leaf1`` shows two VLANs 20 and
23940 configured on that port, with corresponding subnets and gateway IPs.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800240
241Note that there is no specific ordering required in the ``ips`` or
242``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets.
243
244In a future release, we will correlate VLAN and subnets configuration in a more
245readable way.
246
247Native VLAN on Tagged Ports
248^^^^^^^^^^^^^^^^^^^^^^^^^^^
249
250An additional configuration ``vlan-native`` possible on tagged ports includes
251the ability to specify a VLAN (and thus a bridging domain) for incoming
252untagged packets.
253
254Typically, such configuration in trunk ports in traditional networks is
255referred to a native VLAN.
256
257.. code-block:: json
258
259 {
260 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700261 "device:leaf1/24" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800262 "interfaces" : [ {
263 "name" : "serverA-intf",
264 "ips" : [ "10.0.2.254/24", "10.0.4.254/24", "10.0.1.254/24" ],
265 "vlan-tagged" : [ 20, 40 ],
266 "vlan-native" : 10
267 }]
268 }
269 }
270 }
271
272Note that it is also necessary to configure the subnet/gateway IP corresponding
273to the native VLAN if you wish to route out of that VLAN.
274
Carmelo Cascone43989982021-10-12 00:01:19 -0700275Configuring interface for IPv6 [#f1]_
276^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Wailok Shum2f05cd32021-09-30 22:18:19 +0800277
278It is similar to configure IPv6 routing. Simply replace the addresses in
279``ips`` with IPv6 addresses. For example:
280
281.. code-block:: json
282
283 {
284 "ports" : {
Carmelo Cascone43989982021-10-12 00:01:19 -0700285 "device:leaf1/24" : {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800286 "interfaces" : [ {
287 "name" : "serverA-intf",
288 "ips" : [ "10.0.2.254/24", "2000::1ff/120" ],
289 "vlan-tagged" : [ 20, 40 ]
290 }]
291 }
292 }
293 }
294
295.. note::
296 There is a known issue that breaks dynamic VLAN configuration.
297 Until the issue get resolved, you need to restart the switch agent to reinstall the flows.
298
Carmelo Cascone43989982021-10-12 00:01:19 -0700299IPv6 Router Advertisement [#f1]_
300^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Wailok Shum2f05cd32021-09-30 22:18:19 +0800301
302Router Advertisement overview
303"""""""""""""""""""""""""""""
304
305Router advertisement application is for enabling **Router Advertisement** and
306**Router Solicitation** functionalities supported by IPv6 routers.
307
308More details are available in `RFC 4861 <https://tools.ietf.org/html/rfc4861>`_.
309
310Application identifies which IPv6 interfaces are currently configured in the
311system and it will try to send out **unsolicited Router Advertisement** (RA)
312messages from these interfaces.
313
314Each such RA message will have two mandatory options named **Source link-layer
315address** and **MTU**.
316
317Additional RA option **prefix** can be enabled using component configuration
318**raGlobalPrefixConfStatus**.
319
320Application also processes **Router Solicitations** (RS) sent from hosts. Upon
321receiving RS on a particular interface application stops RA transmission in
322that interface and immediately sends RA targeted to the solicited host. After
323that application continues unsolicited RA transmission on that interface.
324
325Activate and configure RA
326"""""""""""""""""""""""""
327
328RA application can be activated from CLI by running
329
330.. code-block:: console
331
332 onos> app activate routeradvertisement
333
334Behavior of RA application is controlled by ONOS component configuration
335subsystem and following are possible configuration options.
336
337- ``raThreadDelay``: Delay between consecutive RA transmissions
338
339- ``raPoolSize``: Capacity of thread pool to be used for RA transmissions
340
341- ``raFlagMbitStatus``: RA flag “Managed address configuration”
342 enabled/disabled
343
344- ``raFlagObitStatus``: RA flag “Other configuration” enabled/disabled
345
346- ``raOptionPrefixStatus``: RA Option “prefix” is enabled/disabled. Router
347 prefixes will be available in RA only if this flag is “true”
348
349- ``raGlobalPrefixConfStatus``: Enable switch level global prefix
350 configuration.
351 Once ``raGlobalPrefixConfStatus`` is enabled, RA prefix option is generated
352 from port configuration of device, see for more details.
353
354To set the options, following the command (example for ``raOptionPrefixStatus``)
355
356.. code-block:: console
357
358 onos> cfg set org.onosproject.ra.RouterAdvertisementManager raOptionPrefixStatus true
359
360Prefix details are picked up from network interface configuration.
361
362RA app will filter out link-local IPs while preparing prefixes.
363
364For example, in following configuration, Prefix will include only
365**2001:0558:FF10:04C9::2:1ff/120**.
366
367.. code-block:: json
368
369 {
370 "ports": {
Carmelo Cascone43989982021-10-12 00:01:19 -0700371 "device:leaf2/16": {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800372 "interfaces": [{
373 "ips": [ "192.168.114.1/24", "2001:0558:FF10:04C9::2:1ff/120", "FE80::4EA8:2AFF:FE24:8E5F/120" ],
374 "vlan-untagged": "11",
375 "name": "18-15"
376 }]
377 }
378 }
379 }
380
381Global prefix configuration
382"""""""""""""""""""""""""""
383
384In some cases, users may want to have a set of global prefix **advertised on
385all edge interfaces**.
386
387Such prefixes can be configured in **devices** section of network configuration
388in the following way.
389
390.. code-block:: json
391
392 {
393 "devices": {
Carmelo Cascone43989982021-10-12 00:01:19 -0700394 "device:leaf2": {
Wailok Shum2f05cd32021-09-30 22:18:19 +0800395 "routeradvertisement" : {
396 "prefixes": [ "2001:0558:FF10:04C9::3:1ff/120"]
397 }
398 }
399 }
400 }
401
402.. note::
403 When global prefix is configured, RA app will ignore any prefixes
404 configured on switch interfaces.
405
406Notes about interface config
407^^^^^^^^^^^^^^^^^^^^^^^^^^^^
408
409There is no need to configure ports on switches that are meant to connect to
410other switches.
411
412The VLAN (untagged or tagged) configuration is only meant for ports that are
413connected to hosts (edge ports).
414
415.. image:: ../images/config-vlan.png
416
417Furthermore, note that the same VLAN can be configured on multiple ToRs - e.g.
418VLAN 20 in the figure above.
419
420However this does not mean that the ports are in the same bridging domain,
421because in the fabric, the communication between ToRs is through a routed
422network.
423
424In other words, a host on VLAN 20 (untagged or tagged) connected to one ToR can
425communicate with another host on VLAN 20 (untagged or tagged) connected to a
426different ToR, but the MAC addresses will change as the traffic goes through a
427routed network.
428
429Please do not use this feature to connect switches in unsupported topologies as
430shown in the example below.
431
432The fabric is not designed to be one big Ethernet fabric. The bridging domain
Carmelo Cascone43989982021-10-12 00:01:19 -0700433is restricted to one ToR.
Wailok Shum2f05cd32021-09-30 22:18:19 +0800434
435If the bridging domain is extended across two ToRs directly linked to each
436other, there is a chance of loops.
437
438In other words, the ToRs/Leafs are not standalone 802.1Q bridges, and should
439not be used as such.
440
441.. image:: ../images/config-vlan-invalid.png
Carmelo Cascone43989982021-10-12 00:01:19 -0700442
443.. rubric:: Footnotes
444
445.. [#f1] IPv6 support on the data plane (P4 program) is still work-in-progress.