Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
| 5 | BESS UPF Deployment |
| 6 | =================== |
| 7 | |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 8 | This section describes how to configure and deploy BESS UPF. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 9 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 10 | Network Configuration |
| 11 | --------------------- |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 12 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 13 | BESS UPF enabled edge setup requires three additional user plane subnets |
| 14 | apart from the default K8S subnets. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 15 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 16 | * **enb**: Used to provide eNB/gNB radios with connectivity to SD-Core and |
| 17 | UPF. |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 18 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 19 | * **access**: Used to provide UPF with connectivity to eNBs. |
| 20 | |
| 21 | * **core**: Used to provide UPF with edge services, local breakout, and |
| 22 | internet access. This naming comes from the P4 UPF, and is used for uniformity |
| 23 | |
| 24 | To help your understanding, the following example ACE environment will be used |
| 25 | in the rest of the guide. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 26 | |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 27 | .. image:: images/bess-upf-example-network.svg |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 28 | |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 29 | .. note:: |
| 30 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 31 | Admin and out-of-band networks are not depicted in the diagram. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 32 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 33 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 34 | | Network | VLAN ID | Subnet | Interface | IP address | |
| 35 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 36 | | k8smgmt | 1 | 192.168.1.0/24 (gw: 192.168.1.1) | management router | 192.168.1.254 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 37 | | | | +-------------------+---------------+ |
| 38 | | | | | compute1 | 192.168.1.3 | |
| 39 | | | | +-------------------+---------------+ |
| 40 | | | | | compute2 | 192.168.1.4 | |
| 41 | | | | +-------------------+---------------+ |
| 42 | | | | | compute3 | 192.168.1.5 | |
| 43 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 44 | | enb | 2 | 192.168.2.0/24 (gw: 192.168.2.1) | enb1 | 192.168.2.10 | |
| 45 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 46 | | access | 3 | 192.168.3.0/24 (gw: 192.168.3.1) | upf1 access | 192.168.3.10 | |
| 47 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 48 | | core | 4 | 192.168.4.0/24 (gw: 192.168.4.1) | management router | 192.168.4.254 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 49 | | | | +-------------------+---------------+ |
| 50 | | | | | upf1 core | 192.168.4.10 | |
| 51 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 52 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 53 | It is assumed that the management router has the only external routable address |
| 54 | and acts as a router connecting the Aether pod to the outside. |
| 55 | |
| 56 | This means that all uplink packets leaving the Aether pod need to be |
| 57 | masqueraded with the external address of the management router or the k8smgmt |
| 58 | address if the destination is Aether Central. |
| 59 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 60 | Also, in order for downlink traffic to UE to be delivered to its destination, |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 61 | it must be forwarded to the UPF's core interface. This adds additional routes |
| 62 | to the management router and L3 switch. |
| 63 | |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 64 | |
Charles Chan | 8486967 | 2022-03-04 15:54:42 -0800 | [diff] [blame] | 65 | .. note:: |
| 66 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 67 | There is a work-in-progress on interface consolidation in BESS UPF, which |
| 68 | will merge ``radio``, ``access``, and ``fabric`` into only one interface and |
Charles Chan | 8486967 | 2022-03-04 15:54:42 -0800 | [diff] [blame] | 69 | simplify the configuration significantly. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 70 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 71 | Required Cluster Resources |
| 72 | -------------------------- |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 73 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 74 | Before proceeding with the deployment, make sure the cluster has sufficient |
| 75 | resources to run BESS UPF by running the command below. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 76 | |
| 77 | .. code-block:: shell |
| 78 | |
Hyunsun Moon | 53a18a7 | 2021-08-10 16:57:32 -0700 | [diff] [blame] | 79 | $ kubectl get nodes -o json | jq '.items[].status.allocatable' |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 80 | { |
| 81 | "cpu": "95", |
| 82 | "ephemeral-storage": "1770223432846", |
| 83 | "hugepages-1Gi": "32Gi", |
| 84 | "intel.com/intel_sriov_netdevice": "32", |
| 85 | "intel.com/intel_sriov_vfio": "32", |
| 86 | "memory": "360749956Ki", |
| 87 | "pods": "110" |
| 88 | } |
| 89 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 90 | For best performance, BESS UPF requires the following resources: |
| 91 | |
| 92 | * 2 dedicated cores (``"cpu"``) |
| 93 | * 2 1GiB HugePages (``"hugepages-1Gi"``) |
| 94 | * 2 SRIOV Virtual Functions bound to **vfio-pci** driver (``"intel.com/intel_sriov_vfio"``) |
| 95 | |
| 96 | For environments where these resources are not available, contact Ops team for |
| 97 | advanced configuration. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 98 | |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 99 | Configure and Deploy |
| 100 | -------------------- |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 101 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 102 | Download ``aether-app-configs`` if you don't have it already in your |
| 103 | development machine. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 104 | |
| 105 | .. code-block:: shell |
| 106 | |
| 107 | $ cd $WORKDIR |
| 108 | $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-app-configs" |
| 109 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 110 | Move the directory to ``apps/bess-upf/upf1`` and create a Helm values file for |
| 111 | the new cluster as shown below. Don't forget to replace the IP addresses in |
| 112 | the example configuration with the addresses of the actual cluster. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 113 | |
Hyunsun Moon | 2b2bf9a | 2021-08-01 05:29:48 -0700 | [diff] [blame] | 114 | .. code-block:: yaml |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 115 | |
| 116 | $ cd $WORKDIR/aether-app-configs/apps/bess-upf/upf1 |
| 117 | $ mkdir overlays/prd-ace-test |
| 118 | $ vi overlays/prd-ace-test/values.yaml |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 119 | # SPDX-FileCopyrightText: 2021-present Open Networking Foundation <info@opennetworking.org> |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 120 | |
| 121 | config: |
| 122 | upf: |
| 123 | enb: |
| 124 | subnet: "192.168.2.0/24" |
| 125 | access: |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 126 | ip: "192.168.3.10/24" |
| 127 | gateway: "192.168.3.1" |
Hyunsun Moon | 6001f94 | 2021-08-01 10:35:04 -0700 | [diff] [blame] | 128 | vlan: 3 |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 129 | core: |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 130 | ip: "192.168.4.10/24" |
| 131 | gateway: "192.168.4.1" |
Hyunsun Moon | 6001f94 | 2021-08-01 10:35:04 -0700 | [diff] [blame] | 132 | vlan: 4 |
Hyunsun Moon | ba65df2 | 2021-08-10 20:01:06 -0700 | [diff] [blame] | 133 | # Add below when connecting to 5G core |
| 134 | #cfgFiles: |
| 135 | # upf.json: |
Charles Chan | 8486967 | 2022-03-04 15:54:42 -0800 | [diff] [blame] | 136 | # gtppsc: true |
Hyunsun Moon | ba65df2 | 2021-08-10 20:01:06 -0700 | [diff] [blame] | 137 | # cpiface: |
Charles Chan | 8486967 | 2022-03-04 15:54:42 -0800 | [diff] [blame] | 138 | # dnn: "internet" # should match the one configured in ROC |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 139 | |
| 140 | |
Zack Williams | b7d4515 | 2022-03-11 09:37:34 -0700 | [diff] [blame^] | 141 | Update ``fleet.yaml`` in the same directory to let Fleet use the custom |
| 142 | configuration when deploying BESS UPF to the new cluster. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 143 | |
Hyunsun Moon | 2b2bf9a | 2021-08-01 05:29:48 -0700 | [diff] [blame] | 144 | .. code-block:: yaml |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 145 | |
| 146 | $ vi fleet.yaml |
| 147 | # add following block at the end |
| 148 | - name: prd-ace-test |
| 149 | clusterSelector: |
| 150 | matchLabels: |
| 151 | management.cattle.io/cluster-display-name: ace-test |
| 152 | helm: |
| 153 | valuesFiles: |
| 154 | - overlays/prd-ace-test/values.yaml |
| 155 | |
| 156 | |
Hyunsun Moon | 2b2bf9a | 2021-08-01 05:29:48 -0700 | [diff] [blame] | 157 | Submit your changes. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 158 | |
| 159 | .. code-block:: shell |
| 160 | |
| 161 | $ cd $WORKDIR/aether-app-configs |
| 162 | $ git status |
| 163 | $ git add . |
| 164 | $ git commit -m "Add BESS UPF configs for test ACE" |
| 165 | $ git review |
| 166 | |
| 167 | |
| 168 | Go to Fleet dashboard and wait until the cluster status becomes **Active**. |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 169 | It can take up to 1 min for Fleet to fetch the configuration updates. |