Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
Zack Williams | 5d2d678 | 2021-08-26 13:08:44 -0700 | [diff] [blame] | 5 | SD-Fabric Deployment (Beta) |
| 6 | =========================== |
| 7 | |
| 8 | .. note:: |
| 9 | |
| 10 | SD-Fabric using the P4 UPF is a beta feature in the Aether 1.5 release, and |
| 11 | the hardware and software setup is not required if using the BESS UPF. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 12 | |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 13 | Update aether-pod-configs |
| 14 | ------------------------- |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 15 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 16 | ``aether-app-configs`` is a git project hosted on **gerrit.opencord.org** and |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 17 | we placed the following materials in it. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 18 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 19 | - Rancher Fleet's configuration to install SD-Fabric applications on Rancher, |
| 20 | including ONOS, Stratum, Telegraf and PFCP-Agent. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 21 | - Customized configuration for each application (helm values). |
| 22 | - Application specific configuration files, including ONOS network configuration and Stratum chassis config. |
| 23 | |
| 24 | Here is an example folder structure: |
| 25 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 26 | .. code-block:: bash |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 27 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 28 | ╰─$ tree aether-dev/app/onos aether-dev/app/stratum aether-dev/app/pfcp-agent aether-dev/app/telegraf |
| 29 | ├── fleet.yaml |
| 30 | ├── kustomization.yaml |
| 31 | ├── overlays |
| 32 | │ ├── dev-pairedleaves-tucson |
| 33 | │ │ └── values.yaml |
| 34 | │ ├── dev-pdp-menlo |
| 35 | │ │ └── values.yaml |
| 36 | │ └── dev-sdfabric-menlo |
| 37 | │ └── values.yaml |
| 38 | └── registry-sealed-secret.yaml |
| 39 | aether-dev/app/stratum |
| 40 | ├── fleet.yaml |
| 41 | └── overlays |
| 42 | ├── dev-pairedleaves-tucson |
| 43 | │ ├── kustomization.yaml |
| 44 | │ ├── leaf1 |
| 45 | │ ├── leaf2 |
| 46 | │ ├── qos-config-leaf1.yaml |
| 47 | │ ├── qos-config-leaf2.yaml |
| 48 | │ └── values.yaml |
| 49 | └── dev-sdfabric-menlo |
| 50 | ├── kustomization.yaml |
| 51 | ├── menlo-sdfabric-leaf1 |
| 52 | ├── menlo-sdfabric-leaf2 |
| 53 | └── values.yaml |
| 54 | aether-dev/app/pfcp-agent |
| 55 | ├── fleet.yaml |
| 56 | └── overlays |
| 57 | ├── dev-pairedleaves-tucson |
| 58 | │ └── values.yaml |
| 59 | └── dev-sdfabric-menlo |
| 60 | └── values.yaml |
| 61 | aether-dev/app/telegraf |
| 62 | ├── fleet.yaml |
| 63 | └── overlays |
| 64 | ├── dev-pairedleaves-tucson |
| 65 | │ └── values.yaml |
| 66 | └── dev-sdfabric-menlo |
| 67 | └── values.yaml |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 68 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 69 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 70 | App folder |
| 71 | """""""""" |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 72 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 73 | Rancher Fleet reads ``fleet.yaml`` to know where to download the Helm Chart manifest and |
| 74 | how to customize the deployment for each target clusters. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 75 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 76 | Here is the example of ``fleet.yaml`` which downloads SD-Fabric(1.0.18) Helm Chart from |
| 77 | **https://charts.aetherproject.org** and then use the **overlays/$cluster_name/values.yaml** |
| 78 | to customize each cluster. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 79 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 80 | .. code-block:: YAML |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 81 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 82 | # SPDX-FileCopyrightText: 2021-present Open Networking Foundation <info@opennetworking.org> |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 83 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 84 | defaultNamespace: tost |
| 85 | helm: |
| 86 | releaseName: sdfabric |
| 87 | repo: https://charts.aetherproject.org |
| 88 | chart: sdfabric |
| 89 | version: 1.0.18 |
| 90 | helm: |
| 91 | values: |
| 92 | import: |
| 93 | stratum: |
| 94 | enabled: false |
| 95 | targetCustomizations: |
| 96 | - name: dev-sdfabric-menlo |
| 97 | clusterSelector: |
| 98 | matchLabels: |
| 99 | management.cattle.io/cluster-display-name: dev-sdfabric-menlo |
| 100 | helm: |
| 101 | valuesFiles: |
| 102 | - overlays/dev-sdfabric-menlo/values.yaml |
| 103 | - name: dev-pairedleaves-tucson |
| 104 | clusterSelector: |
| 105 | matchLabels: |
| 106 | management.cattle.io/cluster-display-name: dev-pairedleaves-tucson |
| 107 | helm: |
| 108 | valuesFiles: |
| 109 | - overlays/dev-pairedleaves-tucson/values.yaml |
| 110 | - name: dev-pdp-menlo |
| 111 | clusterSelector: |
| 112 | matchLabels: |
| 113 | management.cattle.io/cluster-display-name: dev-pdp-menlo |
| 114 | helm: |
| 115 | valuesFiles: |
| 116 | - overlays/dev-pdp-menlo/values.yaml |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 117 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 118 | |
| 119 | |
| 120 | **values.yaml** used to custom your sdfabric Helm chart values and please check |
Hung-Wei Chiu | 3accfe9 | 2021-11-02 12:23:53 -0700 | [diff] [blame] | 121 | `SD-Fabric Helm chart <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/HEAD/sdfabric/README.md>`_ |
| 122 | to see how to configure it. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 123 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 124 | ONOS App |
| 125 | """""""" |
| 126 | |
| 127 | For the ONOS application, the most import configuration is network configuration (netcfg) |
| 128 | which is environment-dependent configuration and you should configure it properly. |
| 129 | |
| 130 | netcfg is configured in the Helm Value files and please check the following example. |
| 131 | |
| 132 | .. code-block:: bash |
| 133 | |
| 134 | ╰─$ cat aether-app-configs/aether-dev/app/onos/overlays/dev-sdfabric-menlo/values.yaml 130 ↵ |
| 135 | # SPDX-FileCopyrightText: 2020-present Open Networking Foundation <info@opennetworking.org> |
| 136 | |
| 137 | # Value file for SDFabric helm chart. |
| 138 | ... |
| 139 | onos-classic: |
| 140 | config: |
| 141 | componentConfig: |
| 142 | "org.onosproject.net.host.impl.HostManager": > |
| 143 | { |
| 144 | "monitorHosts": "true", |
| 145 | "probeRate": "10000" |
| 146 | } |
| 147 | "org.onosproject.provider.general.device.impl.GeneralDeviceProvider": > |
| 148 | { |
| 149 | "readPortId": true |
| 150 | } |
| 151 | netcfg: > |
| 152 | { |
| 153 | ..... |
| 154 | } |
| 155 | |
| 156 | |
| 157 | |
| 158 | Please check |
| 159 | `SD-Fabric Configuration Guide <https://docs.sd-fabric.org/master/configuration/network.html>`_ |
| 160 | to learn more about network configuration. |
| 161 | |
| 162 | |
| 163 | Stratum App |
| 164 | """"""""""" |
| 165 | |
| 166 | Stratum reads the chassis config from the Kubernetes configmap resource but it doesn't support the function |
| 167 | to dynamically reload the chassis config, which means we have to restart the Stratum pod every time |
| 168 | when we update the chassis config. |
| 169 | |
| 170 | In order to solve this problem without modifying Stratum's source code, we have introduced the Kustomize to |
| 171 | the deployment process. Kustomize supports the function called configMapGenerator which generates the configmap |
| 172 | with a hash suffix in its name and then inject this hash-based name to the spec section of Stratum YAML file. |
| 173 | |
| 174 | See the following example, you can see the configmap name isn't fixed. |
| 175 | |
| 176 | .. code-block: bash |
| 177 | |
| 178 | ╰─$ kc -n tost get daemonset stratum -o json |
| 179 | | jq '.spec.template.spec.volumes | .[] | select(.name == "chassis-config")' |
| 180 | { |
| 181 | "configMap": { |
| 182 | "defaultMode": 484, |
| 183 | "name": "stratum-chassis-configs-7t6tt25654" |
| 184 | }, |
| 185 | "name": "chassis-config" |
| 186 | } |
| 187 | |
| 188 | |
| 189 | From the view of the Kubernetes, when it notices the spec of the YAML file is changed, it will redeploy whole |
| 190 | Stratum application, which means Stratum will read the updated chassis config eventually. |
| 191 | |
| 192 | .. code-block:: bash |
| 193 | |
| 194 | ╰─$ tree aether-dev/app/stratum |
| 195 | ├── fleet.yaml |
| 196 | └── overlays |
| 197 | ├── dev-pairedleaves-tucson |
| 198 | │ ├── kustomization.yaml |
| 199 | │ ├── leaf1 |
| 200 | │ ├── leaf2 |
| 201 | │ ├── qos-config-leaf1.yaml |
| 202 | │ ├── qos-config-leaf2.yaml |
| 203 | │ └── values.yaml |
| 204 | └── dev-sdfabric-menlo |
| 205 | ├── kustomization.yaml |
| 206 | ├── menlo-sdfabric-leaf1 |
| 207 | ├── menlo-sdfabric-leaf2 |
| 208 | └── values.yaml |
| 209 | |
| 210 | ╰─$ cat aether-dev/app/stratum/overlays/dev-pairedleaves-tucson/kustomization.yaml |
| 211 | # SPDX-FileCopyrightText: 2021-present Open Networking Foundation <info@opennetworking.org> |
| 212 | |
| 213 | configMapGenerator: |
| 214 | - name: stratum-chassis-configs |
| 215 | files: |
| 216 | - leaf1 |
| 217 | - leaf2 |
| 218 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 219 | .. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 220 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 221 | Check `SD-Fabric Doc <https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/HEAD/sdfabric/README.md>`_ |
| 222 | to learn how to write the chassis config and don't forget to add the file name into the kustomization.yaml file |
| 223 | once you set up your chassis config. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 224 | |
| 225 | .. attention:: |
| 226 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 227 | The switch-dependent config file should be named as **${hostname}**. |
| 228 | For example, if the host name of your Tofino switch is **my-leaf**, please name config file **my-leaf**. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 229 | |
| 230 | .. |
| 231 | TODO: Add an example based on the recommended topology |
| 232 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 233 | Telegraf App |
| 234 | """""""""""" |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 235 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 236 | Below is the example directory structure of Telegraf application. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 237 | |
| 238 | .. code-block:: |
| 239 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 240 | ╰─$ tree aether-dev/app/telegraf 255 ↵ |
| 241 | aether-dev/app/telegraf |
| 242 | ├── fleet.yaml |
| 243 | └── overlays |
| 244 | ├── dev-pairedleaves-tucson |
| 245 | │ └── values.yaml |
| 246 | └── dev-sdfabric-menlo |
| 247 | └── values.yaml |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 248 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 249 | |
| 250 | The **values.yaml** used to override the ONOS-Telegraf Helm Chart and its environment-dependent. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 251 | Please pay attention to the **inputs.addresses** section. |
| 252 | Telegraf will read data from stratum so we need to specify all Tofino switch’s IP addresses here. |
| 253 | Taking Menlo staging pod as example, there are four switches so we fill out 4 IP addresses. |
| 254 | |
| 255 | .. code-block:: yaml |
| 256 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 257 | config: |
| 258 | outputs: |
| 259 | - prometheus_client: |
| 260 | metric_version: 2 |
| 261 | listen: ":9273" |
| 262 | inputs: |
| 263 | - cisco_telemetry_gnmi: |
| 264 | addresses: |
| 265 | - 10.92.1.81:9339 |
| 266 | - 10.92.1.82:9339 |
| 267 | - 10.92.1.83:9339 |
| 268 | - 10.92.1.84:9339 |
| 269 | redial: 10s |
| 270 | - cisco_telemetry_gnmi.subscription: |
| 271 | name: stratum_counters |
| 272 | origin: openconfig-interfaces |
| 273 | path: /interfaces/interface[name=*]/state/counters |
| 274 | sample_interval: 5000ns |
| 275 | subscription_mode: sample |
| 276 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 277 | |
| 278 | Create Your Own Configs |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 279 | """"""""""""""""""""""" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 280 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 281 | Assume we would like to deploy the SD-Fabric to the ace-example cluster in the development environment. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 282 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 283 | 1. Modify the fleet.yaml to customize your cluster with specific value file. |
| 284 | 2. Add your Helm Values into the overlays folder. |
| 285 | 3. Have to add the chassis config file into the kustomization.yaml for Stratum application. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 286 | |
| 287 | .. code-block:: console |
| 288 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 289 | ╰─$ git st |
| 290 | On branch master |
| 291 | Your branch is up to date with 'origin/master'. |
| 292 | |
| 293 | Changes to be committed: |
| 294 | (use "git restore --staged <file>..." to unstage) |
| 295 | modified: aether-dev/app/onos/fleet.yaml |
| 296 | new file: aether-dev/app/onos/overlays/dev-my-cluster/values.yaml |
| 297 | modified: aether-dev/app/stratum/fleet.yaml |
| 298 | new file: aether-dev/app/stratum/overlays/dev-my-cluster/kustomization.yaml |
| 299 | new file: aether-dev/app/stratum/overlays/dev-my-cluster/menlo-sdfabric-leaf1 |
| 300 | new file: aether-dev/app/stratum/overlays/dev-my-cluster/menlo-sdfabric-leaf2 |
| 301 | new file: aether-dev/app/stratum/overlays/dev-my-cluster/values.yaml |
| 302 | modified: aether-dev/app/telegraf/fleet.yaml |
| 303 | new file: aether-dev/app/telegraf/overlays/dev-my-cluster/values.yaml |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 304 | |
| 305 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 306 | Quick recap |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 307 | """"""""""" |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 308 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 309 | To recap, most of the files in **app** folder can be copied from existing examples. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 310 | However, there are a few files we need to pay extra attentions to. |
| 311 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 312 | - ``fleet.yaml`` in each app folder |
| 313 | - Chassis config in **app/stratum/overlays/$cluster_name/** folder |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 314 | There should be one chassis config for each switch. The file name needs to be |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 315 | **${hostname}** |
| 316 | - **values.yaml** in **telegraf** folder need to be updated with all switch |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 317 | IP addresses |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 318 | |
| 319 | Double check these files and make sure they have been updated accordingly. |
| 320 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 321 | Create a review request |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 322 | """"""""""""""""""""""" |
| 323 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 324 | We also need to create a gerrit review request, similar to what we have done in |
| 325 | the **Aether Runtime Deployment**. |
| 326 | |
| 327 | Please refer to :doc:`Aether Runtime Deployment <runtime_deployment>` to |
| 328 | create a review request. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 329 | |
Hung-Wei Chiu | 3accfe9 | 2021-11-02 12:23:53 -0700 | [diff] [blame] | 330 | Deploy to ACE cluster |
| 331 | """"""""""""""""""""" |
| 332 | |
| 333 | SD-Fabric is environment dependent application and you have to prepare correct |
| 334 | configurations for both ONOS and Stratum to make it work. |
| 335 | |
Hung-Wei Chiu | 3accfe9 | 2021-11-02 12:23:53 -0700 | [diff] [blame] | 336 | Check below section to learn more about how we setup the Jenkins job and how it works |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 337 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 338 | Create SD-Fabric deployment job in Jenkins |
| 339 | ------------------------------------------ |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 340 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 341 | We have been using the Rancher Fleet to deploy SD-Fabric as the GitOps approach which means every change |
| 342 | we push to the Git repo will be synced to the target cluster automatically. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 343 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 344 | However, ONOS doesn't support the incremental upgrade which means we have to delete all ONOS instance and |
| 345 | then create all instance again every time we want to upgrade ONOS application. |
| 346 | |
| 347 | Rancher Fleet doesn't support the full recreation during the Application upgrade and that's reason we have |
| 348 | created a Jenkins job to recreate the ONOSs application. |
| 349 | |
| 350 | You have to add the Jenkins job for new cluster by modifying ``aether-ci-management`` |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 351 | |
| 352 | Download the ``aether-ci-management`` repository. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 353 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 354 | .. code-block:: shell |
| 355 | |
| 356 | $ cd $WORKDIR |
| 357 | $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-ci-management" |
| 358 | |
| 359 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 360 | Create Your Own Jenkins Job |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 361 | """"""""""""""""""""""""""" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 362 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 363 | Modify jjb/repos/sdfabric.yaml to add your cluster. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 364 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 365 | For example, we want to deploy the SD-Fabric to our new cluster **my-cluster** which is on the staging environment. |
| 366 | Add the following content into jjb/repo/sdfabric.yaml. |
| 367 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 368 | |
| 369 | .. code-block:: yaml |
| 370 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 371 | --- a/jjb/repos/sdfabric.yaml |
| 372 | +++ b/jjb/repos/sdfabric.yaml |
| 373 | @@ -50,6 +50,17 @@ |
| 374 | - "deploy-sdfabric-app": |
| 375 | - "deploy-debug" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 376 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 377 | +- project: |
| 378 | + name: my-cluster |
| 379 | + disable-job: false |
| 380 | + fleet-workspace: 'aether-dev' |
| 381 | + properties: |
| 382 | + - onf-infra-onfstaff-private |
| 383 | + jobs: |
| 384 | + - "deploy-sdfabric-app": |
| 385 | + - "deploy-debug" |
| 386 | + |
| 387 | + |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 388 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 389 | If your cluster is on the production environment, you have to change both **terraform_env** and **fleet-workspace** |
Hung-Wei Chiu | 3accfe9 | 2021-11-02 12:23:53 -0700 | [diff] [blame] | 390 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 391 | Trigger SD-Fabric deployment in Jenkins |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 392 | --------------------------------------------------------------- |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 393 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 394 | Whenever a change is merged into **aether-app-config**, |
| 395 | the Jenkins job should be triggered automatically to (re)deploy SD-Fabric . |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 396 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 397 | You can also manually trigger the job to redeploy SD-Fabric if needed and below |
| 398 | is an example of default parameters when you run the job. |
| 399 | |
| 400 | .. image:: images/jenkins-sdfabric-params.png |
| 401 | :width: 480px |
| 402 | |
| 403 | |
| 404 | If you want to capture all SD-Fabric related containers logs before redeploying them, |
| 405 | please enable ``POD_LOG`` option. |
| 406 | The Jenkins job helps to redeploy ONOS, Stratum and PFCP-Agent application and the default |
| 407 | options is ONOS and Stratum, you can redeploy what you want by click those ``REDEPLOY_XXXX`` |
| 408 | options. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 409 | |
Charles Chan | 2cb05c7 | 2020-12-09 16:31:20 -0800 | [diff] [blame] | 410 | |
| 411 | Verification |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 412 | ------------ |
| 413 | |
Charles Chan | 2cb05c7 | 2020-12-09 16:31:20 -0800 | [diff] [blame] | 414 | Fabric connectivity should be fully ready at this point. |
| 415 | We should verify that **all servers**, including compute nodes and the management server, |
| 416 | have an IP address and are **able to reach each other via fabric interface** before continuing the next step. |
| 417 | |
| 418 | This can be simply done by running a **ping** command from one server to another server's fabric IP. |
| 419 | |
| 420 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 421 | Troubleshooting |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 422 | --------------- |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 423 | |
| 424 | The deployment process involves the following steps: |
| 425 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 426 | 1. Jenkins Job (For ONOS Only) |
| 427 | 2. Rancher Fleet upgrade application based on Git change |
| 428 | 3. Applications be deployed into Kubernetes cluster |
| 429 | 4. ONOS/Stratum will read the configuration (network config, chassis config) |
| 430 | 5. Pod become running |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 431 | |
| 432 | Taking ONOS as an example, here's what you can do to troubleshoot. |
| 433 | |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 434 | You can see the log message of the first step in Jenkins console. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 435 | If something goes wrong, the status of the Jenkins job will be in red. |
Hung-Wei Chiu | b392a16 | 2022-02-15 11:09:33 -0800 | [diff] [blame^] | 436 | If Jenkins doesn't report any error message, the next step is going to Rancher Fleet's |
| 437 | portal to ensure Fleet works as expected. |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 438 | |
| 439 | Accessing the Stratum CLI |
| 440 | """"""""""""""""""""""""" |
| 441 | |
| 442 | You can login to the Stratum container running on a switch using this script: |
| 443 | |
| 444 | .. code-block:: sh |
| 445 | |
| 446 | #!/bin/bash |
| 447 | echo 'Attaching to Stratum container. Ctrl-P Ctrl-Q to exit' |
| 448 | echo 'Press Enter to continue...' |
| 449 | DOCKER_ID=`docker ps | grep stratum-bf | awk '{print $1}'` |
| 450 | docker attach $DOCKER_ID |
| 451 | |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 452 | You should then see the ``bf_sde`` prompt: |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 453 | |
| 454 | .. code-block:: sh |
| 455 | |
| 456 | bf_sde> pm |
| 457 | bf_sde.pm> show -a |
| 458 | |
| 459 | Accessing the ONOS CLI |
| 460 | """""""""""""""""""""" |
| 461 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 462 | After setting up kubectl to access the SD-Fabric pods, run: |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 463 | |
| 464 | .. code-block:: sh |
| 465 | |
| 466 | $ kubectl get pods -n tost |
| 467 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 468 | Pick a SD-Fabric pod, and make a port forward to it, then login to it with the |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 469 | ``onos`` CLI tool: |
| 470 | |
| 471 | .. code-block:: sh |
| 472 | |
| 473 | $ kubectl -n tost port-forward onos-tost-onos-classic-0 8181 8101 |
| 474 | $ onos karaf@localhost |
| 475 | |
| 476 | In some rare cases, you may need to access the ONOS master instance CLI, in |
| 477 | which case you can run ``roles``: |
| 478 | |
| 479 | .. code-block:: sh |
| 480 | |
| 481 | karaf@root > roles |
| 482 | device:devswitch1: master=onos-tost-onos-classic-1, standbys=[ onos-tost-onos-classic-0 ] |
| 483 | |
| 484 | Above lines show that ``onos-tost-onos-classic-1`` is the master. So switch to |
| 485 | that by killing the port forward, starting a new one pointing at the master, |
| 486 | then logging into that one: |
| 487 | |
| 488 | .. code-block:: sh |
| 489 | |
| 490 | $ ps ax | grep -i kubectl |
| 491 | # returns kubectl commands running, pick the port-forward one and kill it |
| 492 | $ kill 0123 |
| 493 | $ kubectl -n tost port-forward onos-tost-onos-classic-1 8181 8101 |
| 494 | $ onos karaf@localhost |