Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
Zack Williams | 5d2d678 | 2021-08-26 13:08:44 -0700 | [diff] [blame] | 5 | SD-Fabric Deployment (Beta) |
| 6 | =========================== |
| 7 | |
| 8 | .. note:: |
| 9 | |
| 10 | SD-Fabric using the P4 UPF is a beta feature in the Aether 1.5 release, and |
| 11 | the hardware and software setup is not required if using the BESS UPF. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 12 | |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 13 | Update aether-pod-configs |
| 14 | ------------------------- |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 15 | |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 16 | ``aether-pod-configs`` is a git project hosted on **gerrit.opencord.org** and |
| 17 | we placed the following materials in it. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 18 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 19 | - Terraform scripts to install SD-Fabric applications on Rancher, including ONOS, Stratum and Telegraf. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 20 | - Customized configuration for each application (helm values). |
| 21 | - Application specific configuration files, including ONOS network configuration and Stratum chassis config. |
| 22 | |
| 23 | Here is an example folder structure: |
| 24 | |
| 25 | .. code-block:: console |
| 26 | |
| 27 | ╰─$ tree staging/ace-menlo/tost |
| 28 | staging/ace-menlo/tost |
| 29 | ├── app_map.tfvars |
| 30 | ├── backend.tf |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 31 | ├── deepinsight |
| 32 | │ ├── README.md |
| 33 | │ ├── deepinsight-topo.json |
| 34 | │ └── deepinsight-topo.json.license |
| 35 | ├── main.tf -> ../../../common/tost/main.tf |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 36 | ├── onos |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 37 | │ ├── app_map.tfvars |
| 38 | │ ├── backend.tf |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 39 | │ ├── kubeconfig -> ../../../../common/tost/apps/onos/kubeconfig/ |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 40 | │ ├── main.tf -> ../../../../common/tost/apps/onos/main.tf |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 41 | │ ├── onos.yaml |
| 42 | │ └── variables.tf -> ../../../../common/tost/apps/onos/variables.tf |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 43 | ├── stratum |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 44 | │ ├── app_map.tfvars |
| 45 | │ ├── backend.tf |
| 46 | │ ├── main.tf -> ../../../../common/tost/apps/stratum/main.tf |
| 47 | │ ├── menlo-staging-leaf-1-chassis-config.pb.txt |
| 48 | │ ├── menlo-staging-leaf-2-chassis-config.pb.txt |
| 49 | │ ├── menlo-staging-spine-1-chassis-config.pb.txt |
| 50 | │ ├── menlo-staging-spine-2-chassis-config.pb.txt |
| 51 | │ ├── stratum.yaml |
| 52 | │ ├── tost-dev-chassis-config.pb.txt |
| 53 | │ └── variables.tf -> ../../../../common/tost/apps/stratum/variables.tf |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 54 | ├── telegraf |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 55 | │ ├── app_map.tfvars |
| 56 | │ ├── backend.tf |
| 57 | │ ├── main.tf -> ../../../../common/tost/apps/telegraf/main.tf |
| 58 | │ ├── telegraf.yaml |
| 59 | │ └── variables.tf -> ../../../../common/tost/apps/telegraf/variables.tf |
| 60 | └── variables.tf -> ../../../common/tost/variables.tf |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 61 | |
| 62 | There are four Terraform scripts inside **tost** directory and are responsible for managing each service. |
| 63 | |
| 64 | Root folder |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 65 | """"""""""" |
| 66 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 67 | Terraform reads **app_map.tfvars** to know which application will be installed on Rancher |
| 68 | and which version and customized values need to apply to. |
| 69 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 70 | Here is the example of **app_map.tfvars** which defines prerequisite apps for SD-Fabric |
| 71 | as well as project and namespace in which SD-fabric apps will be provisioned. |
Hyunsun Moon | fc751aa | 2020-11-11 18:49:47 -0800 | [diff] [blame] | 72 | Note that currently we don't have any prerequisite so we left this blank intentionally. |
| 73 | It can be used to specify prerequisites in the future. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 74 | |
| 75 | .. code-block:: |
| 76 | |
| 77 | project_name = "tost" |
| 78 | namespace_name = "tost" |
| 79 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 80 | |
| 81 | ONOS folder |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 82 | """"""""""" |
| 83 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 84 | All files under **onos** directory are related to ONOS application. |
Hyunsun Moon | fc751aa | 2020-11-11 18:49:47 -0800 | [diff] [blame] | 85 | The **app_map.tfvars** in this folder describes the information about ONOS helm chart. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 86 | |
| 87 | In this example, we specify the **onos-tost** helm chart version to **0.1.18** and load **onos.yaml** |
| 88 | as custom value files. |
| 89 | |
| 90 | .. code-block:: |
| 91 | |
| 92 | apps = ["onos"] |
| 93 | |
| 94 | app_map = { |
| 95 | onos = { |
| 96 | app_name = "onos-tost" |
| 97 | project_name = "tost" |
| 98 | target_namespace = "onos-tost" |
| 99 | catalog_name = "onos" |
| 100 | template_name = "onos-tost" |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 101 | template_version = "0.1.40" |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 102 | values_yaml = ["onos.yaml"] |
| 103 | } |
| 104 | } |
| 105 | |
| 106 | **onos.yaml** used to custom your ONOS-tost Helm chart values and please pay attention to the last section, config. |
| 107 | |
| 108 | .. code-block:: yaml |
| 109 | |
| 110 | onos-classic: |
| 111 | image: |
| 112 | tag: master |
| 113 | pullPolicy: Always |
| 114 | replicas: 1 |
| 115 | atomix: |
| 116 | replicas: 1 |
| 117 | logging: |
| 118 | config: | |
| 119 | # Common pattern layout for appenders |
| 120 | log4j2.stdout.pattern = %d{RFC3339} %-5level [%c{1}] %msg%n%throwable |
| 121 | |
| 122 | # Root logger |
| 123 | log4j2.rootLogger.level = INFO |
| 124 | |
| 125 | # OSGi appender |
| 126 | log4j2.rootLogger.appenderRef.PaxOsgi.ref = PaxOsgi |
| 127 | log4j2.appender.osgi.type = PaxOsgi |
| 128 | log4j2.appender.osgi.name = PaxOsgi |
| 129 | log4j2.appender.osgi.filter = * |
| 130 | |
| 131 | # stdout appender |
| 132 | log4j2.rootLogger.appenderRef.Console.ref = Console |
| 133 | log4j2.appender.console.type = Console |
| 134 | log4j2.appender.console.name = Console |
| 135 | log4j2.appender.console.layout.type = PatternLayout |
| 136 | log4j2.appender.console.layout.pattern = ${log4j2.stdout.pattern} |
| 137 | |
| 138 | # SSHD logger |
| 139 | log4j2.logger.sshd.name = org.apache.sshd |
| 140 | log4j2.logger.sshd.level = INFO |
| 141 | |
| 142 | # Spifly logger |
| 143 | log4j2.logger.spifly.name = org.apache.aries.spifly |
| 144 | log4j2.logger.spifly.level = WARN |
| 145 | |
| 146 | # SegmentRouting logger |
| 147 | log4j2.logger.segmentrouting.name = org.onosproject.segmentrouting |
| 148 | log4j2.logger.segmentrouting.level = DEBUG |
| 149 | |
| 150 | config: |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 151 | netcfg: > |
| 152 | { |
| 153 | "devices": { |
| 154 | "device:leaf1": { |
| 155 | "segmentrouting": { |
| 156 | "ipv4NodeSid": 201, |
| 157 | "ipv4Loopback": "10.128.100.38", |
| 158 | "routerMac": "00:00:0A:80:64:26", |
| 159 | "isEdgeRouter": true, |
| 160 | "adjacencySids": [] |
| 161 | }, |
| 162 | } |
| 163 | } |
| 164 | } |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 165 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 166 | |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 167 | **config.netcfg** is environment dependent and please change it to fit your environment. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 168 | |
| 169 | .. |
| 170 | TODO: Add an example based on the recommended topology |
| 171 | |
| 172 | Stratum folder |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 173 | """""""""""""" |
| 174 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 175 | Stratum uses a similar directory structure as ONOS for Terraform and its configuration files. |
| 176 | |
| 177 | The customize value file is named **stratum.yaml** |
| 178 | |
| 179 | .. code-block:: |
| 180 | |
| 181 | app_map = { |
| 182 | stratum= { |
| 183 | app_name = "stratum" |
| 184 | project_name = "tost" |
| 185 | target_namespace = "stratum" |
| 186 | catalog_name = "stratum" |
| 187 | template_name = "stratum" |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 188 | template_version = "0.1.13" |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 189 | values_yaml = ["stratum.yaml"] |
| 190 | } |
| 191 | } |
| 192 | |
| 193 | Like ONOS, **stratum.yaml** used to customize Stratum Helm Chart and please pay attention to the config section. |
| 194 | |
| 195 | .. code-block:: yaml |
| 196 | |
| 197 | image: |
| 198 | registry: registry.aetherproject.org |
| 199 | repository: tost/stratum-bfrt |
| 200 | tag: 9.2.0-4.14.49 |
| 201 | pullPolicy: Always |
| 202 | pullSecrets: |
| 203 | - aether-registry-credential |
| 204 | |
| 205 | extraParams: |
| 206 | - "-max_log_size=0" |
| 207 | - '-write_req_log_file=""' |
| 208 | - '-read_req_log_file=""' |
| 209 | - "-v=0" |
| 210 | - "-stderrthreshold=0" |
| 211 | - "-bf_switchd_background=false" |
| 212 | |
| 213 | nodeSelector: |
| 214 | node-role.aetherproject.org: switch |
| 215 | |
| 216 | tolerations: |
| 217 | - effect: NoSchedule |
| 218 | value: switch |
| 219 | key: node-role.aetherproject.org |
| 220 | |
| 221 | config: |
| 222 | server: gerrit.opencord.org |
| 223 | repo: aether-pod-configs |
| 224 | folder: staging/ace-onf-menlo/tost/stratum |
| 225 | |
| 226 | Stratum has the same deployment workflow as ONOS. |
| 227 | Once it is deployed to Kubernetes, it will read switch-dependent config files from the aether-pod-configs repo. |
| 228 | The key folder indicates that relative path of configs. |
| 229 | |
| 230 | .. attention:: |
| 231 | |
| 232 | The switch-dependent config file should be named as **${hostname}-chassis-config.pb.txt**. |
| 233 | For example, if the host name of your Tofino switch is **my-leaf**, please name config file **my-leaf-config.pb.txt**. |
| 234 | |
| 235 | .. |
| 236 | TODO: Add an example based on the recommended topology |
| 237 | |
| 238 | Telegraf folder |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 239 | """"""""""""""" |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 240 | |
| 241 | The app_map.tfvars specify the Helm Chart version and the filename of the custom Helm value file. |
| 242 | |
| 243 | .. code-block:: |
| 244 | |
| 245 | apps=["telegraf"] |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 246 | app_map = { |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 247 | telegraf = { |
| 248 | app_name = "telegraf" |
| 249 | project_name = "tost" |
| 250 | target_namespace = "tost" |
| 251 | catalog_name = "aether" |
| 252 | template_name = "tost-telegraf" |
| 253 | template_version = "0.1.1" |
| 254 | values_yaml = ["telegraf.yaml"] |
| 255 | } |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 256 | } |
| 257 | |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 258 | The **telegraf.yaml** used to override the ONOS-Telegraf Helm Chart and its environment-dependent. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 259 | Please pay attention to the **inputs.addresses** section. |
| 260 | Telegraf will read data from stratum so we need to specify all Tofino switch’s IP addresses here. |
| 261 | Taking Menlo staging pod as example, there are four switches so we fill out 4 IP addresses. |
| 262 | |
| 263 | .. code-block:: yaml |
| 264 | |
| 265 | podAnnotations: |
| 266 | field.cattle.io/workloadMetrics: '[{"path":"/metrics","port":9273,"schema":"HTTP"}]' |
| 267 | |
| 268 | config: |
| 269 | outputs: |
| 270 | - prometheus_client: |
| 271 | metric_version: 2 |
| 272 | listen: ":9273" |
| 273 | inputs: |
| 274 | - cisco_telemetry_gnmi: |
| 275 | addresses: |
| 276 | - 10.92.1.81:9339 |
| 277 | - 10.92.1.82:9339 |
| 278 | - 10.92.1.83:9339 |
| 279 | - 10.92.1.84:9339 |
| 280 | redial: 10s |
| 281 | - cisco_telemetry_gnmi.subscription: |
| 282 | name: stratum_counters |
| 283 | origin: openconfig-interfaces |
| 284 | path: /interfaces/interface[name=*]/state/counters |
| 285 | sample_interval: 5000ns |
| 286 | subscription_mode: sample |
| 287 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 288 | |
| 289 | Create Your Own Configs |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 290 | """"""""""""""""""""""" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 291 | |
| 292 | The easiest way to create your own configs is running the template script. |
| 293 | |
| 294 | Assumed we would like to set up the **ace-example** pod in the production environment. |
| 295 | |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 296 | 1. open the **tools/ace_config.yaml** (You should already have this file when you finish VPN bootstrap stage) |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 297 | 2. fill out all required variables |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 298 | 3. perform the makefile command to generate configuration and directory for SD-Fabric |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 299 | 4. update **onos.yaml** for ONOS |
| 300 | 5. update **${hostname}-chassis-config.pb.txt** for Stratum |
| 301 | 6. commit your change and open the Gerrit patch |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 302 | |
| 303 | .. code-block:: console |
| 304 | |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 305 | vim tools/ace_config.yaml |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 306 | make -C tools/ tost |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 307 | vim production/ace-example/tost/onos/onos.yaml |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 308 | vim production/ace-example/tost/stratum/*${hostname}-chassis-config.pb.txt** |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 309 | git add commit |
| 310 | git review |
| 311 | |
| 312 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 313 | Quick recap |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 314 | """"""""""" |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 315 | |
| 316 | To recap, most of the files in **tost** folder can be copied from existing examples. |
| 317 | However, there are a few files we need to pay extra attentions to. |
| 318 | |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 319 | - **onos.yaml** in **onos** folder |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 320 | - Chassis config in **stratum** folder |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 321 | There should be one chassis config for each switch. The file name needs to be |
| 322 | **${hostname}-chassis-config.pb.txt** |
| 323 | - **telegraf.yaml** in **telegraf** folder need to be updated with all switch |
| 324 | IP addresses |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 325 | |
| 326 | Double check these files and make sure they have been updated accordingly. |
| 327 | |
| 328 | |
| 329 | Create a review request |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 330 | """"""""""""""""""""""" |
| 331 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 332 | We also need to create a gerrit review request, similar to what we have done in |
| 333 | the **Aether Runtime Deployment**. |
| 334 | |
| 335 | Please refer to :doc:`Aether Runtime Deployment <runtime_deployment>` to |
| 336 | create a review request. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 337 | |
| 338 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 339 | Create SD-Fabric (named TOST in Jenkins) deployment job in Jenkins |
| 340 | ------------------------------------------------------------------ |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 341 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 342 | There are three major components in the Jenkins system, the Jenkins pipeline |
| 343 | and Jenkins Job Builder and Jenkins Job. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 344 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 345 | We follow the Infrastructure as Code principle to place three major components |
| 346 | in a Git repo, ``aether-ci-management`` |
| 347 | |
| 348 | Download the ``aether-ci-management`` repository. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 349 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 350 | .. code-block:: shell |
| 351 | |
| 352 | $ cd $WORKDIR |
| 353 | $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-ci-management" |
| 354 | |
| 355 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 356 | Here is the example of folder structure, we put everything related to three |
| 357 | major components under the jjb folder. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 358 | |
| 359 | .. code-block:: console |
| 360 | |
| 361 | $ tree -d jjb |
| 362 | jjb |
| 363 | ├── ci-management |
| 364 | ├── global |
| 365 | │ ├── jenkins-admin -> ../../global-jjb/jenkins-admin |
| 366 | │ ├── jenkins-init-scripts -> ../../global-jjb/jenkins-init-scripts |
| 367 | │ ├── jjb -> ../../global-jjb/jjb |
| 368 | │ └── shell -> ../../global-jjb/shell |
| 369 | ├── pipeline |
| 370 | ├── repos |
| 371 | ├── shell |
| 372 | └── templates |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 373 | |
| 374 | |
| 375 | Jenkins pipeline |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 376 | """""""""""""""" |
| 377 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 378 | Jenkins pipeline runs the Terraform scripts to install desired applications |
| 379 | into the specified Kubernetes cluster. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 380 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 381 | Both ONOS and Stratum will read configuration files (network config, chassis |
| 382 | config) from aether-pod-config. |
| 383 | |
| 384 | The default git branch is master. For testing purpose, we also provide two |
| 385 | parameters to specify the number of reviews and patchset. |
| 386 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 387 | We will explain more in the next section. |
| 388 | |
| 389 | .. note:: |
| 390 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 391 | Currently, we don’t perform the incremental upgrade for SD-Fabric application. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 392 | Instead, we perform the clean installation. |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 393 | In the pipeline script, Terraform will destroy all existing resources and |
| 394 | then create them again. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 395 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 396 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 397 | We put all pipeline scripts under the pipeline directory, the language of the |
| 398 | pipeline script is groovy. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 399 | |
| 400 | .. code-block:: console |
| 401 | |
| 402 | $ tree pipeline |
| 403 | pipeline |
| 404 | ├── aether-in-a-box.groovy |
| 405 | ├── artifact-release.groovy |
| 406 | ├── cd-pipeline-charts-postrelease.groovy |
| 407 | ├── cd-pipeline-dockerhub-postrelease.groovy |
| 408 | ├── cd-pipeline-postrelease.groovy |
| 409 | ├── cd-pipeline-terraform.groovy |
| 410 | ├── docker-publish.groovy |
| 411 | ├── ng40-func.groovy |
| 412 | ├── ng40-scale.groovy |
| 413 | ├── reuse-scan-gerrit.groovy |
| 414 | ├── reuse-scan-github.groovy |
| 415 | ├── tost-onos.groovy |
| 416 | ├── tost-stratum.groovy |
| 417 | ├── tost-telegraf.groovy |
| 418 | └── tost.groovy |
| 419 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 420 | Currently, we had five pipeline scripts for SD-Fabric deployment. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 421 | |
| 422 | 1. tost-onos.groovy |
| 423 | 2. tost-stratum.groovy |
| 424 | 3. tost-telegraf.groovy |
| 425 | 4. tost.groovy |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 426 | 5. tost-onos-debug.groovy |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 427 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 428 | tost-[onos/stratum/telegraf].groovy are used to deploy the individual |
| 429 | application respectively, and tost.groovy is a high level script, used to |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 430 | deploy whole SD-Fabric application, it will execute the above three scripts in its |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 431 | pipeline script. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 432 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 433 | tost-onos-debug.groovy is used to dump the debug information from the ONOS controller |
| 434 | and it will be executed automatically when ONOS is deployed. |
| 435 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 436 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 437 | Jenkins jobs |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 438 | """""""""""" |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 439 | |
| 440 | Jenkins job is the task unit in the Jenkins system. A Jenkins job contains the following information: |
| 441 | |
| 442 | - Jenkins pipeline |
| 443 | - Parameters for Jenkins pipeline |
| 444 | - Build trigger |
| 445 | - Source code management |
| 446 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 447 | We created one Jenkins job for each SD-Fabric component, per Aether edge. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 448 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 449 | We have four Jenkins jobs (HostPath provisioner, ONOS, Stratum and Telegraf) |
| 450 | for each edge as of today. |
| 451 | |
| 452 | There are 10+ parameters in Jenkins jobs and they can be divided into two |
| 453 | parts, cluster-level and application-level. |
| 454 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 455 | Here is an example of supported parameters. |
| 456 | |
| 457 | .. image:: images/jenkins-onos-params.png |
| 458 | :width: 480px |
| 459 | |
| 460 | Application level |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 461 | ''''''''''''''''' |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 462 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 463 | - **GERRIT_CHANGE_NUMBER/GERRIT_PATCHSET_NUMBER**: tell the pipeline script to read |
| 464 | the config for aether-pod-configs repo from a specified gerrit review, instead of the |
| 465 | HEAD branch. It’s good for developer to test its change before merge. |
| 466 | - **onos_user**: used to login ONOS controller |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 467 | - **git_repo/git_server/git_user/git_password_env**: information of git |
| 468 | repository, **git_password_env** is a key for Jenkins Credential system. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 469 | |
| 470 | Cluster level |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 471 | ''''''''''''' |
| 472 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 473 | - **gcp_credential**: Google Cloud Platform credential for remote storage, used |
| 474 | by Terraform. |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 475 | - **terraform_dir**: The root directory of the SD-Fabric directory. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 476 | - **rancher_cluster**: target Rancher cluster name. |
| 477 | - **rancher_api_env**: Rancher credential to access Rancher, used by Terraform. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 478 | |
| 479 | .. note:: |
| 480 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 481 | Typically, developer only focus on **GERRIT_CHANGE_NUMBER** and **GERRIT_PATCHSET_NUMBER**. The rest of them are managed by OPs. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 482 | |
| 483 | Jenkins Job Builder (JJB) |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 484 | """"""""""""""""""""""""" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 485 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 486 | We prefer to apply the IaC (Infrastructure as Code) for everything. We use the |
| 487 | JJB (Jenkins Job Builder) to create new Jenkins Job, including the Jenkins |
| 488 | pipeline. We need to clone a set of Jenkins jobs when a new edge is deployed. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 489 | |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 490 | In order to provide the flexibility and avoid re-inventing the wheel, we used |
| 491 | the job template to declare your job. Thanks to the JJB, we can use the |
| 492 | parameters in the job template to render different kinds of jobs easily. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 493 | |
| 494 | All the template files are placed under templates directory. |
| 495 | |
| 496 | .. code-block:: console |
| 497 | |
| 498 | ╰─$ tree templates |
| 499 | templates |
| 500 | ├── aether-in-a-box.yaml |
| 501 | ├── archive-artifacts.yaml |
| 502 | ├── artifact-release.yml |
| 503 | ├── cd-pipeline-terraform.yaml |
| 504 | ├── docker-publish-github.yaml |
| 505 | ├── docker-publish.yaml |
| 506 | ├── helm-lint.yaml |
| 507 | ├── make-test.yaml |
| 508 | ├── ng40-nightly.yaml |
| 509 | ├── ng40-test.yaml |
| 510 | ├── private-docker-publish.yaml |
| 511 | ├── private-make-test.yaml |
| 512 | ├── publish-helm-repo.yaml |
| 513 | ├── reuse-gerrit.yaml |
| 514 | ├── reuse-github.yaml |
| 515 | ├── sync-dir.yaml |
| 516 | ├── tost.yaml |
| 517 | ├── verify-licensed.yaml |
| 518 | └── versioning.yaml |
| 519 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 520 | We defined all SD-Fabric required job templates in tost.yaml and here is its partial |
Zack Williams | 794532a | 2021-03-18 17:38:36 -0700 | [diff] [blame] | 521 | content. |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 522 | |
| 523 | .. code-block:: yaml |
| 524 | |
| 525 | - job-template: |
| 526 | name: "{name}-onos" |
| 527 | id: "deploy-onos" |
| 528 | project-type: pipeline |
| 529 | dsl: !include-raw-escape: jjb/pipeline/tost-onos.groovy |
| 530 | triggers: |
| 531 | - onf-infra-tost-gerrit-trigger: |
| 532 | gerrit-server-name: '{gerrit-server-name}' |
| 533 | trigger_command: "apply" |
| 534 | pattern: "{terraform_dir}/tost/onos/.*" |
| 535 | logrotate: |
| 536 | daysToKeep: 7 |
| 537 | numToKeep: 10 |
| 538 | artifactDaysToKeep: 7 |
| 539 | artifactNumToKeep: 10 |
| 540 | parameters: |
| 541 | - string: |
| 542 | name: gcp_credential |
| 543 | default: "{google_bucket_access}" |
| 544 | - string: |
| 545 | name: rancher_cluster |
| 546 | default: "{rancher_cluster}" |
| 547 | - string: |
| 548 | name: rancher_api_env |
| 549 | default: "{rancher_api}" |
| 550 | - string: |
| 551 | name: git_repo |
| 552 | default: "aether-pod-configs" |
| 553 | - string: |
| 554 | name: git_server |
| 555 | default: "gerrit.opencord.org" |
| 556 | - string: |
| 557 | name: git_ssh_user |
| 558 | default: "jenkins" |
| 559 | |
| 560 | |
| 561 | |
| 562 | |
| 563 | Once we have the job template, we need to tell the JJB, we want to use the job template to create our own jobs. |
| 564 | Here comes the concept of project, you need to define job templates you want to use and the values of all parameters. |
| 565 | |
| 566 | |
| 567 | We put all project yaml files under the repo directory and here is the example |
| 568 | |
| 569 | .. code-block:: console |
| 570 | |
| 571 | ╰─$ tree repos 130 ↵ |
| 572 | repos |
| 573 | ├── aether-helm-charts.yaml |
| 574 | ├── aether-in-a-box.yaml |
| 575 | ├── cd-pipeline-terraform.yaml |
| 576 | ├── ng40-test.yaml |
| 577 | ├── spgw.yaml |
| 578 | └── tost.yaml |
| 579 | |
| 580 | |
| 581 | Following is the example of tost projects, we defined three projects here, and each project has different |
| 582 | parameters and Jenkins jobs it wants to use. |
| 583 | |
| 584 | .. code-block:: yaml |
| 585 | |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 586 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 587 | - project: |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 588 | name: deploy-tucson-pairedleaves-dev |
| 589 | rancher_cluster: "dev-pairedleaves-tucson" |
| 590 | terraform_dir: "staging/dev-pairedleaves-tucson" |
| 591 | rancher_api: "{rancher_staging_access}" |
| 592 | properties: |
| 593 | - onf-infra-onfstaff-private |
| 594 | jobs: |
| 595 | - "deploy" |
| 596 | - "deploy-onos" |
| 597 | - "deploy-stratum" |
| 598 | - "deploy-telegraf" |
| 599 | - "debug-tost" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 600 | |
| 601 | |
| 602 | Create Your Own Jenkins Job |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 603 | """"""""""""""""""""""""""" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 604 | |
| 605 | Basically, if you don't need to customize the Jenkins pipeline script and the job configuration, the only thing |
| 606 | you need to do is modify the repos/tost.yaml to add your project. |
| 607 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 608 | For example, we would like to deploy the SD-Fabric to our production pod, let's assume it named "tost-example". |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 609 | Add the following content into repos/tost.yaml |
| 610 | |
| 611 | .. code-block:: yaml |
| 612 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 613 | |
Hung-Wei Chiu | d06947b | 2021-08-10 16:10:07 -0700 | [diff] [blame] | 614 | - project: |
| 615 | name: deploy-tost-example-production |
| 616 | rancher_cluster: "ace-test-example" |
| 617 | terraform_dir: "production/tost-example" |
| 618 | rancher_api: "{rancher_production_access}" |
| 619 | properties: |
| 620 | - onf-infra-onfstaff-private |
| 621 | jobs: |
| 622 | - "deploy" |
| 623 | - "deploy-onos" |
| 624 | - "deploy-stratum" |
| 625 | - "deploy-telegraf" |
| 626 | - "debug-tost" |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 627 | |
| 628 | .. note:: |
| 629 | |
| 630 | The **terraform_dir** indicates the directory location in aether-pod-configs repo, please ensure your Terraform scripts |
| 631 | already there before running the Jenkins job. |
| 632 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 633 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 634 | Trigger SD-Fabric (named TOST in Jenkins) deployment in Jenkins |
| 635 | --------------------------------------------------------------- |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 636 | |
Hung-Wei Chiu | f7cadb3 | 2020-11-19 04:49:35 +0000 | [diff] [blame] | 637 | Whenever a change is merged into **aether-pod-config**, |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 638 | the Jenkins job should be triggered automatically to (re)deploy SD-Fabric (named TOST in Jenkins). |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 639 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 640 | You can also type the comment **apply** in the Gerrit patch, it will trigger Jenkins jobs to deploy SD-Fabric for you. |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 641 | |
Charles Chan | 2cb05c7 | 2020-12-09 16:31:20 -0800 | [diff] [blame] | 642 | |
| 643 | Verification |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 644 | ------------ |
| 645 | |
Charles Chan | 2cb05c7 | 2020-12-09 16:31:20 -0800 | [diff] [blame] | 646 | Fabric connectivity should be fully ready at this point. |
| 647 | We should verify that **all servers**, including compute nodes and the management server, |
| 648 | have an IP address and are **able to reach each other via fabric interface** before continuing the next step. |
| 649 | |
| 650 | This can be simply done by running a **ping** command from one server to another server's fabric IP. |
| 651 | |
| 652 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 653 | Troubleshooting |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 654 | --------------- |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 655 | |
| 656 | The deployment process involves the following steps: |
| 657 | |
| 658 | 1. Jenkins Job |
| 659 | 2. Jenkins Pipeline |
| 660 | 3. Clone Git Repository |
| 661 | 4. Execute Terraform scripts |
| 662 | 5. Rancher start to install applications |
| 663 | 6. Applications be deployed into Kubernetes cluster |
| 664 | 7. ONOS/Stratum will read the configuration (network config, chassis config) |
| 665 | 8. Pod become running |
| 666 | |
| 667 | Taking ONOS as an example, here's what you can do to troubleshoot. |
| 668 | |
| 669 | You can see the log message of the first 4 steps in Jenkins console. |
| 670 | If something goes wrong, the status of the Jenkins job will be in red. |
| 671 | If Jenkins doesn't report any error message, the next step is going to Rancher's portal |
| 672 | to ensure the Answers is same as the *onos.yaml* in *aether-pod-configs*. |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 673 | |
| 674 | Accessing the Stratum CLI |
| 675 | """"""""""""""""""""""""" |
| 676 | |
| 677 | You can login to the Stratum container running on a switch using this script: |
| 678 | |
| 679 | .. code-block:: sh |
| 680 | |
| 681 | #!/bin/bash |
| 682 | echo 'Attaching to Stratum container. Ctrl-P Ctrl-Q to exit' |
| 683 | echo 'Press Enter to continue...' |
| 684 | DOCKER_ID=`docker ps | grep stratum-bf | awk '{print $1}'` |
| 685 | docker attach $DOCKER_ID |
| 686 | |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 687 | You should then see the ``bf_sde`` prompt: |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 688 | |
| 689 | .. code-block:: sh |
| 690 | |
| 691 | bf_sde> pm |
| 692 | bf_sde.pm> show -a |
| 693 | |
| 694 | Accessing the ONOS CLI |
| 695 | """""""""""""""""""""" |
| 696 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 697 | After setting up kubectl to access the SD-Fabric pods, run: |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 698 | |
| 699 | .. code-block:: sh |
| 700 | |
| 701 | $ kubectl get pods -n tost |
| 702 | |
Hung-Wei Chiu | 6ed7910 | 2021-08-24 14:27:26 -0700 | [diff] [blame] | 703 | Pick a SD-Fabric pod, and make a port forward to it, then login to it with the |
Zack Williams | bcc4c2b | 2021-04-07 16:58:03 -0700 | [diff] [blame] | 704 | ``onos`` CLI tool: |
| 705 | |
| 706 | .. code-block:: sh |
| 707 | |
| 708 | $ kubectl -n tost port-forward onos-tost-onos-classic-0 8181 8101 |
| 709 | $ onos karaf@localhost |
| 710 | |
| 711 | In some rare cases, you may need to access the ONOS master instance CLI, in |
| 712 | which case you can run ``roles``: |
| 713 | |
| 714 | .. code-block:: sh |
| 715 | |
| 716 | karaf@root > roles |
| 717 | device:devswitch1: master=onos-tost-onos-classic-1, standbys=[ onos-tost-onos-classic-0 ] |
| 718 | |
| 719 | Above lines show that ``onos-tost-onos-classic-1`` is the master. So switch to |
| 720 | that by killing the port forward, starting a new one pointing at the master, |
| 721 | then logging into that one: |
| 722 | |
| 723 | .. code-block:: sh |
| 724 | |
| 725 | $ ps ax | grep -i kubectl |
| 726 | # returns kubectl commands running, pick the port-forward one and kill it |
| 727 | $ kill 0123 |
| 728 | $ kubectl -n tost port-forward onos-tost-onos-classic-1 8181 8101 |
| 729 | $ onos karaf@localhost |
| 730 | |