Hung-Wei Chiu | 77c969e | 2020-10-23 18:13:07 +0000 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
| 5 | ========================== |
| 6 | Aether Run-Time Deployment |
| 7 | ========================== |
Hyunsun Moon | 5ca6bfb | 2020-10-29 22:29:12 -0700 | [diff] [blame^] | 8 | This section describes how to install Aether edge runtime and connectivity edge applications. |
| 9 | Aether provides GitOps based automated deployment, |
| 10 | so we just need to create a couple of patches to aether-pod-configs repository. |
| 11 | |
| 12 | Before you begin |
| 13 | ================ |
| 14 | Make sure you have the edge pod checklist ready. Specifically, the following information is required in this section. |
| 15 | |
| 16 | * Management network subnet |
| 17 | * K8S pod and service IP ranges |
| 18 | * List of servers and switches, and their management IP addresses |
| 19 | |
| 20 | Download aether-pod-configs repository |
| 21 | ====================================== |
| 22 | First, download the aether-pod-configs repository to your development machine. |
| 23 | |
| 24 | .. code-block:: shell |
| 25 | |
| 26 | $ cd $WORKDIR |
| 27 | $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-pod-configs" |
| 28 | |
| 29 | Create first patch to add ACE admin user |
| 30 | ======================================== |
| 31 | The first patch is to add a new ACE admin with full access to `EdgeApps` project. |
| 32 | Here is an example review request https://gerrit.opencord.org/c/aether-pod-configs/+/21393 you can refer to with the commands below. |
| 33 | Please replace "new" keyword with the name of the new ACE. |
| 34 | |
| 35 | .. code-block:: diff |
| 36 | |
| 37 | $ cd $WORKDIR/aether-pod-configs/production |
| 38 | $ vi user_map.tfvars |
| 39 | # Add the new cluster admin user to the end of the list |
| 40 | |
| 41 | $ git diff |
| 42 | diff --git a/production/user_map.tfvars b/production/user_map.tfvars |
| 43 | index c0ec3a3..6b9ffb4 100644 |
| 44 | --- a/production/user_map.tfvars |
| 45 | +++ b/production/user_map.tfvars |
| 46 | @@ -40,5 +40,10 @@ user_map = { |
| 47 | username = "menlo" |
| 48 | password = "changeme" |
| 49 | global_roles = ["user-base", "catalogs-use"] |
| 50 | + }, |
| 51 | + new_admin = { |
| 52 | + username = "new" |
| 53 | + password = "changeme" |
| 54 | + global_roles = ["user-base", "catalogs-use"] |
| 55 | } |
| 56 | } |
| 57 | |
| 58 | $ git add production/user_map.tfvars |
| 59 | $ git commit -m "Add admin user for new ACE" |
| 60 | $ git review |
| 61 | |
| 62 | The second patch has dependency on the first patch, so please make sure the first patch is merged before proceeding. |
| 63 | |
| 64 | Create second patch to install edge runtime and apps |
| 65 | ==================================================== |
| 66 | Now create another patch that will eventually install K8S and edge applications |
| 67 | including monitoring and logging stacks as well as Aether connected edge. |
| 68 | Unlike the first patch, this patch requires creating and editing multiple files. |
| 69 | Here is an example of the patch https://gerrit.opencord.org/c/aether-pod-configs/+/21395. |
| 70 | Please replace cluster names and IP addresses in this example accordingly. |
| 71 | |
| 72 | Update cluster_map.tfvars |
| 73 | ^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 74 | The first file to edit is `cluster_map.tfvars`. |
| 75 | Move the directory to `aether-pod-configs/production`, open `cluster_map.tfvars` file, and add the new ACE cluster information at the end of the map. |
| 76 | This change is required to register a new K8S cluster to Rancher, and update ACC and AMP clusters for inter-cluster service discovery. |
| 77 | |
| 78 | .. code-block:: diff |
| 79 | |
| 80 | $ cd $WORKDIR/aether-pod-configs/production |
| 81 | $ vi cluster_map.tfvars |
| 82 | # Edit the file and add the new cluster information to the end of the map |
| 83 | |
| 84 | $ git diff cluster_map.tfvars |
| 85 | diff --git a/production/cluster_map.tfvars b/production/cluster_map.tfvars |
| 86 | index c944352..a6d05a8 100644 |
| 87 | --- a/production/cluster_map.tfvars |
| 88 | +++ b/production/cluster_map.tfvars |
| 89 | @@ -89,6 +89,16 @@ cluster_map = { |
| 90 | kube_dns_cluster_ip = "10.53.128.10" |
| 91 | cluster_domain = "prd.menlo.aetherproject.net" |
| 92 | calico_ip_detect_method = "can-reach=www.google.com" |
| 93 | + }, |
| 94 | + ace-new = { |
| 95 | + cluster_name = "ace-new" |
| 96 | + management_subnets = ["10.94.1.0/24"] |
| 97 | + k8s_version = "v1.18.8-rancher1-1" |
| 98 | + k8s_pod_range = "10.54.0.0/17" |
| 99 | + k8s_cluster_ip_range = "10.54.128.0/17" |
| 100 | + kube_dns_cluster_ip = "10.54.128.10" |
| 101 | + cluster_domain = "prd.new.aetherproject.net" |
| 102 | + calico_ip_detect_method = "can-reach=www.google.com" |
| 103 | } |
| 104 | } |
| 105 | } |
| 106 | |
| 107 | Update vpn_map.tfvars |
| 108 | ^^^^^^^^^^^^^^^^^^^^^ |
| 109 | The second file to edit is `vpn_map.tfvars`. |
| 110 | Move the directory to `aether-pod-configs/production`, open `vpn_map.tfvars` file, and add VPN tunnel information at the end of the map. |
| 111 | Unless you have specific preference, set ASN and BGP peer addresses to the next available vales in the map. |
| 112 | This change is required to add tunnels and router interfaces to Aether central. |
| 113 | |
| 114 | .. code-block:: diff |
| 115 | |
| 116 | $ cd $WORKDIR/aether-pod-configs/production |
| 117 | $ vi vpn_map.tfvars |
| 118 | # Edit the file and add VPN tunnel information to the end of the map |
| 119 | |
| 120 | $ git diff vpn_map.tfvars |
| 121 | diff --git a/production/vpn_map.tfvars b/production/vpn_map.tfvars |
| 122 | index 3c1f9b9..dd62fce 100644 |
| 123 | --- a/production/vpn_map.tfvars |
| 124 | +++ b/production/vpn_map.tfvars |
| 125 | @@ -24,5 +24,15 @@ vpn_map = { |
| 126 | bgp_peer_ip_address_1 = "169.254.0.6" |
| 127 | bgp_peer_ip_range_2 = "169.254.1.5/30" |
| 128 | bgp_peer_ip_address_2 = "169.254.1.6" |
| 129 | + }, |
| 130 | + ace-new = { |
| 131 | + peer_name = "production-ace-new" |
| 132 | + peer_vpn_gateway_address = "111.222.333.444" |
| 133 | + tunnel_shared_secret = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" |
| 134 | + bgp_peer_asn = "65003" |
| 135 | + bgp_peer_ip_range_1 = "169.254.0.9/30" |
| 136 | + bgp_peer_ip_address_1 = "169.254.0.10" |
| 137 | + bgp_peer_ip_range_2 = "169.254.1.9/30" |
| 138 | + bgp_peer_ip_address_2 = "169.254.1.10" |
| 139 | } |
| 140 | } |
| 141 | |
| 142 | Create ACE specific state directory |
| 143 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 144 | Next step is to create a directory containing Terraform configs |
| 145 | that define desired state of Rancher and GCP resources for the new ACE cluster, |
| 146 | and ACE specific configurations such as IP addresses of the ACE cluster nodes. |
| 147 | |
| 148 | |
| 149 | Let's create a new directory under `aether-pod-configs/production` and |
| 150 | symbolic links to predefined Terraform configs(`*.tf` files) that will add |
| 151 | cluster, projects and applications in Rancher and VPN tunnels and firewall rules in GCP for the new ACE. |
| 152 | And note that Aether maintains a separate Terraform state per ACE. |
| 153 | So we will create a remote Terraform state definition for the new ACE, too. |
| 154 | |
| 155 | .. code-block:: shell |
| 156 | |
| 157 | # Create symbolic links to pre-defined Terraform configs |
| 158 | $ cd $WORKDIR/aether-pod-configs/production |
| 159 | $ mkdir ace-new && cd ace-new |
| 160 | $ ln -s ../../common/ace-custom/* . |
| 161 | |
| 162 | $ export CLUSTER_NAME=ace-new |
| 163 | $ export CLUSTER_DOMAIN=prd.new.aetherproject.net |
| 164 | |
| 165 | # Create Terraform state definition file |
| 166 | $ cat >> backend.tf << EOF |
| 167 | # SPDX-FileCopyrightText: 2020-present Open Networking Foundation <info@opennetworking.org> |
| 168 | |
| 169 | terraform { |
| 170 | backend "gcs" { |
| 171 | bucket = "aether-terraform-bucket" |
| 172 | prefix = "product/${CLUSTER_NAME}" |
| 173 | } |
| 174 | } |
| 175 | EOF |
| 176 | |
| 177 | # Confirm the changes |
| 178 | $ tree . |
| 179 | . |
| 180 | ├── alerts.tf -> ../../common/ace-custom/alerts.tf |
| 181 | ├── backend.tf |
| 182 | ├── cluster.tf -> ../../common/ace-custom/cluster.tf |
| 183 | ├── gcp_fw.tf -> ../../common/ace-custom/gcp_fw.tf |
| 184 | ├── gcp_ha_vpn.tf -> ../../common/ace-custom/gcp_ha_vpn.tf |
| 185 | ├── main.tf -> ../../common/ace-custom/main.tf |
| 186 | └── variables.tf -> ../../common/ace-custom/variables.tf |
| 187 | |
| 188 | |
| 189 | Now create another file called `cluster_val.tfvars` that defines all cluster nodes including switches and servers. |
| 190 | ACE can have various number of servers and switches but note that an odd number of *servers* can have `etcd` and `controlplane` roles. |
| 191 | Also, switches are not allowed to play a K8S master or normal worker role. |
| 192 | So don’t forget to add `node-role.aetherproject.org=switch` to labels and `node-role.aetherproject.org=switch:NoSchedule` to taints. |
| 193 | |
| 194 | |
| 195 | If the ACE requires any special settings, different set of projects for example, |
| 196 | please take a closer look at `variables.tf` file and override the default values specified there to `cluster_val.tfvars`, too. |
| 197 | |
| 198 | .. code-block:: shell |
| 199 | |
| 200 | $ cd $WORKDIR/aether-pod-configs/production/$CLUSTER_NAME |
| 201 | $ vi cluster_val.tfvars |
| 202 | # SPDX-FileCopyrightText: 2020-present Open Networking Foundation <info@opennetworking.org> |
| 203 | |
| 204 | cluster_name = "ace-new" |
| 205 | cluster_admin = "new_admin" |
| 206 | cluster_nodes = { |
| 207 | new-prd-leaf1 = { |
| 208 | user = "root" |
| 209 | private_key = "~/.ssh/id_rsa_terraform" |
| 210 | host = "10.94.1.3" |
| 211 | roles = ["worker"] |
| 212 | labels = ["node-role.aetherproject.org=switch"] |
| 213 | taints = ["node-role.aetherproject.org=switch:NoSchedule"] |
| 214 | }, |
| 215 | new-server-1 = { |
| 216 | user = "terraform" |
| 217 | private_key = "~/.ssh/id_rsa_terraform" |
| 218 | host = "10.94.1.3" |
| 219 | roles = ["etcd", "controlplane", "worker"] |
| 220 | labels = [] |
| 221 | taints = [] |
| 222 | }, |
| 223 | new-server-2 = { |
| 224 | user = "terraform" |
| 225 | private_key = "~/.ssh/id_rsa_terraform" |
| 226 | host = "10.94.1.4" |
| 227 | roles = ["etcd", "controlplane", "worker"] |
| 228 | labels = [] |
| 229 | taints = [] |
| 230 | }, |
| 231 | new-server-3 = { |
| 232 | user = "terraform" |
| 233 | private_key = "~/.ssh/id_rsa_terraform" |
| 234 | host = "10.94.1.5" |
| 235 | roles = ["etcd", "controlplane", "worker"] |
| 236 | labels = [] |
| 237 | taints = [] |
| 238 | } |
| 239 | } |
| 240 | |
| 241 | projects = [ |
| 242 | "system_apps", |
| 243 | "connectivity_edge_up4", |
| 244 | "edge_apps" |
| 245 | ] |
| 246 | |
| 247 | Lastly, we will create a couple of overriding values files for the managed applications, |
| 248 | one for DNS server for UEs and the other for the connectivity edge application, omec-upf-pfcp-agent. |
| 249 | |
| 250 | .. code-block:: shell |
| 251 | |
| 252 | $ cd $WORKDIR/aether-pod-configs/production/$CLUSTER_NAME |
| 253 | $ mkdir app_values && cd app_values |
| 254 | |
| 255 | $ export CLUSTER_NAME=ace-new |
| 256 | $ export CLUSTER_DOMAIN=prd.new.aetherproject.net |
| 257 | $ export K8S_DNS=10.54.128.10 # same address as kube_dns_cluster_ip |
| 258 | $ export UE_DNS=10.54.128.11 # next address of kube_dns_cluster_ip |
| 259 | |
| 260 | # Create ace-coredns overriding values file |
| 261 | $ cat >> ace-coredns.yml << EOF |
| 262 | # SPDX-FileCopyrightText: 2020-present Open Networking Foundation <info@opennetworking.org> |
| 263 | |
| 264 | serviceType: ClusterIP |
| 265 | service: |
| 266 | clusterIP: ${UE_DNS} |
| 267 | servers: |
| 268 | - zones: |
| 269 | - zone: . |
| 270 | port: 53 |
| 271 | plugins: |
| 272 | - name: errors |
| 273 | - name: health |
| 274 | configBlock: |- |
| 275 | lameduck 5s |
| 276 | - name: ready |
| 277 | - name: prometheus |
| 278 | parameters: 0.0.0.0:9153 |
| 279 | - name: forward |
| 280 | parameters: . /etc/resolv.conf |
| 281 | - name: cache |
| 282 | parameters: 30 |
| 283 | - name: loop |
| 284 | - name: reload |
| 285 | - name: loadbalance |
| 286 | - zones: |
| 287 | - zone: apps.svc.${CLUSTER_DOMAIN} |
| 288 | port: 53 |
| 289 | plugins: |
| 290 | - name: errors |
| 291 | - name: forward |
| 292 | parameters: . ${K8S_DNS} |
| 293 | - name: cache |
| 294 | parameters: 30 |
| 295 | EOF |
| 296 | |
| 297 | # Create PFCP agent overriding values file |
| 298 | $ cat >> omec-upf-pfcp-agent.yml << EOF |
| 299 | # SPDX-FileCopyrightText: 2020-present Open Networking Foundation <info@opennetworking.org> |
| 300 | |
| 301 | config: |
| 302 | pfcp: |
| 303 | cfgFiles: |
| 304 | upf.json: |
| 305 | p4rtciface: |
| 306 | p4rtc_server: "onos-tost-onos-classic-hs.tost.svc.${CLUSTER_DOMAIN}" |
| 307 | EOF |
| 308 | |
| 309 | Make sure the ace-new directory has all necessary files and before a review request. |
| 310 | |
| 311 | .. code-block:: shell |
| 312 | |
| 313 | $ cd $WORKDIR/aether-pod-configs/production/$CLUSTER_NAME |
| 314 | $ tree . |
| 315 | . |
| 316 | ├── alerts.tf -> ../../common/ace-custom/alerts.tf |
| 317 | ├── app_values |
| 318 | │ ├── ace-coredns.yml |
| 319 | │ └── omec-upf-pfcp-agent.yml |
| 320 | ├── backend.tf |
| 321 | ├── cluster.tf -> ../../common/ace-custom/cluster.tf |
| 322 | ├── cluster_val.tfvars |
| 323 | ├── gcp_fw.tf -> ../../common/ace-custom/gcp_fw.tf |
| 324 | ├── gcp_ha_vpn.tf -> ../../common/ace-custom/gcp_ha_vpn.tf |
| 325 | ├── main.tf -> ../../common/ace-custom/main.tf |
| 326 | └── variables.tf -> ../../common/ace-custom/variables.tf |
| 327 | |
| 328 | Create a review request |
| 329 | ^^^^^^^^^^^^^^^^^^^^^^^ |
| 330 | Now the patch is ready to review. The final step is to create a pull request! |
| 331 | Once the patch is accepted and merged, CD pipeline will install ACE runtime based on the patch. |
| 332 | |
| 333 | .. code-block:: shell |
| 334 | |
| 335 | $ cd $WORKDIR/aether-pod-configs/production |
| 336 | $ git status |
| 337 | On branch ace-new |
| 338 | Changes not staged for commit: |
| 339 | (use "git add <file>..." to update what will be committed) |
| 340 | (use "git checkout -- <file>..." to discard changes in working directory) |
| 341 | |
| 342 | modified: cluster_map.tfvars |
| 343 | modified: vpn_map.tfvars |
| 344 | |
| 345 | Untracked files: |
| 346 | (use "git add <file>..." to include in what will be committed) |
| 347 | |
| 348 | ace-new/ |
| 349 | |
| 350 | $ git add . |
| 351 | $ git commit -m "Add new ACE" |
| 352 | $ git review |
Hung-Wei Chiu | 77c969e | 2020-10-23 18:13:07 +0000 | [diff] [blame] | 353 | |