Hung-Wei Chiu | 77c969e | 2020-10-23 18:13:07 +0000 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
| 5 | ============= |
| 6 | Bootstrapping |
| 7 | ============= |
| 8 | |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 9 | VPN |
| 10 | === |
| 11 | This section walks you through how to set up a VPN between ACE and Aether Central in GCP. |
| 12 | We will be using GitOps based Aether CD pipeline for this, |
| 13 | so we just need to create a patch to **aether-pod-configs** repository. |
| 14 | Note that some of the steps described here are not directly related to setting up a VPN, |
| 15 | but rather are a prerequisite for adding a new ACE. |
| 16 | |
Hyunsun Moon | 5c1e0b0 | 2020-11-20 11:09:00 -0800 | [diff] [blame^] | 17 | .. attention:: |
| 18 | |
| 19 | If you are adding another ACE to an existing VPN connection, go to |
| 20 | :ref:`Add ACE to an existing VPN connection <add_ace_to_vpn>` |
| 21 | |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 22 | Before you begin |
| 23 | ---------------- |
| 24 | * Make sure firewall in front of ACE allows UDP port 500, UDP port 4500, and ESP packets |
| 25 | from **gcpvpn1.infra.aetherproject.net(35.242.47.15)** and **gcpvpn2.infra.aetherproject.net(34.104.68.78)** |
| 26 | * Make sure that the external IP on ACE side is owned by or routed to the management node |
| 27 | |
| 28 | To help your understanding, the following sample ACE environment will be used in the rest of this section. |
| 29 | Make sure to replace the sample values when you actually create a review request. |
| 30 | |
| 31 | +-----------------------------+----------------------------------+ |
| 32 | | Management node external IP | 128.105.144.189 | |
| 33 | +-----------------------------+----------------------------------+ |
| 34 | | ASN | 65003 | |
| 35 | +-----------------------------+----------------------------------+ |
| 36 | | GCP BGP IP address | Tunnel 1: 169.254.0.9/30 | |
| 37 | | +----------------------------------+ |
| 38 | | | Tunnel 2: 169.254.1.9/30 | |
| 39 | +-----------------------------+----------------------------------+ |
| 40 | | ACE BGP IP address | Tunnel 1: 169.254.0.10/30 | |
| 41 | | +----------------------------------+ |
| 42 | | | Tunnel 2: 169.254.1.10/30 | |
| 43 | +-----------------------------+----------------------------------+ |
| 44 | | PSK | UMAoZA7blv6gd3IaArDqgK2s0sDB8mlI | |
| 45 | +-----------------------------+----------------------------------+ |
| 46 | | Management Subnet | 10.91.0.0/24 | |
| 47 | +-----------------------------+----------------------------------+ |
| 48 | | K8S Subnet | Pod IP: 10.66.0.0/17 | |
| 49 | | +----------------------------------+ |
| 50 | | | Cluster IP: 10.66.128.0/17 | |
| 51 | +-----------------------------+----------------------------------+ |
| 52 | |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 53 | Download aether-pod-configs repository |
| 54 | -------------------------------------- |
| 55 | .. code-block:: shell |
| 56 | |
| 57 | $ cd $WORKDIR |
| 58 | $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-pod-configs" |
| 59 | |
Hyunsun Moon | 0e080e4 | 2020-11-18 12:53:13 -0800 | [diff] [blame] | 60 | .. _update_global_resource: |
| 61 | |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 62 | Update global resource maps |
| 63 | --------------------------- |
| 64 | Add a new ACE information at the end of the following global resource maps. |
| 65 | |
| 66 | * user_map.tfvars |
| 67 | * cluster_map.tfvars |
| 68 | * vpn_map.tfvars |
| 69 | |
| 70 | As a note, you can find several other global resource maps under the `production` directory. |
| 71 | Resource definitions that need to be shared among clusters or are better managed in a |
| 72 | single file to avoid configuration conflicts are maintained in this way. |
| 73 | |
| 74 | .. code-block:: diff |
| 75 | |
| 76 | $ cd $WORKDIR/aether-pod-configs/production |
| 77 | $ vi user_map.tfvars |
| 78 | |
| 79 | # Add the new cluster admin user at the end of the map |
| 80 | $ git diff user_map.tfvars |
| 81 | --- a/production/user_map.tfvars |
| 82 | +++ b/production/user_map.tfvars |
| 83 | @@ user_map = { |
| 84 | username = "menlo" |
| 85 | password = "changeme" |
| 86 | global_roles = ["user-base", "catalogs-use"] |
| 87 | + }, |
| 88 | + test_admin = { |
| 89 | + username = "test" |
| 90 | + password = "changeme" |
| 91 | + global_roles = ["user-base", "catalogs-use"] |
| 92 | } |
| 93 | } |
| 94 | |
| 95 | .. code-block:: diff |
| 96 | |
| 97 | $ cd $WORKDIR/aether-pod-configs/production |
| 98 | $ vi cluster_map.tfvars |
| 99 | |
| 100 | # Add the new K8S cluster information at the end of the map |
| 101 | $ git diff cluster_map.tfvars |
| 102 | --- a/production/cluster_map.tfvars |
| 103 | +++ b/production/cluster_map.tfvars |
| 104 | @@ cluster_map = { |
| 105 | kube_dns_cluster_ip = "10.53.128.10" |
| 106 | cluster_domain = "prd.menlo.aetherproject.net" |
| 107 | calico_ip_detect_method = "can-reach=www.google.com" |
| 108 | + }, |
| 109 | + ace-test = { |
| 110 | + cluster_name = "ace-test" |
| 111 | + management_subnets = ["10.91.0.0/24"] |
| 112 | + k8s_version = "v1.18.8-rancher1-1" |
| 113 | + k8s_pod_range = "10.66.0.0/17" |
| 114 | + k8s_cluster_ip_range = "10.66.128.0/17" |
| 115 | + kube_dns_cluster_ip = "10.66.128.10" |
| 116 | + cluster_domain = "prd.test.aetherproject.net" |
| 117 | + calico_ip_detect_method = "can-reach=www.google.com" |
| 118 | } |
| 119 | } |
| 120 | } |
| 121 | |
| 122 | .. code-block:: diff |
| 123 | |
| 124 | $ cd $WORKDIR/aether-pod-configs/production |
| 125 | $ vi vpn_map.tfvars |
| 126 | |
| 127 | # Add VPN and tunnel information at the end of the map |
| 128 | $ git diff vpn_map.tfvars |
| 129 | --- a/production/vpn_map.tfvars |
| 130 | +++ b/production/vpn_map.tfvars |
| 131 | @@ vpn_map = { |
| 132 | bgp_peer_ip_address_1 = "169.254.0.6" |
| 133 | bgp_peer_ip_range_2 = "169.254.1.5/30" |
| 134 | bgp_peer_ip_address_2 = "169.254.1.6" |
| 135 | + }, |
| 136 | + ace-test = { |
| 137 | + peer_name = "production-ace-test" |
| 138 | + peer_vpn_gateway_address = "128.105.144.189" |
| 139 | + tunnel_shared_secret = "UMAoZA7blv6gd3IaArDqgK2s0sDB8mlI" |
| 140 | + bgp_peer_asn = "65003" |
| 141 | + bgp_peer_ip_range_1 = "169.254.0.9/30" |
| 142 | + bgp_peer_ip_address_1 = "169.254.0.10" |
| 143 | + bgp_peer_ip_range_2 = "169.254.1.9/30" |
| 144 | + bgp_peer_ip_address_2 = "169.254.1.10" |
| 145 | } |
| 146 | } |
| 147 | |
| 148 | .. note:: |
| 149 | Unless you have a specific requirement, set ASN and BGP addresses to the next available values in the map. |
| 150 | |
| 151 | |
| 152 | Create ACE specific configurations |
| 153 | ---------------------------------- |
| 154 | In this step, we will create a directory under `production` with the same name as ACE, |
| 155 | and add several Terraform configurations and Ansible inventory needed to configure a VPN connection. |
| 156 | Throughout the deployment procedure, this directory will contain all ACE specific configurations. |
| 157 | |
| 158 | Run the following commands to auto-generate necessary files under the target ACE directory. |
| 159 | |
| 160 | .. code-block:: shell |
| 161 | |
| 162 | $ cd $WORKDIR/aether-pod-configs/tools |
Hyunsun Moon | 0e080e4 | 2020-11-18 12:53:13 -0800 | [diff] [blame] | 163 | $ cp ace_env /tmp/ace_env |
| 164 | $ vi /tmp/ace_env |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 165 | # Set environment variables |
| 166 | |
Hyunsun Moon | 0e080e4 | 2020-11-18 12:53:13 -0800 | [diff] [blame] | 167 | $ source /tmp/ace_env |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 168 | $ make vpn |
| 169 | Created ../production/ace-test |
| 170 | Created ../production/ace-test/main.tf |
| 171 | Created ../production/ace-test/variables.tf |
| 172 | Created ../production/ace-test/gcp_fw.tf |
| 173 | Created ../production/ace-test/gcp_ha_vpn.tf |
| 174 | Created ../production/ace-test/ansible |
| 175 | Created ../production/ace-test/backend.tf |
| 176 | Created ../production/ace-test/cluster_val.tfvars |
| 177 | Created ../production/ace-test/ansible/hosts.ini |
| 178 | Created ../production/ace-test/ansible/extra_vars.yml |
| 179 | |
| 180 | .. attention:: |
| 181 | The predefined templates are tailored to Pronto BOM. You'll need to fix `cluster_val.tfvars` and `ansible/extra_vars.yml` |
| 182 | when using a different BOM. |
| 183 | |
| 184 | Create a review request |
| 185 | ----------------------- |
| 186 | .. code-block:: shell |
| 187 | |
| 188 | $ cd $WORKDIR/aether-pod-configs/production |
| 189 | $ git status |
| 190 | On branch tools |
| 191 | Changes not staged for commit: |
| 192 | |
| 193 | modified: cluster_map.tfvars |
| 194 | modified: user_map.tfvars |
| 195 | modified: vpn_map.tfvars |
| 196 | |
| 197 | Untracked files: |
| 198 | (use "git add <file>..." to include in what will be committed) |
| 199 | |
| 200 | ace-test/ |
| 201 | |
| 202 | $ git add . |
| 203 | $ git commit -m "Add test ACE" |
| 204 | $ git review |
| 205 | |
| 206 | Once the review request is accepted and merged, |
| 207 | CD pipeline will create VPN tunnels on both GCP and the management node. |
| 208 | |
| 209 | Verify VPN connection |
| 210 | --------------------- |
| 211 | You can verify the VPN connections after successful post-merge job |
| 212 | by checking the routing table on the management node and trying to ping to one of the central cluster VMs. |
| 213 | Make sure two tunnel interfaces, `gcp_tunnel1` and `gcp_tunnel2`, exist |
| 214 | and three additional routing entries via one of the tunnel interfaces. |
| 215 | |
| 216 | .. code-block:: shell |
| 217 | |
Hyunsun Moon | 5c1e0b0 | 2020-11-20 11:09:00 -0800 | [diff] [blame^] | 218 | # Verify routings |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 219 | $ netstat -rn |
| 220 | Kernel IP routing table |
| 221 | Destination Gateway Genmask Flags MSS Window irtt Iface |
| 222 | 0.0.0.0 128.105.144.1 0.0.0.0 UG 0 0 0 eno1 |
| 223 | 10.45.128.0 169.254.0.9 255.255.128.0 UG 0 0 0 gcp_tunnel1 |
| 224 | 10.52.128.0 169.254.0.9 255.255.128.0 UG 0 0 0 gcp_tunnel1 |
| 225 | 10.66.128.0 10.91.0.8 255.255.128.0 UG 0 0 0 eno1 |
| 226 | 10.91.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1 |
| 227 | 10.168.0.0 169.254.0.9 255.255.240.0 UG 0 0 0 gcp_tunnel1 |
| 228 | 128.105.144.0 0.0.0.0 255.255.252.0 U 0 0 0 eno1 |
| 229 | 169.254.0.8 0.0.0.0 255.255.255.252 U 0 0 0 gcp_tunnel1 |
| 230 | 169.254.1.8 0.0.0.0 255.255.255.252 U 0 0 0 gcp_tunnel2 |
| 231 | |
Hyunsun Moon | 5c1e0b0 | 2020-11-20 11:09:00 -0800 | [diff] [blame^] | 232 | # Verify ACC VM access |
| 233 | $ ping 10.168.0.6 |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 234 | |
Hyunsun Moon | 5c1e0b0 | 2020-11-20 11:09:00 -0800 | [diff] [blame^] | 235 | # Verify ACC K8S cluster access |
| 236 | $ nslookup kube-dns.kube-system.svc.prd.acc.gcp.aetherproject.net 10.52.128.10 |
| 237 | |
| 238 | You can further verify whether the ACE routes are propagated well to GCP |
| 239 | by checking GCP dashboard **VPC Network > Routes > Dynamic**. |
| 240 | |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 241 | |
| 242 | Post VPN setup |
| 243 | -------------- |
| 244 | Once you verify the VPN connections, please update `ansible` directory name to `_ansible` to prevent |
| 245 | the ansible playbook from running again. |
| 246 | Note that it is no harm to re-run the ansible playbook but not recommended. |
| 247 | |
| 248 | .. code-block:: shell |
| 249 | |
| 250 | $ cd $WORKDIR/aether-pod-configs/production/$ACE_NAME |
| 251 | $ mv ansible _ansible |
| 252 | $ git add . |
| 253 | $ git commit -m "Mark ansible done for test ACE" |
| 254 | $ git review |
| 255 | |
Hyunsun Moon | 5c1e0b0 | 2020-11-20 11:09:00 -0800 | [diff] [blame^] | 256 | .. _add_ace_to_vpn: |
| 257 | |
| 258 | Add another ACE to an existing VPN connection |
| 259 | --------------------------------------------- |
| 260 | VPN connections can be shared when there are multiple ACE clusters in a site. |
| 261 | In order to add ACE to an existing VPN connection, |
| 262 | you'll have to SSH into the management node and manually update BIRD configuration. |
| 263 | |
| 264 | .. note:: |
| 265 | |
| 266 | This step needs improvements in the future. |
| 267 | |
| 268 | .. code-block:: shell |
| 269 | |
| 270 | $ sudo vi /etc/bird/bird.conf |
| 271 | protocol static { |
| 272 | ... |
| 273 | route 10.66.128.0/17 via 10.91.0.10; |
| 274 | |
| 275 | # Add routings for the new ACE's K8S cluster IP range via cluster nodes |
| 276 | # TODO: Configure iBGP peering with Calico nodes and dynamically learn these routings |
| 277 | route <NEW-ACE-CLUSTER-IP> via <SERVER1> |
| 278 | route <NEW-ACE-CLUSTER-IP> via <SERVER2> |
| 279 | route <NEW-ACE-CLUSTER-IP> via <SERVER3> |
| 280 | } |
| 281 | |
| 282 | filter gcp_tunnel_out { |
| 283 | # Add the new ACE's K8S cluster IP range and the management subnet if required to the list |
| 284 | if (net ~ [ 10.91.0.0/24, 10.66.128.0/17, <NEW-ACE-CLUSTER-IP-RANGE> ]) then accept; |
| 285 | else reject; |
| 286 | } |
| 287 | # Save and exit |
| 288 | |
| 289 | $ sudo birdc configure |
| 290 | |
| 291 | # Confirm the static routes are added |
| 292 | $ sudo birdc show route |
Hyunsun Moon | a79c742 | 2020-11-18 04:52:56 -0800 | [diff] [blame] | 293 | |
Charles Chan | 4a10722 | 2020-10-30 17:23:48 -0700 | [diff] [blame] | 294 | OS Installation - Switches |
| 295 | ========================== |
| 296 | |
| 297 | .. note:: |
| 298 | |
| 299 | This part will be done automatically once we have a DHCP and HTTP server set up in the infrastructure. |
| 300 | For now, we need to download and install the ONL image manually. |
| 301 | |
| 302 | Install ONL with Docker |
| 303 | ----------------------- |
| 304 | First, enter **ONIE rescue mode**. |
| 305 | |
| 306 | Set up IP and route |
| 307 | ^^^^^^^^^^^^^^^^^^^ |
| 308 | .. code-block:: console |
| 309 | |
| 310 | # ip addr add 10.92.1.81/24 dev eth0 |
| 311 | # ip route add default via 10.92.1.1 |
| 312 | |
| 313 | - `10.92.1.81/24` should be replaced by the actual IP and subnet of the ONL. |
| 314 | - `10.92.1.1` should be replaced by the actual default gateway. |
| 315 | |
| 316 | Download and install ONL |
| 317 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 318 | |
| 319 | .. code-block:: console |
| 320 | |
| 321 | # wget https://github.com/opennetworkinglab/OpenNetworkLinux/releases/download/v1.3.2/ONL-onf-ONLPv2_ONL-OS_2020-10-09.1741-f7428f2_AMD64_INSTALLED_INSTALLER |
| 322 | # sh ONL-onf-ONLPv2_ONL-OS_2020-10-09.1741-f7428f2_AMD64_INSTALLED_INSTALLER |
| 323 | |
| 324 | The switch will reboot automatically once the installer is done. |
| 325 | |
| 326 | .. note:: |
| 327 | |
| 328 | Alternatively, we can `scp` the ONL installer into ONIE manually. |
| 329 | |
| 330 | Setup BMC for remote console access |
| 331 | ----------------------------------- |
| 332 | Log in to the BMC from ONL by |
| 333 | |
| 334 | .. code-block:: console |
| 335 | |
| 336 | # ssh root@192.168.0.1 # pass: 0penBmc |
| 337 | |
| 338 | on `usb0` interface. |
| 339 | |
| 340 | Once you are in the BMC, run the following commands to setup IP and route (or offer a fixed IP with DHCP) |
| 341 | |
| 342 | .. code-block:: console |
| 343 | |
| 344 | # ip addr add 10.92.1.85/24 dev eth0 |
| 345 | # ip route add default via 10.92.1.1 |
| 346 | |
| 347 | - `10.92.1.85/24` should be replaced by the actual IP and subnet of the BMC. |
| 348 | Note that it should be different from the ONL IP. |
| 349 | - `10.92.1.1` should be replaced by the actual default gateway. |
| 350 | |
| 351 | BMC uses the same ethernet port as ONL management so you should give it an IP address in the same subnet. |
| 352 | BMC address will preserve during ONL reboot, but won’t be preserved during power outage. |
| 353 | |
| 354 | To log in to ONL console from BMC, run |
| 355 | |
| 356 | .. code-block:: console |
| 357 | |
| 358 | # /usr/local/bin/sol.sh |
| 359 | |
| 360 | If `sol.sh` is unresponsive, please try to restart the mainboard with |
| 361 | |
| 362 | .. code-block:: console |
| 363 | |
| 364 | # wedge_power.sh restart |
| 365 | |
| 366 | Setup network and host name for ONL |
| 367 | ----------------------------------- |
| 368 | |
| 369 | .. code-block:: console |
| 370 | |
| 371 | # hostnamectl set-hostname <host-name> |
| 372 | |
| 373 | # vim.tiny /etc/hosts # update accordingly |
| 374 | # cat /etc/hosts # example |
| 375 | 127.0.0.1 localhost |
| 376 | 10.92.1.81 menlo-staging-spine-1 |
| 377 | |
| 378 | # vim.tiny /etc/network/interfaces.d/ma1 # update accordingly |
| 379 | # cat /etc/network/interfaces.d/ma1 # example |
| 380 | auto ma1 |
| 381 | iface ma1 inet static |
| 382 | address 10.92.1.81 |
| 383 | netmask 255.255.255.0 |
| 384 | gateway 10.92.1.1 |
| 385 | dns-nameservers 8.8.8.8 |