blob: 38a27e46c5b2772d94d3f85cdb9df9b9ce401968 [file] [log] [blame]
Zack Williams794532a2021-03-18 17:38:36 -07001..
2 SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
3 SPDX-License-Identifier: Apache-2.0
4
5VPN Bootstrap
6=============
7
Hyunsun Moondb739a32021-08-12 01:59:21 -07008This section guides you through setting up a VPN connection between Aether
9Central in GCP and ACE.
Hyunsun Moon049b5052021-07-30 12:41:03 -070010We will be using GitOps based Aether CI/CD system for this and what you need to do is
Zack Williams1ae109e2021-07-27 11:17:04 -070011create a patch for the new edge in ``aether-pod-configs``, where all edge infrastructure
Hyunsun Moonbbad67d2021-08-12 00:47:27 -070012configuration is stored.
13
14Here is a brief overview of each step. Note that some of the steps described here are not
15directly related to setting up a VPN, but are prerequisites for adding a new edge.
16
17**1. Add deployment jobs**
Hyunsun Moondb739a32021-08-12 01:59:21 -070018Each edge has its own Jenkins jobs that build and execute an infrastructure change plan
19based on the configurations specified in aether-pod-configs.
20In this step, you'll add those jobs to Aether CI/CD system for the new edge.
Hyunsun Moonbbad67d2021-08-12 00:47:27 -070021
22**2. Update global resource maps**
23aether-pod-configs maintains complete list of clusters, VPN connections, and users
24in separate global resource files. Before adding edge specific configurations,
25it is required to update those global resource maps first.
26
27**3. Generate Ansible and Terraform configs**
28In this step, you'll add Ansible and Terraform configs necessary to install and
Zack Williams1ae109e2021-07-27 11:17:04 -070029configure VPN software at the edge and set up VPN gateway, router, and firewall
30on GCP.
Hyunsun Moonbbad67d2021-08-12 00:47:27 -070031
32**4. Submit your changes**
Hyunsun Moondb739a32021-08-12 01:59:21 -070033Finally, submit your aether-pod-configs changes to run the deployment job added
34in the first step.
Hyunsun Moonbbad67d2021-08-12 00:47:27 -070035
36.. attention::
37
38 If you are adding another ACE to an existing VPN connection, go to
Hyunsun Moondb739a32021-08-12 01:59:21 -070039 :ref:`Add ACE to an existing VPN connection <add_ace_to_vpn>`.
40
41.. attention::
42
43 Make sure that UDP port 500, UDP port 4500, and ESP from **gcpvpn1.infra.aetherproject.net(35.242.47.15)**
44 and **gcpvpn2.infra.aetherproject.net(34.104.68.78)** are allowed in the firewall at the edge.
Zack Williams794532a2021-03-18 17:38:36 -070045
Hyunsun Moon049b5052021-07-30 12:41:03 -070046.. _add_deployment_jobs:
47
Hyunsun Moona703edf2021-07-29 15:55:15 -070048Add deployment jobs
49-------------------
Hyunsun Moonfabe9742021-08-01 06:41:44 -070050First, you need to add Jenkins jobs to Aether CI/CD system that build and apply infrastructure change
Hyunsun Moon049b5052021-07-30 12:41:03 -070051plans for the new edge. This can be done by creating a patch to **aether-ci-management** repository.
Zack Williams794532a2021-03-18 17:38:36 -070052
Hyunsun Moona703edf2021-07-29 15:55:15 -070053Download **aether-ci-management** repository.
Zack Williams794532a2021-03-18 17:38:36 -070054
Hyunsun Moona703edf2021-07-29 15:55:15 -070055.. code-block:: shell
56
57 $ cd $WORKDIR
58 $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-ci-management"
59
Hyunsun Moon2009a192021-10-29 13:46:52 -070060Add the jobs for the new cluster at the end of the ``cd-pipeline-terraform-ace-prd`` project job list.
Hyunsun Moona703edf2021-07-29 15:55:15 -070061Make sure to add both pre-merge and post-merge jobs.
62Note that the cluster name specified here will be used in the rest of the deployment procedure.
63
64.. code-block:: diff
65
66 $ cd $WORKDIR/aether-ci-management
67 $ vi jjb/repos/cd-pipeline-terraform.yaml
68
69 # Add jobs for the new cluster
70 diff jjb/repos/cd-pipeline-terraform.yamll
71 --- a/jjb/repos/cd-pipeline-terraform.yaml
72 +++ b/jjb/repos/cd-pipeline-terraform.yaml
73 @@ -227,3 +227,9 @@
74 - 'cd-pipeline-terraform-postmerge-cluster':
Hyunsun Moona703edf2021-07-29 15:55:15 -070075 cluster: 'ace-eks'
76 + - 'cd-pipeline-terraform-premerge-cluster':
Hyunsun Moona703edf2021-07-29 15:55:15 -070077 + cluster: 'ace-test'
Hyunsun Moon2009a192021-10-29 13:46:52 -070078 + disable-job: false
Hyunsun Moona703edf2021-07-29 15:55:15 -070079 + - 'cd-pipeline-terraform-postmerge-cluster':
Hyunsun Moona703edf2021-07-29 15:55:15 -070080 + cluster: 'ace-test'
Hyunsun Moon2009a192021-10-29 13:46:52 -070081 + disable-job: false
Hyunsun Moona703edf2021-07-29 15:55:15 -070082
Hyunsun Moon2b2bf9a2021-08-01 05:29:48 -070083Submit your change and wait for the jobs you just added available in Aether Jenkins.
Hyunsun Moona703edf2021-07-29 15:55:15 -070084
85.. code-block:: shell
86
87 $ git status
88 Changes not staged for commit:
89
90 modified: jjb/repos/cd-pipeline-terraform.yaml
91
92 $ git add .
93 $ git commit -m "Add test ACE deployment job"
94 $ git review
95
Zack Williams794532a2021-03-18 17:38:36 -070096
Andy Bavier802cce52021-07-29 12:15:15 -070097Get access to encrypted files in aether-pod-configs repository
98--------------------------------------------------------------
99
Andy Bavier718fd902021-07-30 10:10:55 -0700100`git-crypt <https://github.com/AGWA/git-crypt>`_ is used to securely store encrypted files
101in the aether-pod-configs repository. Before proceeding, (1) install git-crypt and `gpg <https://gnupg.org/>`_,
Andy Bavier6fc0ff12021-08-02 09:29:25 -0700102(2) create a GPG keypair, and (3) ask a member of the Aether OPs team to add your public key
Andy Bavier802cce52021-07-29 12:15:15 -0700103to the aether-pod-configs keyring. To create the keypair follow these steps:
104
105.. code-block:: shell
106
107 $ gpg --full-generate-key
108 $ gpg --output <key-name>.gpg --armor --export <your-email-address>
109
Zack Williams794532a2021-03-18 17:38:36 -0700110.. _update_global_resource:
111
112Update global resource maps
113---------------------------
114
Hyunsun Moona703edf2021-07-29 15:55:15 -0700115Download aether-pod-configs repository.
116
117.. code-block:: shell
118
119 $ cd $WORKDIR
120 $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-pod-configs"
Andy Bavier802cce52021-07-29 12:15:15 -0700121 $ git-crypt unlock
Hyunsun Moona703edf2021-07-29 15:55:15 -0700122
Hyunsun Moonfabe9742021-08-01 06:41:44 -0700123Add the new cluster information at the end of the following global resource maps.
Zack Williams794532a2021-03-18 17:38:36 -0700124
Zack Williams794532a2021-03-18 17:38:36 -0700125* ``cluster_map.tfvars``
126* ``vpn_map.tfvars``
127
Zack Williams794532a2021-03-18 17:38:36 -0700128.. code-block:: diff
129
130 $ cd $WORKDIR/aether-pod-configs/production
Zack Williams794532a2021-03-18 17:38:36 -0700131 $ vi cluster_map.tfvars
132
133 # Add the new K8S cluster information at the end of the map
134 $ git diff cluster_map.tfvars
135 --- a/production/cluster_map.tfvars
136 +++ b/production/cluster_map.tfvars
137 @@ cluster_map = {
138 kube_dns_cluster_ip = "10.53.128.10"
139 cluster_domain = "prd.menlo.aetherproject.net"
140 calico_ip_detect_method = "can-reach=www.google.com"
141 + },
142 + ace-test = {
143 + cluster_name = "ace-test"
Hyunsun Moona703edf2021-07-29 15:55:15 -0700144 + management_subnets = ["10.32.4.0/24"]
Zack Williams794532a2021-03-18 17:38:36 -0700145 + k8s_version = "v1.18.8-rancher1-1"
Hyunsun Moona703edf2021-07-29 15:55:15 -0700146 + k8s_pod_range = "10.33.0.0/17"
147 + k8s_cluster_ip_range = "10.33.128.0/17"
148 + kube_dns_cluster_ip = "10.33.128.10"
Zack Williams794532a2021-03-18 17:38:36 -0700149 + cluster_domain = "prd.test.aetherproject.net"
150 + calico_ip_detect_method = "can-reach=www.google.com"
151 }
152 }
153 }
154
155.. code-block:: diff
156
157 $ cd $WORKDIR/aether-pod-configs/production
158 $ vi vpn_map.tfvars
159
160 # Add VPN and tunnel information at the end of the map
161 $ git diff vpn_map.tfvars
162 --- a/production/vpn_map.tfvars
163 +++ b/production/vpn_map.tfvars
164 @@ vpn_map = {
165 bgp_peer_ip_address_1 = "169.254.0.6"
166 bgp_peer_ip_range_2 = "169.254.1.5/30"
167 bgp_peer_ip_address_2 = "169.254.1.6"
168 + },
169 + ace-test = {
170 + peer_name = "production-ace-test"
Hyunsun Moona703edf2021-07-29 15:55:15 -0700171 + peer_vpn_gateway_address = "66.201.42.222"
Zack Williams794532a2021-03-18 17:38:36 -0700172 + tunnel_shared_secret = "UMAoZA7blv6gd3IaArDqgK2s0sDB8mlI"
173 + bgp_peer_asn = "65003"
174 + bgp_peer_ip_range_1 = "169.254.0.9/30"
175 + bgp_peer_ip_address_1 = "169.254.0.10"
176 + bgp_peer_ip_range_2 = "169.254.1.9/30"
177 + bgp_peer_ip_address_2 = "169.254.1.10"
178 }
179 }
180
181.. note::
Hyunsun Moonbbad67d2021-08-12 00:47:27 -0700182 Use `this site <https://cloud.google.com/network-connectivity/docs/vpn/how-to/generating-pre-shared-key/>`_
183 to generate a strong tunnel shared secret.
184
185.. note::
Hyunsun Moon35868bc2021-08-11 15:01:13 -0700186 Unless you have a specific requirement, set ASN to the next available value in the map.
187 For BGP peer IP range and address, use the next available /30 subnet in the map.
Zack Williams794532a2021-03-18 17:38:36 -0700188
189
Hyunsun Moonbbad67d2021-08-12 00:47:27 -0700190Generate Ansible and Terraform configurations
191---------------------------------------------
Zack Williams794532a2021-03-18 17:38:36 -0700192
Hyunsun Moonfabe9742021-08-01 06:41:44 -0700193In this step, we will create a directory under ``production`` with the same name
Hyunsun Moonbbad67d2021-08-12 00:47:27 -0700194as the cluster, and add Ansible and Terraform configurations needed
195to configure a VPN in ACE and GCP using a tool.
Zack Williams794532a2021-03-18 17:38:36 -0700196
197.. code-block:: shell
198
199 $ cd $WORKDIR/aether-pod-configs/tools
Hyunsun Moona703edf2021-07-29 15:55:15 -0700200 $ cp ace_config.yaml.example ace_config.yaml
Hyunsun Moonbbad67d2021-08-12 00:47:27 -0700201
202 # Set all values in ace_config.yaml
Hyunsun Moona703edf2021-07-29 15:55:15 -0700203 $ vi ace_config.yaml
Zack Williams794532a2021-03-18 17:38:36 -0700204
Zack Williams794532a2021-03-18 17:38:36 -0700205 $ make vpn
206 Created ../production/ace-test
Hyunsun Moon049b5052021-07-30 12:41:03 -0700207 Created ../production/ace-test/provider.tf
208 Created ../production/ace-test/cluster.tf
Zack Williams794532a2021-03-18 17:38:36 -0700209 Created ../production/ace-test/gcp_ha_vpn.tf
Hyunsun Moon049b5052021-07-30 12:41:03 -0700210 Created ../production/ace-test/gcp_fw.tf
Zack Williams794532a2021-03-18 17:38:36 -0700211 Created ../production/ace-test/backend.tf
212 Created ../production/ace-test/cluster_val.tfvars
Hyunsun Moon049b5052021-07-30 12:41:03 -0700213 Created ../production/ace-test/ansible
Zack Williams794532a2021-03-18 17:38:36 -0700214 Created ../production/ace-test/ansible/hosts.ini
215 Created ../production/ace-test/ansible/extra_vars.yml
216
Zack Williams794532a2021-03-18 17:38:36 -0700217
Hyunsun Moon2b2bf9a2021-08-01 05:29:48 -0700218Submit your change
Hyunsun Moona703edf2021-07-29 15:55:15 -0700219------------------
Zack Williams794532a2021-03-18 17:38:36 -0700220
221.. code-block:: shell
222
223 $ cd $WORKDIR/aether-pod-configs/production
224 $ git status
225 On branch tools
226 Changes not staged for commit:
227
228 modified: cluster_map.tfvars
Zack Williams794532a2021-03-18 17:38:36 -0700229 modified: vpn_map.tfvars
230
231 Untracked files:
232 (use "git add <file>..." to include in what will be committed)
233
234 ace-test/
235
236 $ git add .
237 $ git commit -m "Add test ACE"
238 $ git review
239
Hyunsun Moonbbad67d2021-08-12 00:47:27 -0700240Wait for a while until the post-merge job finishes after the change is merged.
Zack Williams794532a2021-03-18 17:38:36 -0700241
242Verify VPN connection
243---------------------
244
Hyunsun Moonfabe9742021-08-01 06:41:44 -0700245You can verify the VPN connections by checking
Hyunsun Moonbbad67d2021-08-12 00:47:27 -0700246the routing table from the management server and trying to ping to one of the
Zack Williams794532a2021-03-18 17:38:36 -0700247central cluster VMs.
248
Zack Williams1ae109e2021-07-27 11:17:04 -0700249Be sure there are two tunnel interfaces, ``gcp_tunnel1`` and ``gcp_tunnel2``,
Hyunsun Moonbbad67d2021-08-12 00:47:27 -0700250and three additional routing entries via one of the tunnel interfaces.
Zack Williams794532a2021-03-18 17:38:36 -0700251
252.. code-block:: shell
253
254 # Verify routings
255 $ netstat -rn
256 Kernel IP routing table
257 Destination Gateway Genmask Flags MSS Window irtt Iface
Hyunsun Moona703edf2021-07-29 15:55:15 -0700258 0.0.0.0 66.201.42.209 0.0.0.0 UG 0 0 0 eno1
259 10.32.4.0 0.0.0.0 255.255.255.128 U 0 0 0 eno2
260 10.32.4.128 0.0.0.0 255.255.255.128 U 0 0 0 mgmt800
Zack Williams794532a2021-03-18 17:38:36 -0700261 10.45.128.0 169.254.0.9 255.255.128.0 UG 0 0 0 gcp_tunnel1
262 10.52.128.0 169.254.0.9 255.255.128.0 UG 0 0 0 gcp_tunnel1
Hyunsun Moona703edf2021-07-29 15:55:15 -0700263 10.33.128.0 10.32.4.138 255.255.128.0 UG 0 0 0 mgmt800
Zack Williams794532a2021-03-18 17:38:36 -0700264 10.168.0.0 169.254.0.9 255.255.240.0 UG 0 0 0 gcp_tunnel1
Hyunsun Moona703edf2021-07-29 15:55:15 -0700265 66.201.42.208 0.0.0.0 255.255.252.0 U 0 0 0 eno1
Zack Williams794532a2021-03-18 17:38:36 -0700266 169.254.0.8 0.0.0.0 255.255.255.252 U 0 0 0 gcp_tunnel1
267 169.254.1.8 0.0.0.0 255.255.255.252 U 0 0 0 gcp_tunnel2
268
269 # Verify ACC VM access
270 $ ping 10.168.0.6
271
Hyunsun Moona703edf2021-07-29 15:55:15 -0700272 # Verify ACC K8S Service access
Zack Williams794532a2021-03-18 17:38:36 -0700273 $ nslookup kube-dns.kube-system.svc.prd.acc.gcp.aetherproject.net 10.52.128.10
274
Hyunsun Moonfabe9742021-08-01 06:41:44 -0700275You can also login to GCP console and check if the edge subnets exist in
276**VPC Network > Routes > Dynamic**.
Zack Williams794532a2021-03-18 17:38:36 -0700277
278
279Post VPN setup
280--------------
281
Hyunsun Moonfabe9742021-08-01 06:41:44 -0700282Once you verify the VPN connections, update ``ansible`` directory name to
283``_ansible`` to prevent the ansible playbook from being rerun.
Zack Williams794532a2021-03-18 17:38:36 -0700284
285.. code-block:: shell
286
287 $ cd $WORKDIR/aether-pod-configs/production/$ACE_NAME
288 $ mv ansible _ansible
289 $ git add .
Hyunsun Moona703edf2021-07-29 15:55:15 -0700290 $ git commit -m "Ansible done for test ACE"
Zack Williams794532a2021-03-18 17:38:36 -0700291 $ git review
292
293.. _add_ace_to_vpn:
294
295Add another ACE to an existing VPN connection
296"""""""""""""""""""""""""""""""""""""""""""""
297
298VPN connections can be shared when there are multiple ACE clusters in a site.
Hyunsun Moonfabe9742021-08-01 06:41:44 -0700299In order to add another cluster to an existing VPN connection, you'll have to SSH into the
Zack Williams794532a2021-03-18 17:38:36 -0700300management node and manually update BIRD configuration.
301
302.. note::
303
304 This step needs improvements in the future.
305
306.. code-block:: shell
307
308 $ sudo vi /etc/bird/bird.conf
309 protocol static {
Hyunsun Moona703edf2021-07-29 15:55:15 -0700310 # Routings for the existing cluster
Zack Williams794532a2021-03-18 17:38:36 -0700311 ...
Hyunsun Moona703edf2021-07-29 15:55:15 -0700312 route 10.33.128.0/17 via 10.32.4.138;
Zack Williams794532a2021-03-18 17:38:36 -0700313
314 # Add routings for the new ACE's K8S cluster IP range via cluster nodes
315 # TODO: Configure iBGP peering with Calico nodes and dynamically learn these routings
316 route <NEW-ACE-CLUSTER-IP> via <SERVER1>
317 route <NEW-ACE-CLUSTER-IP> via <SERVER2>
318 route <NEW-ACE-CLUSTER-IP> via <SERVER3>
319 }
320
321 filter gcp_tunnel_out {
322 # Add the new ACE's K8S cluster IP range and the management subnet if required to the list
Hyunsun Moona703edf2021-07-29 15:55:15 -0700323 if (net ~ [ 10.32.4.0/24, 10.33.128.0/17, <NEW-ACE-CLUSTER-MGMT-SUBNET>, <NEW-ACE-CLUSTER-IP-RANGE> ]) then accept;
Zack Williams794532a2021-03-18 17:38:36 -0700324 else reject;
325 }
326 # Save and exit
327
328 $ sudo birdc configure
329
330 # Confirm the static routes are added
331 $ sudo birdc show route
332