[CORD-3070]
bridge/bond configuration playbook for fabric/management networks
Change-Id: I06d188d9640739acd9426ab92321c133e8aaeb61
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..057602b
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,2 @@
+generated-files/*
+*.retry
diff --git a/interface-config/README.md b/interface-config/README.md
new file mode 100644
index 0000000..e860009
--- /dev/null
+++ b/interface-config/README.md
@@ -0,0 +1,78 @@
+# Interface configuration
+
+CORD will require various interfaces bonded then bridged, when used with VTN.
+
+This playbook creates interfaces in the following topology for both fabric and
+management networks. The example below is for the fabric interfaces - replace
+`fabric` with `management` for all variable names for that interface:
+
+A bond interface is defined, named `fabricbond`, which contains physical
+interfaces connected to the switching fabric.
+
+A `fabricbridge` is defined, which contains:
+
+- fabricbond (containing with physical interfaces)
+- veth pairs for connecting to VM's and containers
+
+The bridge is assigned an IP address, specified by `fabric_net_ip_cidr`, which
+is an IP address in CIDR format (ex: `10.234.56.7/8`) which is used to create
+the IP address of the interface, it's network mask, broadcast address, and so
+on. The `ipcalc` tool can be useful for calculating these values.
+
+The interface configuration files are stored in
+`/etc/network/interfaces.d/management.cfg` and
+`/etc/network/interfaces.d/fabric.cfg`.
+
+Once these files are created, the interfaces they describe are brought up using
+`ifup`, and should be recreated if the system reboots.
+
+## Using the playbook
+
+### Defining an Inventory
+
+An example ansible inventory is given in `example-inventory.yaml`, and uses the
+[YAML inventory
+format](https://docs.ansible.com/ansible/latest/plugins/inventory/yaml.html)
+
+For this playbook to function, you must define one or both of the the
+`fabric_net_ip_cidr` or `management_net_ip_cidr` variables as described above
+for each host. If you don't define one of these variables, the corresponding
+bridge or bond will not be created.
+
+### Running the playbook
+
+`ansible-playbook -i <inventory_file> prep-interfaces-playbook.yaml`
+
+## Selecting interfaces
+
+There are three ways to select which interfaces are added to each bond. You may
+use these additively in any combination - they'll simply add more interfaces to
+the bond.
+
+The three methods are:
+
+- *By interface name*: Specific interface names can be listed for each bond.
+ This is useful if you have a specific configuration and know the names of
+ interfaces before you configure the system, or if you have multiple of the
+ same manufacturer of NIC. This is used by adding interface names (as listed
+ in `ansible_intefaces` when running the `setup` task) to the
+ `(fabric|management)_net_interfaces` list.
+
+- *By kernel module name*: By creating a list of kernel module names for the
+ bond device, interfaces can be selected. For example, every interface
+ using the `em` or `mlx4` driver might be in the `fabric` bond. The specific
+ driver names can be found by running ansible's setup module on a host, and
+ looking through the output. For example, if you have an interface on the
+ system named `eth0`, the driver name would befound in `ansible_eth0.module`.
+ This is used by adding kernel module names to the
+ `(fabric|management)_net_kmods` list.
+
+- *By hardware (MAC) address*: The MAC addresses of the interfaces can be
+ listed. This can be useful in systems where the intefaces change names
+ through kernel or driver updates. This is used by adding 48-bit hardware
+ addresses in the lowercase, zero padded, colon delimited hex format
+ (`01:23:45:67:89:ab`) to the `(fabric|management)_net_hwaddrs` list.
+
+These variables are optional, and set on a per-host basis. Not setting them
+will result in an empty bond in a bridge.
+
diff --git a/interface-config/example_inventory.yaml b/interface-config/example_inventory.yaml
new file mode 100644
index 0000000..992bd94
--- /dev/null
+++ b/interface-config/example_inventory.yaml
@@ -0,0 +1,17 @@
+# Example inventory file for network configuration
+
+all:
+ hosts:
+
+ host1.example.dns:
+ management_net_ip_cidr: "10.123.45.6/24"
+ fabric_net_ip_cidr: "10.234.5.6/24"
+ management_net_interfaces: ['eth2']
+ fabric_net_interfaces: ['eth3']
+
+ host2.example.dns:
+ management_net_ip_cidr: "10.123.45.7/24"
+ fabric_net_ip_cidr: "10.234.5.7/24"
+ management_net_hwaddrs: ['52:54:00:28:44:be']
+ fabric_net_kmods: ['mlx4_core']
+
diff --git a/interface-config/prep-interfaces-playbook.yaml b/interface-config/prep-interfaces-playbook.yaml
new file mode 100644
index 0000000..213f10b
--- /dev/null
+++ b/interface-config/prep-interfaces-playbook.yaml
@@ -0,0 +1,24 @@
+---
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# prep-interfaces-playbook.yml
+# Preps network interfaces and bridges/bonds for CORD
+
+- name: Configure network interfaces
+ hosts: all
+ become: yes
+ roles:
+ - interface-config
+
diff --git a/interface-config/roles/interface-config/defaults/main.yml b/interface-config/roles/interface-config/defaults/main.yml
new file mode 100644
index 0000000..9459236
--- /dev/null
+++ b/interface-config/roles/interface-config/defaults/main.yml
@@ -0,0 +1,29 @@
+---
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# interface-config/defaults/main.yml
+
+# management network configuration
+management_net_ip_cidr: ""
+management_net_interfaces: []
+management_net_kmods: []
+management_net_hwaddrs: []
+
+# fabric network configuration
+fabric_net_ip_cidr: ""
+fabric_net_interfaces: []
+fabric_net_kmods: []
+fabric_net_hwaddrs: []
+
diff --git a/interface-config/roles/interface-config/tasks/main.yml b/interface-config/roles/interface-config/tasks/main.yml
new file mode 100644
index 0000000..458d0c1
--- /dev/null
+++ b/interface-config/roles/interface-config/tasks/main.yml
@@ -0,0 +1,126 @@
+---
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# interface-config/tasks/main.yml
+
+- name: Configure management network if management_net_ip_cidr is set/valid
+ when: management_net_ip_cidr | ipaddr
+ set_fact:
+ configure_management_interfaces: true
+
+- name: Configure fabric network if fabric_net_ip_cidr is set/valid
+ when: fabric_net_ip_cidr | ipaddr
+ set_fact:
+ configure_fabric_interfaces: true
+
+- name: Install bridging/bonding utilities
+ apt:
+ name: "{{ item }}"
+ update_cache: yes
+ cache_valid_time: 3600
+ with_items:
+ - bridge-utils
+ - ifenslave
+
+# management network
+- name: Find management interfaces given management_net_kmods
+ when: ( configure_management_interfaces ) and ( management_net_kmods | length > 0 )
+ set_fact:
+ management_net_interfaces: "{{ [ hostvars[inventory_hostname]['ansible_' ~ item[0]] ] | selectattr('module', 'defined') | selectattr('module', 'equalto', item[1]) | map(attribute='device') | list | union(management_net_interfaces) }}"
+ with_nested:
+ - "{{ ansible_interfaces }}"
+ - "{{ management_net_kmods }}"
+
+- name: Find management interfaces given management_net_hwaddrs
+ when: ( configure_management_interfaces ) and ( management_net_hwaddrs | length > 0 )
+ set_fact:
+ management_net_interfaces: "{{ [ hostvars[inventory_hostname]['ansible_' ~ item[0]] ] | selectattr('macaddress', 'defined') | selectattr('macaddress', 'equalto', item[1]) | map(attribute='device') | list | union(management_net_interfaces) }}"
+ with_nested:
+ - "{{ ansible_interfaces }}"
+ - "{{ management_net_hwaddrs }}"
+
+# fabric network
+- name: Find fabric interfaces given fabric_net_kmods
+ when: ( configure_fabric_interfaces ) and ( fabric_net_kmods | length > 0 )
+ set_fact:
+ fabric_net_interfaces: "{{ [ hostvars[inventory_hostname]['ansible_' ~ item[0]] ] | selectattr('module', 'defined') | selectattr('module', 'equalto', item[1]) | map(attribute='device') | list | union(fabric_net_interfaces) }}"
+ with_nested:
+ - "{{ ansible_interfaces }}"
+ - "{{ fabric_net_kmods }}"
+
+- name: Find fabric interfaces given fabric_net_hwaddrs
+ when: ( configure_fabric_interfaces ) and ( fabric_net_hwaddrs | length > 0 )
+ set_fact:
+ fabric_net_interfaces: "{{ [ hostvars[inventory_hostname]['ansible_' ~ item[0]] ] | selectattr('macaddress', 'defined') | selectattr('macaddress', 'equalto', item[1]) | map(attribute='device') | list | union(fabric_net_interfaces) }}"
+ with_nested:
+ - "{{ ansible_interfaces }}"
+ - "{{ fabric_net_hwaddrs }}"
+
+# all interfaces discovered
+- name: Print lists of interfaces
+ debug:
+ msg: "{{ item }}"
+ with_items:
+ - "Ansible detected interfaces: {{ ansible_interfaces | join(', ') }}"
+ - "Proposed management network interfaces: {{ management_net_interfaces | join(', ') }}"
+ - "Proposed fabric network interfaces: {{ fabric_net_interfaces | join(', ') }}"
+
+- name: Make sure all interfaces exist, and aren't shared between lists
+ assert:
+ that:
+ - fabric_net_interfaces | difference(ansible_interfaces) | length == 0
+ - management_net_interfaces | difference(ansible_interfaces) | length == 0
+ - management_net_interfaces | intersect(fabric_net_interfaces) | length == 0
+
+- name: Configure management network interfaces
+ when: configure_management_interfaces | default(false)
+ template:
+ src: management.cfg.j2
+ dest: "/etc/network/interfaces.d/management.cfg"
+ owner: root
+ group: root
+ mode: 0644
+ register: management_net_config
+
+- name: Configure fabric network interfaces
+ when: configure_fabric_interfaces | default(false)
+ template:
+ src: fabric.cfg.j2
+ dest: "/etc/network/interfaces.d/fabric.cfg"
+ owner: root
+ group: root
+ mode: 0644
+ register: fabric_net_config
+
+- name: Bring up management network interfaces, if reconfigured
+ when: management_net_config.changed
+ command: "ifup {{ item }}"
+ with_flattened:
+ - mgmtbridge
+ - mgmtbond
+ - "{{ management_net_interfaces }}"
+ tags:
+ - skip_ansible_lint # needs to be run before next steps
+
+- name: Bring up fabric network interfaces, if reconfigured
+ when: fabric_net_config.changed
+ command: "ifup {{ item }}"
+ with_flattened:
+ - fabricbridge
+ - fabricbond
+ - "{{ fabric_net_interfaces }}"
+ tags:
+ - skip_ansible_lint # needs to be run before next steps
+
diff --git a/interface-config/roles/interface-config/templates/fabric.cfg.j2 b/interface-config/roles/interface-config/templates/fabric.cfg.j2
new file mode 100644
index 0000000..c67fa79
--- /dev/null
+++ b/interface-config/roles/interface-config/templates/fabric.cfg.j2
@@ -0,0 +1,50 @@
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# CORD Fabric interface configuration
+# # created by automation-tools template fabric.cfg.j2
+
+# fabric bridge between physical and virtual interfaces for VTN
+auto fabricbridge
+iface fabricbridge inet static
+ pre-up ip link add fabricbridge type bridge
+ address {{ fabric_net_ip_cidr | ipaddr('address') }}
+ network {{ fabric_net_ip_cidr | ipaddr('network') }}
+ netmask {{ fabric_net_ip_cidr | ipaddr('netmask') }}
+ broadcast {{ fabric_net_ip_cidr | ipaddr('broadcast') }}
+ bridge_ports fabricbond
+ post-down ip link del fabricbridge
+
+# management bond of physical intefaces
+auto fabricbond
+iface fabricbond inet manual
+ pre-up ip link add fabricbond type bond
+ pre-up ip link set fabricbond up
+ bond-miimon 100
+ bond-slaves none
+ bond-mode active-backup
+ post-down ip link del fabricbond
+
+{% if fabric_net_interfaces %}
+# physical network members of fabricbond
+{% for fab_int in fabric_net_interfaces %}
+auto {{ fab_int }}
+iface {{ fab_int }} inet manual
+ pre-up ip link set {{ fab_int }} master fabricbond
+ bond-master fabricbond
+ bond-primary {{ fabric_net_interfaces | join(' ') }}
+ post-down ip link set dev {{ fab_int }} nomaster
+
+{% endfor %}
+{% endif %}
diff --git a/interface-config/roles/interface-config/templates/management.cfg.j2 b/interface-config/roles/interface-config/templates/management.cfg.j2
new file mode 100644
index 0000000..950a64c
--- /dev/null
+++ b/interface-config/roles/interface-config/templates/management.cfg.j2
@@ -0,0 +1,51 @@
+# Copyright 2017-present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# CORD Management interface configuration
+# created by automation-tools template management.cfg.j2
+
+# management bridge between physical and virtual interfaces for VTN
+auto mgmtbridge
+iface mgmtbridge inet static
+ pre-up ip link add mgmtbridge type bridge
+ bridge_ports mgmtbond vethmgmt0
+ address {{ management_net_ip_cidr | ipaddr('address') }}
+ network {{ management_net_ip_cidr | ipaddr('network') }}
+ netmask {{ management_net_ip_cidr | ipaddr('netmask') }}
+ broadcast {{ management_net_ip_cidr | ipaddr('broadcast') }}
+ post-down ip link del mgmtbridge
+
+# management bond of physical interfaces
+auto mgmtbond
+iface mgmtbond inet manual
+ pre-up ip link add mgmtbond type bond
+ pre-up ip link set mgmtbond up
+ bond-miimon 100
+ bond-slaves none
+ bond-mode active-backup
+ post-down ip link del mgmtbond
+
+{% if management_net_interfaces %}
+# physical network members of mgmtbond
+{% for mgmt_int in management_net_interfaces %}
+auto {{ mgmt_int }}
+iface {{ mgmt_int }} inet manual
+ pre-up ip link set {{ mgmt_int }} master mgmtbond
+ bond-master mgmtbond
+ bond-primary {{ management_net_interfaces | join(' ') }}
+ post-down ip link set dev {{ mgmt_int }} nomaster
+
+{% endfor %}
+{% endif %}
+