pronto_deployment_guide/hw_installation.rst

..
   SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
   SPDX-License-Identifier: Apache-2.0

Hardware Installation
=====================

Hardware installation breaks down into a few steps:

1. `Planning`_
2. `Inventory`_
3. `Rackmount of Equipment`_
4. `Cabling and Network Topology`_
5. `Management Switch Bootstrap`_
6. `Management Server Bootstrap`_
7. `Server Software Bootstrap`_

Installation of the fabric switch hardware is covered in :ref:`OS Installation
- Switches <switch-install>`.

Installation of the radio hardware is covered in :ref:`eNB Installation
<enb-installation>`.

Planning
--------
The planning of the network topology and devices, and required cabling

Once planning is complete, equipment is ordered to match the plan.

Network Cable Plan
""""""""""""""""""

If a 2x2 TOST fabric is used it should be configured as a :doc:`Single-Stage
Leaf-Spine <trellis:supported-topology>`.

- The links between each leaf and spine switch must be made up of two separate
  cables.

- Each compute server is dual-homed via a separate cable to two different leaf
  switches (as in the "paired switches" diagrams).

If only a single P4 switch is used, the :doc:`Simple
<trellis:supported-topology>` topology is used, with two connections from each
compute server to the single switch

Additionally a non-fabric switch is required to provide a set of management
networks.  This management switch is configured with multiple VLANs to separate
the management plane, fabric, and the out-of-band and lights out management
connections on the equipment.

Device Naming
"""""""""""""

Site Design and Bookkeeping
"""""""""""""""""""""""""""

The following items need to be added to `NetBox
<https://netbox.readthedocs.io/en/stable>`_ to describe each edge site:

1. Add a Site for the edge (if one doesn't already exist), which has the
   physical location and contact information for the edge.

2. Add Racks to the Site (if they don't already exist)

3. Add a Tenant for the edge (who owns/manages it), assigned to the ``Pronto``
   or ``Aether`` Tenant Group.

4. Add a VRF (Routing Table) for the edge site.

5. Add a VLAN Group to the edge site, which groups the site's VLANs and
   prevents duplication.

6. Add VLANs for the edge site.  These should be assigned a VLAN Group, the
   Site, and Tenant.

   There can be multiple of the same VLAN in NetBox (VLANs are layer 2, and
   local to the site), but not within the VLAN group.

   The minimal list of VLANs:

     * ADMIN 1
     * UPLINK 10
     * MGMT 800
     * FAB 801

   If you have multiple deployments at a site using the same management server,
   add additional VLANs incremented by 10 for the MGMT/FAB - for example:

     * DEVMGMT 810
     * DEVFAB 801

7. Add IP Prefixes for the site. This should have the Tenant and VRF assigned.

   All edge IP prefixes fit into a ``/22`` sized block.

   The description of the Prefix contains the DNS suffix for all Devices that
   have IP addresses within this Prefix. The full DNS names are generated by
   combining the first ``<devname>`` component of the Device names with this
   suffix.

   An examples using the ``10.0.0.0/22`` block. There are 4 edge
   prefixes, with the following purposes:

     * ``10.0.0.0/25``

        * Has the Server BMC/LOM and Management Switch
        * Assign the ADMIN 1 VLAN
        * Set the description to ``admin.<deployment>.<site>.aetherproject.net`` (or
          ``prontoproject.net``).

     * ``10.0.0.128/25``

        * Has the Server Management plane, Fabric Switch Management/BMC
        * Assign MGMT 800 VLAN
        * Set the description to ``<deployment>.<site>.aetherproject.net`` (or
          ``prontoproject.net``).

     * ``10.0.1.0/25``

        * IP addresses of the qsfp0 port of the Compute Nodes to Fabric switches, devices
          connected to the Fabric like the eNB
        * Assign FAB 801 VLAN
        * Set the description to ``fab1.<deployment>.<site>.aetherproject.net`` (or
          ``prontoproject.net``).

     * ``10.0.1.128/25``

        * IP addresses of the qsfp1 port of the Compute Nodes to fabric switches
        * Assign FAB 801 VLAN
        * Set the description to ``fab2.<deployment>.<site>.aetherproject.net`` (or
          ``prontoproject.net``).

   Additionally, these edge prefixes are used for Kubernetes but don't need to
   be created in NetBox:

     * ``10.0.2.0/24``

        * Kubernetes Pod IP's

     * ``10.0.3.0/24``

        * Kubernetes Cluster IP's

8. Add Devices to the site, for each piece of equipment. These are named with a
   scheme similar to the DNS names used for the pod, given in this format::

     <devname>.<deployment>.<site>

   Examples::

     mgmtserver1.ops1.tucson
     node1.stage1.menlo

   Note that these names are transformed into DNS names using the Prefixes, and
   may have additional components - ``admin`` or ``fabric`` may be added after
   the ``<devname>`` for devices on those networks.

   Set the following fields when creating a device:

     * Site
     * Tenant
     * Rack & Rack Position
     * Serial number

   If a specific Device Type doesn't exist for the device, it must be created,
   which is detailed in the NetBox documentation, or ask the OPs team for help.

9. Add Services to the management server:

    * name: ``dns``
      protocol: UDP
      port: 53

    * name: ``tftp``
      protocol: UDP
      port: 69

   These are used by the DHCP and DNS config to know which servers offer a
   dns service and tftp.

10. Set the MAC address for the physical interfaces on the device.

   You may also need to add physical network interfaces if  aren't already
   created by the Device Type.  An example would be if additional add-in
   network cards were installed.

11. Add any virtual interfaces to the Devices. When creating a virtual
    interface, it should have it's ``label`` field set to the physical network
    interface that it is assigned

    These are needed are two cases for the Pronto deployment:

     1. On the Management Server, there should bet (at least) two VLAN
        interfaces created attached to the ``eno2`` network port, which
        are used to provide connectivity to the management plane and fabric.
        These should be named ``<name of vlan><vlan ID>``, so the MGMT 800 VLAN
        would become a virtual interface named ``mgmt800``, with the label
        ``eno2``.

     2. On the Fabric switches, the ``eth0`` port is shared between the OpenBMC
        interface and the ONIE/ONL installation.  Add a ``bmc`` virtual
        interface with a label of ``eth0`` on each fabric switch, and check the
        ``OOB Management`` checkbox.

12. Create IP addresses for the physical and virtual interfaces.  These should
    have the Tenant and VRF set.

    The Management Server should always have the first IP address in each
    range, and they should be incremental, in this order. Examples are given as
    if there was a single instance of each device - adding additional devices
    would increment the later IP addresses.

      * Management Server

          * ``eno1`` - site provided public IP address, or blank if DHCP
            provided

          * ``eno2`` - 10.0.0.1/25 (first of ADMIN) - set as primary IP
          * ``bmc`` - 10.0.0.2/25 (next of ADMIN)
          * ``mgmt800`` - 10.0.0.129/25 (first of MGMT)
          * ``fab801`` - 10.0.1.1/25 (first of FAB)

      * Management Switch

          * ``gbe1`` - 10.0.0.3/25 (next of ADMIN) - set as primary IP

      * Fabric Switch

          * ``eth0`` - 10.0.0.130/25 (next of MGMT), set as primary IP
          * ``bmc`` - 10.0.0.131/25

      * Compute Server

          * ``eth0`` - 10.0.0.132/25 (next of MGMT), set as primary IP
          * ``bmc`` - 10.0.0.4/25 (next of ADMIN)
          * ``qsfp0`` - 10.0.1.2/25 (next of FAB)
          * ``qsfp1`` - 10.0.1.3/25

      * Other Fabric devices (eNB, etc.)

          * ``eth0`` or other primary interface - 10.0.1.4/25 (next of FAB)

13. Add DHCP ranges to the IP Prefixes for IP's that aren't reserved. These are
    done like any other IP Address, but with the ``Status`` field is set to
    ``DHCP``, and they'll consume the entire range of IP addresses given in the
    CIDR mask.

    For example ``10.0.0.32/27`` as a DHCP block would take up 1/4 of the ADMIN
    prefix.

14. Add Cables between physical interfaces on the devices

    TODO: Explain the cabling topology

Hardware
""""""""

Fabric Switches
'''''''''''''''

Pronto currently uses fabric switches based on the Intel (was Barefoot) Tofino
chipset.  There are multiple variants of this switching chipset, with different
speeds and capabilities.

The specific hardware models in use in Pronto:

* `EdgeCore Wedge100BF-32X
  <https://www.edge-core.com/productsInfo.php?cls=1&cls2=180&cls3=181&id=335>`_
  - a "Dual Pipe" chipset variant, used for the Spine switches

* `EdgeCore Wedge100BF-32QS
  <https://www.edge-core.com/productsInfo.php?cls=1&cls2=180&cls3=181&id=770>`_
  - a "Quad Pipe" chipset variant, used for the Leaf switches

Compute Servers

These servers run Kubernetes and edge applications.

The requirements for these servers:

* AMD64 (aka x86-64) architecture
* Sufficient resources to run Kubernetes
* Two 40GbE or 100GbE Ethernet connections to the fabric switches
* One management 1GbE port

The specific hardware models in use in Pronto:

* `Supermicro 6019U-TRTP2
  <https://www.supermicro.com/en/products/system/1U/6019/SYS-6019U-TRTP2.cfm>`_
  1U server

* `Supermicro 6029U-TR4
  <https://www.supermicro.com/en/products/system/2U/6029/SYS-6029U-TR4.cfm>`_
  2U server

These servers are configured with:

* 2x `Intel Xeon 5220R CPUs
  <https://ark.intel.com/content/www/us/en/ark/products/199354/intel-xeon-gold-5220r-processor-35-75m-cache-2-20-ghz.html>`_,
  each with 24 cores, 48 threads
* 384GB of DDR4 Memory, made up with 12x 16GB ECC DIMMs
* 2TB of nVME Flash Storage
* 2x 6TB SATA Disk storage
* 2x 40GbE ports using an XL710QDA2 NIC

The 1U servers additionally have:

- 2x 1GbE copper network ports
- 2x 10GbE SFP+ network ports

The 2U servers have:

- 4x 1GbE copper network ports

Management Server
'''''''''''''''''

One management server is required, which must have at least two 1GbE network
ports, and runs a variety of network services to support the edge.

The model used in Pronto is a `Supermicro 5019D-FTN4
<https://www.supermicro.com/en/Aplus/system/Embedded/AS-5019D-FTN4.cfm>`_

Which is configured with:

* AMD Epyc 3251 CPU with 8 cores, 16 threads
* 32GB of DDR4 memory, in 2x 16GB ECC DIMMs
* 1TB of nVME Flash storage
* 4x 1GbE copper network ports

Management Switch
'''''''''''''''''

This switch connects the configuration interfaces and management networks on
all the servers and switches together.

In the Pronto deployment this hardware is a `HP/Aruba 2540 Series JL356A
<https://www.arubanetworks.com/products/switches/access/2540-series/>`_.

Inventory
---------

Once equipment arrives, any device needs to be recorded in inventory if it:

1. Connects to the network (has a MAC address)
2. Has a serial number
3. Isn't a subcomponent (disk, add-in card, linecard, etc.) of a larger device.

The following information should be recorded for every device:

- Manufacturer
- Model
- Serial Number
- MAC address (for the primary and any management/BMC/IPMI interfaces)

This information should be be added to the corresponding Devices ONF NetBox
instance.  The accuracy of this information is very important as it is used in
bootstrapping the systems.

Once inventory has been completed, let the Infra team know, and the pxeboot
configuration will be generated to have the OS preseed files corresponding to the
new servers based on their serial numbers.

Rackmount of Equipment
----------------------

Most of the Pronto equipment is in a 19" rackmount form factor.

Guidelines for mounting this equipment:

- The EdgeCore Wedge Switches have a front-to-back (aka "port-to-power") fan
  configuration, so hot air exhaust is out the back of the switch near the
  power inlets, away from the 32 QSFP network ports on the front of the switch.

- The full-depth 1U and 2U Supermicro servers also have front-to-back airflow
  but have most of their ports on the rear of the device.

- Airflow through the rack should be in one direction to avoid heat being
  pulled from one device into another.  This means that to connect the QSFP
  network ports from the servers to the switches, cabling should be routed
  through the rack from front (switch) to back (server).

- The short-depth management HP Switch and 1U Supermicro servers should be
  mounted to the rear of the rack.  They both don't generate an appreciable
  amount of heat, so the airflow direction isn't a significant factor in
  racking them.

Cabling and Network Topology
----------------------------

TODO: Add diagrams of network here, and cabling plan

Management Switch Bootstrap
---------------------------

TODO: Add instructions for bootstrapping management switch, from document that
has the linked config file.

Server Software Bootstrap
-------------------------

Management Server Bootstrap
"""""""""""""""""""""""""""

The management server is bootstrapped into a customized version of the standard
Ubuntu 18.04 OS installer.

The `iPXE boot firmware <https://ipxe.org/>`_. is used to start this process
and is built using the steps detailed in the `ipxe-build
<https://gerrit.opencord.org/plugins/gitiles/ipxe-build>`_. repo, which
generates both USB and PXE chainloadable boot images.

Once a system has been started using these images started, these images will
download a customized script from  an external webserver to continue the boot
process. This iPXE to webserver connection is secured with mutual TLS
authentication, enforced by the nginx webserver.

The iPXE scripts are created by the `pxeboot
<https://gerrit.opencord.org/plugins/gitiles/ansible/role/pxeboot>`_ role,
which creates both a boot menu, downloads the appropriate binaries for
bootstrapping an OS installation, and creates per-node installation preseed files.

The preseed files contain configuration steps to install the OS from the
upstream Ubuntu repos, as well as customization of packages and creating the
``onfadmin`` user.

TODO: convert instructions for bootstrapping the management server with iPXE here.

Once the OS is installed on the management server, Ansible is used to remotely
install software on the management server.

To checkout the ONF ansible repo and enter the virtualenv with the tooling::

  mkdir infra
  cd infra
  repo init -u ssh://<your gerrit username>@gerrit.opencord.org:29418/infra-manifest
  repo sync
  cd ansible
  make galaxy
  source venv_onfansible/bin/activate

Obtain the ``undionly.kpxe`` iPXE artifact for bootstrapping the compute
servers, and put it in the ``files`` directory.

Next, create an inventory file to access the NetBox API.  An example is given
in ``inventory/example-netbox.yml`` - duplicate this file and modify it. Fill
in the ``api_endpoint`` address and ``token`` with an API key you get out of
the NetBox instance.  List the IP Prefixes used by the site in the
``ip_prefixes`` list.

Next, run the ``scripts/netbox_edgeconfig.py`` to generate a host_vars file for
the management server.  Assuming that the management server in the edge is
named ``mgmtserver1.stage1.menlo``, you'd run::

  python scripts/netbox_edgeconfig.py inventory/my-netbox.yml > inventory/host_vars/mgmtserver1.stage1.menlo.yml

One manual change needs to be made to this output - edit the
``inventory/host_vars/mgmtserver1.stage1.menlo.yml`` file and add the following
to the bottom of the file, replacing the IP addresses with the ones that the
management server is configured with on each VLAN. This configures the `netplan
<https://netplan.io>`_ on the management server, and will be automated away
soon::

  # added manually
  netprep_netplan:
    ethernets:
      eno2:
        addresses:
          - 10.0.0.1/25
    vlans:
      mgmt800:
        id: 800
        link: eno2
        addresses:
          - 10.0.0.129/25
      fabr801:
        id: 801
        link: eno2
        addresses:
          - 10.0.1.1/25

Create an inventory file for the management server in
``inventory/menlo-staging.ini`` which contains::

  [mgmt]
  mgmtserver1.stage1.menlo ansible_host=<public ip address> ansible_user="onfadmin" ansible_become_password=<password>

Then, to configure a management server, run::

  ansible-playbook -i inventory/menlo-staging.ini playbooks/aethermgmt-playbook.yml

This installs software with the following functionality:

- VLANs on second Ethernet port to provide connectivity to the rest of the pod.
- Firewall with NAT for routing traffic
- DHCP and TFTP for bootstrapping servers and switches
- DNS for host naming and identification
- HTTP server for serving files used for bootstrapping switches

Compute Server Bootstrap
""""""""""""""""""""""""

Once the management server has finished installation, it will be set to offer
the same iPXE bootstrap file to the computer.

Each node will be booted, and when iPXE loads select the ``Ubuntu 18.04
Installer (fully automatic)`` option.

The nodes can be controlled remotely via their BMC management interfaces - if
the BMC is at ``10.0.0.3`` a remote user can SSH into them with::

  ssh -L 2443:10.0.0.3:443 onfadmin@<mgmt server ip>

And then use their web browser to access the BMC at::

  https://localhost:2443

The default BMC credentials for the Pronto nodes are::

  login: ADMIN
  password: Admin123

Once these nodes are brought up, the installation can continue.