pronto_deployment_guide/troubleshooting.rst

..
   SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
   SPDX-License-Identifier: Apache-2.0

Troubleshooting
===============

Unable to access a system
-------------------------

If it's a system behind another system (ex: the compute nodes behind a
management server) and you're trying to interactively login to it, make sure
that you've enabled SSH Agent Forwarding in your ``~/.ssh/config`` file::

  Host mgmtserver1.prod.site.aetherproject.net
    ForwardAgent yes

If you still have problems after verifying that this is set up, run ssh with
the ``-v`` option, which will print out all the connection details and
whether an agent is used on the second ssh::

  onfadmin@mgmtserver1:~$ ssh onfadmin@node2.mgmt.prod.site.aetherproject.net
  debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
  debug1: channel 1: new [authentication agent connection]
  debug1: confirm auth-agent@openssh.com
  Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-56-generic x86_64)
  ...
  onfadmin@node2:~$

Problems with OS installation
-----------------------------

OS installs, but doesn't boot
"""""""""""""""""""""""""""""

If you've completed the installation but the system won't start the OS, check
these BIOS settings:

- If the startup disk is nVME, under ``Advanced -> PCIe/PCI/PnP Configuration``
  the option ``NVMe Firmware Source`` should be set to ``AMI Native Support``,
  per `this Supermicro FAQ entry
  <https://supermicro.com/support/faqs/faq.cfm?faq=28248>`_.

Unknown MAC addresses
---------------------

Sometimes it's hard to find out all the MAC addresses assigned to network
cards. These can be found in a variety of ways:

1. On servers, the BMC webpage will list the built-in network card MAC
   addresses.

2. If you login to a server, ``ip link`` or ``ip addr`` will show the MAC
   address of each interface, including on add-in cards.

3. If you can login to a server but don't know the BMC IP or MAC address for
   that server, you can find it with ``sudo ipmitool lan print``.

4. If you don't have a login to the server, but can get to the management
   server, ``ip neighbor`` will show the arp table of MAC addresses known to
   that system.  It's output is unsorted  - ``ip neigh | sort`` is easier to
   read.  This can be useful for determining if there's a cabling problem -
   a device plugged into the wrong port of the management switch could show up
   in the DHCP pool range for a different segment.

Cabling issues
--------------

The system may not come up correctly if cabling isn't connected properly.
If you don't have hands-on with the cabling, here are some ways to check on the
cabling remotely:

1. On servers you can check which ports are connected with ``ip link show``::

    $ ip link show
    ...
    3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
        link/ether 3c:ec:ef:4d:55:a8 brd ff:ff:ff:ff:ff:ff
    ...
    5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
        link/ether 3c:ec:ef:4d:55:a9 brd ff:ff:ff:ff:ff:ff

  Ports that are up will show ``state UP``

2. You can determine which remote ports are connected with LLDP, assuming that
   the remote switch supports LLDP and has it enabled. This can be done with
   ``networkctl lldp``, which shows both the name and the MAC address of the
   connected switch on a per-link basis::

      $ networkctl lldp
      LINK             CHASSIS ID        SYSTEM NAME      CAPS        PORT ID           PORT DESCRIPTION
      eno1             10:4f:58:e7:d5:60 Aruba-2540-24…PP ..b........ 10                10
      eno2             10:4f:58:e7:d5:60 Aruba-2540-24…PP ..b........ 1                 1