blob: 4dc03cb7cde41903e5ec08af016b60b61bc1022e [file] [log] [blame]
Matteo Scandolo1f5530b2019-12-17 10:12:31 -08001Troubleshooting
2===============
3
4Here is a collection of useful commands and tools to troubleshoot VOLTHA.
5The project is still in a very early phase, so this section of the guide is more focus
6on how to collection information and packet dumps than it is on operations.
7
Andrea Campanella61fd6662020-07-27 16:56:55 +02008Depending on your issue useful tips and tricks can be found:
Andrea Campanella764a3ce2021-07-19 21:55:02 +02009
Hardik Windlass702b5fa2022-06-29 17:27:01 +053010- `No ONUs are discovered <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-onus.html>`_
11- `Can't authenticate <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-aaa.html>`_
12- `DHCP is not working <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-dhcp.html>`_
13- `No Ping <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-ping.html>`_
Andrea Campanella61fd6662020-07-27 16:56:55 +020014
15
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080016Trace Packets across components
17-------------------------------
18
19Given that ``VOLTHA`` is a collection of microservices we strongly suggest the usage of a log aggregator like
20`stern <https://github.com/wercker/stern>`_. installed. You can then aggregate logs from multiple containers in a simple way,
21for example to track all the ``packet-ins`` and ``packet-outs``:
22
23.. code:: bash
24
25 $ stern -n voltha "voltha|adapter" | grep -E "packet-out|packet-in" > packets.trace
26
27Once you have captured the packets you need, you can see them in wireshark by transforming the logs with:
28
29.. code:: bash
30
31 sed -n 's/.*packet":"\(.*\)",.*$/\1/p' packets.trace | sed -e 's/.\{2\}/& /g' | sed -e 's/^/000000 /' > packets.hex
32
33And then in wireshark select ``File -> Import from Hex Dump``
34
35Or you can decode a single packet using this online tool: `https://hpd.gasmi.net <https://hpd.gasmi.net>`_
36
37To get all the EAPOL packets: ``cat packets.trace | grep 888e``
38
39or if you have `hl <https://github.com/mbornet-hl/hl>`__ installed you can highlight with:
40
41.. code:: bash
42
43 cat packets.trace | grep 888e | hl -m '.*packet-in.*' -g '.*packet-out.*'
44
45To get all the DHCP packets: ``cat packets.trace | grep 8100``
46
47or if you have `hl <https://github.com/mbornet-hl/hl>`__ installed you can highlight with:
48
49.. code:: bash
50
51 cat packets.trace | grep 8100 | hl -m '.*packet-in.*' -g '.*packet-out.*'
52
53OMCI Packet Dump
54----------------
55
56In order to see OMCI packets in Wireshark you'll need to install ``omci.lua`` and ``BinDecHex.lua`` dissectors
Andrea Campanellaf31c4fd2021-12-28 11:40:03 +010057`wiki.wireshark.org/Contrib <https://wiki.wireshark.org/Contrib#protocol-dissectors>`_.
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080058
59To install them on Ubuntu:
60
61.. code:: bash
62
63 mkdir -p $(HOME)/.local/lib/wireshark/plugins
64 cd $(HOME)/.local/lib/wireshark/plugins
65 wget "https://wiki.wireshark.org/Contrib?action=AttachFile&do=get&target=omci.lua"
66 wget "https://wiki.wireshark.org/Contrib?action=AttachFile&do=get&target=BinDecHex.lua"
67
68IMPORTANT - Apply `this <https://ask.wireshark.org/question/4557/bindechexlua-error-bad-argument-to-module-packageseeall/?answer=4573#post-id-4573>`_ fix to BinDecHex.lua.
69
70To capture the OMCI packets for all ONUs:
71
72.. code:: bash
73
Andrea Campanella9f534c72021-03-19 18:44:27 +010074 cat openonu.logs | grep -E "TxOmciMessage|RxOmciMessage" | grep msg > packets.trace
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080075
76To capture the OMCI packets for a particular ONU:
77
78.. code:: bash
79
Andrea Campanella9f534c72021-03-19 18:44:27 +010080 cat openonu.logs | grep -E "TxOmciMessage|RxOmciMessage" | grep msg | grep [deviceId] > packets.trace
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080081
82Once you have the ``omci.dump`` file you need to prepare it to be imported in wireshark with this command:
83
84.. code:: bash
85
Andrea Campanella9f534c72021-03-19 18:44:27 +010086 awk -F"OmciMessage" '/OmciMessage/{print $2}' packets.trace | cut -f3 -d'"' > rawdump
87 cat rawdump | sed -e 's/.\{2\}/& /g' | sed -e 's/^/000000 /' > omci.hex
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080088
89And then in wireshark:
90
91- select ``File -> Import from Hex Dump``
92- select ``Encapsulation Type -> Ethernet``
93- set ``Ethernet -> Ethertype (hex): 88b5``