blob: 3ff63f17a36d3bf4414adf7a04367a743ab0da44 [file] [log] [blame]
Matteo Scandolo1f5530b2019-12-17 10:12:31 -08001Troubleshooting
2===============
3
4Here is a collection of useful commands and tools to troubleshoot VOLTHA.
5The project is still in a very early phase, so this section of the guide is more focus
6on how to collection information and packet dumps than it is on operations.
7
Andrea Campanella61fd6662020-07-27 16:56:55 +02008Depending on your issue useful tips and tricks can be found:
Andrea Campanella764a3ce2021-07-19 21:55:02 +02009
Hardik Windlass702b5fa2022-06-29 17:27:01 +053010- `No ONUs are discovered <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-onus.html>`_
11- `Can't authenticate <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-aaa.html>`_
12- `DHCP is not working <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-dhcp.html>`_
13- `No Ping <https://guide.opencord.org/master/profiles/seba/troubleshoot/no-ping.html>`_
Andrea Campanella61fd6662020-07-27 16:56:55 +020014
15
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080016Trace Packets across components
17-------------------------------
18
19Given that ``VOLTHA`` is a collection of microservices we strongly suggest the usage of a log aggregator like
Joey Armstrong1a9b7d12022-12-19 14:13:07 -050020`stern <https://github.com/stern/stern>`_. Stern is able to aggregate logs from multiple containers in
Joey Armstronga62c74a2022-11-23 13:16:34 -050021a simple way, for example to track all the ``packet-ins`` and ``packet-outs``:
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080022
23.. code:: bash
24
Joey Armstrong1a9b7d12022-12-19 14:13:07 -050025 # Capture to a file
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080026 $ stern -n voltha "voltha|adapter" | grep -E "packet-out|packet-in" > packets.trace
27
Joey Armstrong1a9b7d12022-12-19 14:13:07 -050028 # Live streaming packet capture using the tee command:
29 $ stern -n voltha "voltha|adapter" | grep -E "packet-out|packet-in" | tee packets.trace
30
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080031Once you have captured the packets you need, you can see them in wireshark by transforming the logs with:
32
33.. code:: bash
34
35 sed -n 's/.*packet":"\(.*\)",.*$/\1/p' packets.trace | sed -e 's/.\{2\}/& /g' | sed -e 's/^/000000 /' > packets.hex
36
37And then in wireshark select ``File -> Import from Hex Dump``
38
39Or you can decode a single packet using this online tool: `https://hpd.gasmi.net <https://hpd.gasmi.net>`_
40
Joey Armstrong1a9b7d12022-12-19 14:13:07 -050041To get all the EAPOL packets: ``grep -e '888e' packets.trace``
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080042
Joey Armstrong1a9b7d12022-12-19 14:13:07 -050043or if you have the colorization program `hl <https://github.com/mbornet-hl/hl>`__ installed you can highlight with:
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080044
45.. code:: bash
46
47 cat packets.trace | grep 888e | hl -m '.*packet-in.*' -g '.*packet-out.*'
48
49To get all the DHCP packets: ``cat packets.trace | grep 8100``
50
51or if you have `hl <https://github.com/mbornet-hl/hl>`__ installed you can highlight with:
52
53.. code:: bash
54
55 cat packets.trace | grep 8100 | hl -m '.*packet-in.*' -g '.*packet-out.*'
56
57OMCI Packet Dump
58----------------
59
60In order to see OMCI packets in Wireshark you'll need to install ``omci.lua`` and ``BinDecHex.lua`` dissectors
Andrea Campanellaf31c4fd2021-12-28 11:40:03 +010061`wiki.wireshark.org/Contrib <https://wiki.wireshark.org/Contrib#protocol-dissectors>`_.
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080062
63To install them on Ubuntu:
64
65.. code:: bash
66
Joey Armstrong1a9b7d12022-12-19 14:13:07 -050067 mkdir -p "${HOME}/.local/lib/wireshark/plugins"
68 cd "${HOME}/.local/lib/wireshark/plugins"
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080069 wget "https://wiki.wireshark.org/Contrib?action=AttachFile&do=get&target=omci.lua"
70 wget "https://wiki.wireshark.org/Contrib?action=AttachFile&do=get&target=BinDecHex.lua"
71
Joey Armstrong1a9b7d12022-12-19 14:13:07 -050072- IMPORTANT - Apply `this <https://ask.wireshark.org/question/4557/bindechexlua-error-bad-argument-to-module-packageseeall/?answer=4573#post-id-4573>`_ fix to BinDecHex.lua.
73
74- TODO NOTE - The BinDecHex.lua thread is 4 years old, is the patch still
75 required or has it been incorporated into the code base ?
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080076
77To capture the OMCI packets for all ONUs:
78
79.. code:: bash
80
Andrea Campanella9f534c72021-03-19 18:44:27 +010081 cat openonu.logs | grep -E "TxOmciMessage|RxOmciMessage" | grep msg > packets.trace
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080082
83To capture the OMCI packets for a particular ONU:
84
85.. code:: bash
86
Andrea Campanella9f534c72021-03-19 18:44:27 +010087 cat openonu.logs | grep -E "TxOmciMessage|RxOmciMessage" | grep msg | grep [deviceId] > packets.trace
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080088
89Once you have the ``omci.dump`` file you need to prepare it to be imported in wireshark with this command:
90
91.. code:: bash
92
Andrea Campanella9f534c72021-03-19 18:44:27 +010093 awk -F"OmciMessage" '/OmciMessage/{print $2}' packets.trace | cut -f3 -d'"' > rawdump
94 cat rawdump | sed -e 's/.\{2\}/& /g' | sed -e 's/^/000000 /' > omci.hex
Matteo Scandolo1f5530b2019-12-17 10:12:31 -080095
96And then in wireshark:
97
98- select ``File -> Import from Hex Dump``
99- select ``Encapsulation Type -> Ethernet``
100- set ``Ethernet -> Ethertype (hex): 88b5``