Documenting how to use pprof tool with examples
Change-Id: I24786439ead3b33bf65c6f5c077517de9ba853e1
diff --git a/_static/1_graph_with_mem_leak.png b/_static/1_graph_with_mem_leak.png
new file mode 100644
index 0000000..f68709c
--- /dev/null
+++ b/_static/1_graph_with_mem_leak.png
Binary files differ
diff --git a/_static/2_top_with_mem_leak.png b/_static/2_top_with_mem_leak.png
new file mode 100644
index 0000000..480766e
--- /dev/null
+++ b/_static/2_top_with_mem_leak.png
Binary files differ
diff --git a/_static/3_list_with_mem_leak.png b/_static/3_list_with_mem_leak.png
new file mode 100644
index 0000000..78413dd
--- /dev/null
+++ b/_static/3_list_with_mem_leak.png
Binary files differ
diff --git a/_static/4_top_without_mem_leak.png b/_static/4_top_without_mem_leak.png
new file mode 100644
index 0000000..2b5552d
--- /dev/null
+++ b/_static/4_top_without_mem_leak.png
Binary files differ
diff --git a/index.rst b/index.rst
index 3432470..821b99d 100644
--- a/index.rst
+++ b/index.rst
@@ -167,6 +167,7 @@
VOLTHA system tests <voltha-system-tests/README.md>
testing/index.rst
+ testing/memory_usage_pprof.rst
.. toctree::
:maxdepth: 1
diff --git a/testing/memory_usage_pprof.rst b/testing/memory_usage_pprof.rst
new file mode 100644
index 0000000..8f6a79a
--- /dev/null
+++ b/testing/memory_usage_pprof.rst
@@ -0,0 +1,162 @@
+Memory Usage investigation
+==========================
+
+This page shows how to analyze the memory consumption of a VOLTHA container (written in Go) by using a
+custom built image of the container and the ``pprof`` tool, provided by the GO Language.
+
+
+Build an instrumented docker image
+----------------------------------
+
+If the Makefile and the Helm Charts of your image support building and deploying profiled images (should be the case),
+you can create a profiled image, via the ``make docker-build`` target adding the option ``BUILD_PROFILED=true``.
+The command will generate two images- the normal and the profiled one
+Following is an example for the ``voltha-openonu-adapter-go``, please customize the values according to your
+docker registry and desired tag:
+
+.. code:: bash
+
+ DOCKER_REGISTRY="hhildebr/" DOCKER_TAG="hh-dev" BUILD_PROFILED=true make docker-build
+
+.. code:: bash
+
+ bbsim@bbsim:~$ docker images
+ REPOSITORY TAG IMAGE ID CREATED SIZE
+ hhildebr/voltha-openonu-adapter-go hh-dev-profile e142bba3afea 2 hours ago 725MB
+ hhildebr/voltha-openonu-adapter-go hh-dev 8b6c5436a97d 2 hours ago 37.8MB
+
+Deploy the instrumented image
+-----------------------------
+
+Now you can deploy VOLTHA by using the custom ``<image_name>-profile`` via helm flag.
+Also the profiler flag must be activated.
+Following is a snippet to put into a ``values.yaml`` file.
+
+.. code:: bash
+
+ voltha-adapter-openonu:
+ images:
+ adapter_open_onu_go:
+ repository: hhildebr/voltha-openonu-adapter-go
+ tag: hh-dev-profile
+ pullPolicy: "Never"
+ profiler:
+ enabled: true
+
+Verify and expose the profiler endpoint
+---------------------------------------
+
+To verify that the associated profiler server was started successfully and the endpoint is exposed in kubernetes,
+use the following command:
+
+.. code:: bash
+
+ kubectl get svc --all-namespaces
+
+The expected output is:
+
+.. code:: bash
+
+ NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+ …
+ voltha voltha-voltha-adapter-openonu-profiler ClusterIP None <none> 6060/TCP 2h
+ …
+
+To then be able to communicate with the profile server from your host system, you have to configure an
+appropriate port forwarding. An example is the following:
+
+.. code:: bash
+
+ kubectl -n voltha port-forward --address 0.0.0.0 svc/voltha1-voltha-adapter-openonu-profiler 6060:6060
+
+Use the pprof tool
+------------------
+
+Static Mode
+^^^^^^^^^^^
+
+The profiling tool can be now interacted with and data retrieved.
+Following is an example of how to get the heap usage of the container and save it to a file:
+
+.. code:: bash
+
+ curl http://127.0.0.1:6060/debug/pprof/heap > onu-go-heap.pprof
+
+To get a first idea of the system's status, a graphical overview of the heap usage can be created from the extracted
+data using the ``pprof`` tool and displayed in the browser.
+
+.. code:: bash
+
+ go tool pprof -http=:8080 onu-go-heap.pprof
+
+In order to save the graphical representation of the data permanently, an image file can be created from it
+(PDF-format is also supported):
+
+.. code:: bash
+
+ go tool pprof -png onu-go-heap.pprof
+
+Following is an example with heap data collected after hundreds of cycles of startup and deletion
+of multiple BBSIM ONUs. The individual objects/functions can be seen there in relation to each other.
+Already by the size of the boxes you can recognize the objects/functions where to find the memory leaks.
+
+.. figure:: ../_static/1_graph_with_mem_leak.png
+ :alt: Graphical representation of memory usage
+ :width: 70%
+ :align: center
+
+ Graphical representation of memory usage
+
+Interactive Mode
+^^^^^^^^^^^^^^^^
+
+To start a detailed investigation, ``pprof`` is invoked in interactive mode.
+Since pprof assumes by default that the code belonging to the UUT can be found in the /go/src directory,
+this path part must be removed using the trim_path option and replaced with the actual directory part using
+the source_path option.
+Following is an example for the openonu adapter:
+
+.. code:: bash
+
+ go tool pprof -source_path /home/bbsim/temp_pperf/voltha-openonu-adapter-go -trim_path /go/src /mnt/shared/onu-go-heap.pprof
+
+by using the ``top`` command one can see which are the main originators of memory leaks,
+observing the the first ten objects/functions displayed:
+
+.. figure:: ../_static/2_top_with_mem_leak.png
+ :alt: Ten highest memory functions
+ :width: 80%
+ :align: center
+
+ Ten highest memory functions
+
+If more object/functions are to be displayed, simply add the desired number to the command - e.g. ``top100``.
+
+
+The next step is to use the ``list <function_name>`` command to view the detailed memory allocations.
+
+Example Memory leak analysis and fix
+------------------------------------
+Following is an example applied for the number one of the top10, function ``NewOnuDeviceEntry``.
+
+.. figure:: ../_static/3_list_with_mem_leak.png
+ :alt: List function details before optimization
+ :width: 80%
+ :align: center
+
+ List function details before optimization
+
+Since the single allocation of ``onuDeviceEntry.omciRebootMessageReceivedChannel`` should occupy at most 100kB of
+heap memory (2048 standard OMCI messages), we can conclude that the 85.90MB of occupied memory represents more
+than 850 instances of ``onuDeviceEntry`` not deleted by the GC.
+
+After removing the cause of the memory leak the same test with hundreds of cycles of startup and deletion of multiple
+BBSIM ONUs shows that the issue has been solved. Having a look at the retrieved pprof data shows a significant
+decrease of memory consumption, about 10MB vs 744MB without the patch:
+
+.. figure:: ../_static/4_top_without_mem_leak.png
+ :alt: List function details after optimization
+ :width: 70%
+ :align: center
+
+ List function details after optimization
