testing/memory_usage_pprof.rst - voltha-docs - Gitiles

 Memory Usage investigation
 ==========================

 This page shows how to analyze the memory consumption of a VOLTHA container (written in Go) by using a
 custom built image of the container and the ``pprof`` tool, provided by the GO Language.


 Build an instrumented docker image
 ----------------------------------

 If the Makefile and the Helm Charts of your image support building and deploying profiled images (should be the case),
 you can create a profiled image, via the ``make docker-build`` target adding the option ``BUILD_PROFILED=true``.
 The command will generate two images- the normal and the profiled one
 Following is an example for the ``voltha-openonu-adapter-go``, please customize the values according to your
 docker registry and desired tag:

 .. code:: bash

   DOCKER_REGISTRY="hhildebr/" DOCKER_TAG="hh-dev" BUILD_PROFILED=true  make docker-build

 .. code:: bash

   bbsim@bbsim:~$ docker images
   REPOSITORY                          TAG             IMAGE ID      CREATED      SIZE
   hhildebr/voltha-openonu-adapter-go  hh-dev-profile  e142bba3afea  2 hours ago  725MB
   hhildebr/voltha-openonu-adapter-go  hh-dev          8b6c5436a97d  2 hours ago  37.8MB

 Deploy the instrumented image
 -----------------------------

 Now you can deploy VOLTHA by using the custom ``<image_name>-profile`` via helm flag.
 Also the profiler flag must be activated.
 Following is a snippet to put into a ``values.yaml`` file.

 .. code:: bash

  voltha-adapter-openonu:
   images:
     adapter_open_onu_go:
     repository: hhildebr/voltha-openonu-adapter-go
     tag: hh-dev-profile
     pullPolicy: "Never"
   profiler:
     enabled: true

 Verify and expose the profiler endpoint
 ---------------------------------------

 To verify that the associated profiler server was started successfully and the endpoint is exposed in kubernetes,
 use the following command:

 .. code:: bash

   kubectl get svc --all-namespaces

 The expected output is:

 .. code:: bash

   NAMESPACE   NAME                                   TYPE      CLUSTER-IP EXTERNAL-IP PORT(S)  AGE
   …
   voltha      voltha-voltha-adapter-openonu-profiler ClusterIP None       <none>      6060/TCP 2h
   …

 To then be able to communicate with the profile server from your host system, you have to configure an
 appropriate port forwarding. An example is the following:

 .. code:: bash

   kubectl -n voltha port-forward --address 0.0.0.0 svc/voltha1-voltha-adapter-openonu-profiler 6060:6060

 Use the pprof tool
 ------------------

 Static Mode
 ^^^^^^^^^^^

 The profiling tool can be now interacted with and data retrieved.
 Following is an example of how to get the heap usage of the container and save it to a file:

 .. code:: bash

   curl http://127.0.0.1:6060/debug/pprof/heap > onu-go-heap.pprof

 To get a first idea of the system's status, a graphical overview of the heap usage can be created from the extracted
 data using the ``pprof`` tool and displayed in the browser.

 .. code:: bash

   go tool pprof -http=:8080  onu-go-heap.pprof

 In order to save the graphical representation of the data permanently, an image file can be created from it
 (PDF-format is also supported):

 .. code:: bash

   go tool pprof -png  onu-go-heap.pprof

 Following is an example with heap data collected after hundreds of cycles of startup and deletion
 of multiple BBSIM ONUs. The individual objects/functions can be seen there in relation to each other.
 Already by the size of the boxes you can recognize the objects/functions where to find the memory leaks.

 .. figure:: ../_static/1_graph_with_mem_leak.png
    :alt: Graphical representation of memory usage
    :width: 70%
    :align: center

    Graphical representation of memory usage

 Interactive Mode
 ^^^^^^^^^^^^^^^^

 To start a detailed investigation, ``pprof`` is invoked in interactive mode.
 Since pprof assumes by default that the code belonging to the UUT can be found in the /go/src directory,
 this path part must be removed using the trim_path option and replaced with the actual directory part using
 the source_path option.
 Following is an example for the openonu adapter:

 .. code:: bash

   go tool pprof  -source_path /home/bbsim/temp_pperf/voltha-openonu-adapter-go -trim_path /go/src /mnt/shared/onu-go-heap.pprof

 by using the ``top`` command one can see which are the main originators of memory leaks,
 observing the the first ten objects/functions displayed:

 .. figure:: ../_static/2_top_with_mem_leak.png
    :alt: Ten highest memory functions
    :width: 80%
    :align: center

    Ten highest memory functions

 If more object/functions are to be displayed, simply add the desired number to the command - e.g. ``top100``.


 The next step is to use the ``list <function_name>`` command to view the detailed memory allocations.

 Example Memory leak analysis and fix
 ------------------------------------
 Following is an example applied for the number one of the top10, function ``NewOnuDeviceEntry``.

 .. figure:: ../_static/3_list_with_mem_leak.png
    :alt: List function details before optimization
    :width: 80%
    :align: center

    List function details before optimization

 Since the single allocation of ``onuDeviceEntry.omciRebootMessageReceivedChannel`` should occupy at most 100kB of
 heap memory (2048 standard OMCI messages), we can conclude that the 85.90MB of occupied memory represents more
 than 850 instances of ``onuDeviceEntry`` not deleted by the GC.

 After removing the cause of the memory leak the same test with hundreds of cycles of startup and deletion of multiple
 BBSIM ONUs shows that the issue has been solved. Having a look at the retrieved pprof data shows a significant
 decrease of memory consumption, about 10MB vs 744MB without the patch:

 .. figure:: ../_static/4_top_without_mem_leak.png
    :alt: List function details after optimization
    :width: 70%
    :align: center

    List function details after optimization
	Memory Usage investigation
	==========================

	This page shows how to analyze the memory consumption of a VOLTHA container (written in Go) by using a
	custom built image of the container and the ``pprof`` tool, provided by the GO Language.


	Build an instrumented docker image
	----------------------------------

	If the Makefile and the Helm Charts of your image support building and deploying profiled images (should be the case),
	you can create a profiled image, via the ``make docker-build`` target adding the option ``BUILD_PROFILED=true``.
	The command will generate two images- the normal and the profiled one
	Following is an example for the ``voltha-openonu-adapter-go``, please customize the values according to your
	docker registry and desired tag:

	.. code:: bash

	DOCKER_REGISTRY="hhildebr/" DOCKER_TAG="hh-dev" BUILD_PROFILED=true make docker-build

	.. code:: bash

	bbsim@bbsim:~$ docker images
	REPOSITORY TAG IMAGE ID CREATED SIZE
	hhildebr/voltha-openonu-adapter-go hh-dev-profile e142bba3afea 2 hours ago 725MB
	hhildebr/voltha-openonu-adapter-go hh-dev 8b6c5436a97d 2 hours ago 37.8MB

	Deploy the instrumented image
	-----------------------------

	Now you can deploy VOLTHA by using the custom ``<image_name>-profile`` via helm flag.
	Also the profiler flag must be activated.
	Following is a snippet to put into a ``values.yaml`` file.

	.. code:: bash

	voltha-adapter-openonu:
	images:
	adapter_open_onu_go:
	repository: hhildebr/voltha-openonu-adapter-go
	tag: hh-dev-profile
	pullPolicy: "Never"
	profiler:
	enabled: true

	Verify and expose the profiler endpoint
	---------------------------------------

	To verify that the associated profiler server was started successfully and the endpoint is exposed in kubernetes,
	use the following command:

	.. code:: bash

	kubectl get svc --all-namespaces

	The expected output is:

	.. code:: bash

	NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
	…
	voltha voltha-voltha-adapter-openonu-profiler ClusterIP None <none> 6060/TCP 2h
	…

	To then be able to communicate with the profile server from your host system, you have to configure an
	appropriate port forwarding. An example is the following:

	.. code:: bash

	kubectl -n voltha port-forward --address 0.0.0.0 svc/voltha1-voltha-adapter-openonu-profiler 6060:6060

	Use the pprof tool
	------------------

	Static Mode
	^^^^^^^^^^^

	The profiling tool can be now interacted with and data retrieved.
	Following is an example of how to get the heap usage of the container and save it to a file:

	.. code:: bash

	curl http://127.0.0.1:6060/debug/pprof/heap > onu-go-heap.pprof

	To get a first idea of the system's status, a graphical overview of the heap usage can be created from the extracted
	data using the ``pprof`` tool and displayed in the browser.

	.. code:: bash

	go tool pprof -http=:8080 onu-go-heap.pprof

	In order to save the graphical representation of the data permanently, an image file can be created from it
	(PDF-format is also supported):

	.. code:: bash

	go tool pprof -png onu-go-heap.pprof

	Following is an example with heap data collected after hundreds of cycles of startup and deletion
	of multiple BBSIM ONUs. The individual objects/functions can be seen there in relation to each other.
	Already by the size of the boxes you can recognize the objects/functions where to find the memory leaks.

	.. figure:: ../_static/1_graph_with_mem_leak.png
	:alt: Graphical representation of memory usage
	:width: 70%
	:align: center

	Graphical representation of memory usage

	Interactive Mode
	^^^^^^^^^^^^^^^^

	To start a detailed investigation, ``pprof`` is invoked in interactive mode.
	Since pprof assumes by default that the code belonging to the UUT can be found in the /go/src directory,
	this path part must be removed using the trim_path option and replaced with the actual directory part using
	the source_path option.
	Following is an example for the openonu adapter:

	.. code:: bash

	go tool pprof -source_path /home/bbsim/temp_pperf/voltha-openonu-adapter-go -trim_path /go/src /mnt/shared/onu-go-heap.pprof

	by using the ``top`` command one can see which are the main originators of memory leaks,
	observing the the first ten objects/functions displayed:

	.. figure:: ../_static/2_top_with_mem_leak.png
	:alt: Ten highest memory functions
	:width: 80%
	:align: center

	Ten highest memory functions

	If more object/functions are to be displayed, simply add the desired number to the command - e.g. ``top100``.


	The next step is to use the ``list <function_name>`` command to view the detailed memory allocations.

	Example Memory leak analysis and fix
	------------------------------------
	Following is an example applied for the number one of the top10, function ``NewOnuDeviceEntry``.

	.. figure:: ../_static/3_list_with_mem_leak.png
	:alt: List function details before optimization
	:width: 80%
	:align: center

	List function details before optimization

	Since the single allocation of ``onuDeviceEntry.omciRebootMessageReceivedChannel`` should occupy at most 100kB of
	heap memory (2048 standard OMCI messages), we can conclude that the 85.90MB of occupied memory represents more
	than 850 instances of ``onuDeviceEntry`` not deleted by the GC.

	After removing the cause of the memory leak the same test with hundreds of cycles of startup and deletion of multiple
	BBSIM ONUs shows that the issue has been solved. Having a look at the retrieved pprof data shows a significant
	decrease of memory consumption, about 10MB vs 744MB without the patch:

	.. figure:: ../_static/4_top_without_mem_leak.png
	:alt: List function details after optimization
	:width: 70%
	:align: center

	List function details after optimization