Using smem on Android

smem is a handy tool for analyzing system memory.

Quick steps
For the impatient, here are some quick steps: * hg clone http://selenic.com/repo/smem * cd smem * agcc smemcap.c -o android-smemcap * adb shell mkdir /data/smem * adb push android-smemcap /data/smem * adb shell "/data/smem/android-smemcap >/data/smem/memdata1.tar" * adb pull /data/smem/memdata1.tar. * smem -S memdata1.tar
 * download smem
 * (install python-matplotlib and python-matplotlib-tk, if needed)
 * cross-compile smemcap for Android, using something like 'agcc'
 * install android-smemcap onto your target
 * collect memory information on target
 * retreive the collected data
 * review the data on host

This document describes how to use 'smem' with Android.

= Introduction = smem is a memory tool for analyzing the amount of memory usage on a Linux system. The 'ps' tool provided with Android only shows the VSIZE and RSS memory columns. Even if you install busybox on an android system, that tool's 'ps' will also only show you only the VSIZE value. VSIZE is the total size of the virtual image, and RSS is the resident set size - the amount of physical pages accounted by the kernel to a particular process. These numbers are not very useful to indicate the actual cost in memory of a particular application or daemon.

On the smem web site it says:  smem is a tool that can give numerous reports on memory usage on Linux systems. Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.

Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure. of the activity of a system during boot.

smem consists of a data capture tool, for use on an android target, and an analysis tool for use on a development host machine.

= Overview of Steps to Use smem = To use smem on an Android System, you need to perform the following steps:

* install the smem tool on your host machine * build smemcap on your host machine (or use a pre-built binary) * install smemcap on your Android target system * use smemcap to capture the memory information from /proc on the target * retrieve the collected data from the system * use the smem tool with the collected data, on the host * view memory information and charts * [optionally] provide additional system information

These steps will be described in detail in the following sections. This paper was written based on Android 1.5 software and SDK.

= Detailed Instructions =

Installing the smem tool
smem can be downloaded from http://selenic.com/repo/smem

Download using the mercurial tool (hg) $ hg clone http://selenic.com/repo/smem destination directory: smem requesting all changes adding changesets adding manifests adding file changes added 37 changesets with 40 changes to 5 files 5 files updated, 0 files merged, 0 files removed, 0 files unresolved

This will create a directory called 'smem' in the current directory.

Usually, you will want to place the 'smem' program on your path. This can be done by copying smem to /usr/local/bin, like so (as root): $ cp smem/smem /usr/local/bin

smem has a few requirements:
 * a reasonably modern kernel (> 2.6.27 or so)
 * a reasonably recent version of Python (2.4 or so)

The smem tool can be used in text mode with no further installation.

However, smem also provides the ability to produce nice charts and graphs. For this, it needs the python matplotlib plotting library.

On Fedora, you can install this with the command: $ yum install python-matplotlib python-matplotlib-tk

Other Linux distributions may also have packages for matplotlib.

Here are links to information if you need to directly download and/or install matplotlib
 * http://matplotlib.sourceforge.net/index.html - home page
 * http://sourceforge.net/projects/matplotlib - download area
 * http://matplotlib.sourceforge.net/users/installing.html - install instructions

Building 'smemcap' (or use pre-built binary)
To collect the memory information from the /proc filesystem on the target you need to run the 'smemcap' program. smem also includes a shell script called 'capture' which can collect the data on a standard Linux desktop. However, the Android environment does not have the necessary shell environment or command line tools for this program to work. So you will need to use the compiled C program.

Compile the smemcap program as a native Android program against the bionic C library. Details about this are outside the scope of this document, but you can use the instructions here: Compiling Native Apps for Android

If you have installed 'agcc', then the following steps should work: $ cd smem $ agcc smemcap.c -o android-smemcap

Pre-built binary (for G1)
Alternatively, you can use a [[Media:Android-smemcap.tar|pre-built capture program]]. This one worked on my Android G1 phone and on an OMAP EVM board.

''Note: Due to constraints on file uploads on this wiki, I had to upload it as a tar file. Download it and untar with: 'tar -xvf Android-smemcap.tar' ''

Installing 'android-smemcap' onto your Android target
To install the android-smemcap program on your target, you just need to copy the program, and make sure it is executable. It is recommended to make a separate directory for smem capture data, on the /data portion of the target file system. (However, any writable directory can be used, such as one on an sdcard.)

Make the smem directory, copy the file: $ adb shell mkdir /data/smem $ adb push android-smemcap /data/smem

You should check that the program is executable: $ adb shell chmod 777 /data/smem/android-smemcap $ adb shell ls -l /data/smem -rwxrwxrwx root    root         7391 2009-06-24 14:47 android-smemcap

Collect memory information on the target
Now, to measure the memory usage on the target, use the android-smemcap program to collect the data from the /proc filesystem.

$ adb shell $ cd /data/smem $ ./android-smemcap >memdata1.tar $ exit

You can also do this directly from the host, using the adb shell: $ adb shell "/data/smem/android-smemcap >/data/smem/memdata1.tar"

Note the use of quotes to have the shell redirection on the target stay on the target. Due to the binary nature of the data output by android-smemcap, you can not redirect it directly through the 'adb shell' command to the host filesystem.

Retrieving the collected data from the system
Now, collect the data from the target, and retrieve it to the host: $ adb pull /data/smem/memdata1.tar.

This will pull the just-created file 'test1.tar' from the target and place it in the current directory.

Review the data on the host
Now, you can use the smem tool on the host, to view the target process data:

$ smem -S memdata1.tar PID User    Command                         Swap      USS      PSS      RSS 32 0       /system/bin/debuggerd              0       64       70      252 30 1000    /system/bin/servicemanager         0       80       86      264 36 1002    /system/bin/dbus-daemon --s        0       92      106      368 39 0       /system/bin/sh /runme.sh           0       72      106      316 37 0       /system/bin/installd               0      100      108      332 2462 0       /system/bin/sh -                   0       84      118      332 31 0       /system/bin/vold                   0      136      144      392 41 0       /sbin/adbd                         0      144      144      156 1 0       /init                              0      152      154      172 3780 0       ./android-smemcap                  0      188      195      384 40 1008    /system/bin/akmd                   0      268      303      688 33 0       /system/bin/rild                   0      444      504      984 35 1013    /system/bin/mediaserver            0      876     1458     2948 181 0       com.android.voicedialer            0      948     1550    10668 263 0       com.instamapper.gpstracker         0     1072     1633    10992 252 0       com.rerware.android.MyBacku        0     1080     1665    11348 289 0       com.android.music                  0     1204     1892    11436 271 0       com.biggu.shopsavvy                0     1236     2050    12344 241 0       com.android.alarmclock             0     1488     2087    12264 215 0       com.android.email                  0     1524     2164    12116 302 0       com.google.android.gm              0     1508     2170    11928 280 0       com.acme.android.powermanag        0     1884     2533    12688 135 0       com.android.mms                    0     2036     2711    12708 164 0       android.process.media              0     2208     2840    13076 412 0       com.biggu.shopsavvy:remote         0     2036     2904    13668 228 0       com.android.calendar               0     2296     3005    13596 34 0       zygote /bin/app_process -Xz        0     1088     3218    16688 202 0       com.google.android.apps.map        0     2472     3334    14324 106 0       com.android.phone                  0     4928     5891    16664 155 0       com.google.process.gapps           0     4628     5933    17408 108 0       android.process.acore              0     7288     9023    29560 66 0       system_server                      0    10188    12636    35056

Viewing information and charts
There are a number of interesting options that smem provides, for filtering, analyzing and visualizing the process data.

Here are some interesting options:
 * show pie chart : 'smem -S memdata1.tar --pie=command'
 * This produces a pie chart labeled by the command names (with pss as the data key)
 * Also: 'smem -S memdata1.tar -P com.* --pie=command'
 * Only showing Android applications (apps starting with "com.")
 * Only showing Android applications (apps starting with "com.")
 * Only showing Android applications (apps starting with "com.")


 * show a bar chart : 'smem -S memdata1.tar --bar=command'
 * See below for a sample


 * show all memory mappings on the system : 'smem -S memdata1.tar -m'


 * show the mappings for a specific program : 'smem -S memdata1.tar -m -P email'


 * show the system libraries : 'smem -S memdata1.tar -m -M /system/lib/*'


 * show memory maps for application packages : 'smem -S memdata1.tar -m -M .*.apk'


 * find the libraries with the most users : 'smem -S memdata1.tar -m -M lib -s pids'
 * That is, show mappings for libraries, with 'lib' in the name, sorted by number of processes (pids) using them.
 * sample output:
 * sample output:

Map                                      PIDs   AVGPSS      PSS /system/lib/hw/copybit.msm7k.so             1        8        8 /system/lib/hw/lights.msm7k.so              1        8        8 /system/lib/hw/sensors.trout.so             1       12       12 /system/lib/libagl.so                       1       48       48 /system/lib/libandroid_servers.so           1       16       16 /system/lib/libhtc_acoustic.so              1        4        4 /system/lib/libhtc_ril.so                   1      168      168 /system/lib/libril.so                       1       28       28 /system/lib/libsoundpool.so                 1        4        4 /system/lib/libsurfaceflinger.so            1      108      108 /system/lib/libsystem_server.so             1        8        8 /system/lib/liba2dp.so                      2        6       12 /system/lib/libaudio.so                     2       14       28 /system/lib/libaudioflinger.so              2       62      124 /system/lib/libcamera.so                    2       14       28 /system/lib/libcameraservice.so             2       20       40 /system/lib/libmediaplayerservice.so        2       52      104 /system/lib/libmedia_jni.so                19        1       35 /system/lib/libwebcore.so                  19       10      191 /system/lib/libEGL.so                      20        0       15 /system/lib/libGLESv1_CM.so                20        0       12 /system/lib/libOmxCore.so                  20        0        4 /system/lib/libaes.so                      20        0        4 /system/lib/libandroid_runtime.so          20       13      267 /system/lib/libbluedroid.so                20        0        8 /system/lib/libcorecg.so                   20        2       52 /system/lib/libcrypto.so                   20       21      424 /system/lib/libdrm1.so                     20        0        7 /system/lib/libdvm.so                      20       18      375 /system/lib/libemoji.so                    20        0        4 /system/lib/libhardware.so                 20        0        7 /system/lib/libicudata.so                  20       13      260 /system/lib/libicui18n.so                  20       15      304 /system/lib/libicuuc.so                    20       18      363 /system/lib/libmedia.so                    20        5      102 /system/lib/libnativehelper.so             20        6      135 /system/lib/libopencoreauthor.so           20       13      265 /system/lib/libopencorecommon.so           20        9      185 /system/lib/libopencoreplayer.so           20       29      593 /system/lib/libpixelflinger.so             20        3       71 /system/lib/libsgl.so                      20       29      583 /system/lib/libskiagl.so                   20        0        4 /system/lib/libsonivox.so                  20        0        7 /system/lib/libsqlite.so                   20       16      325 /system/lib/libssl.so                      20        4       97 /system/lib/libui.so                       20       12      245 /system/lib/libvorbisidec.so               20        4       94 /system/lib/libdbus.so                     21        0       16 /system/lib/libexpat.so                    21        3       67 /system/lib/libgps.so                      21        2       52 /system/lib/libhardware_legacy.so          21        1       40 /system/lib/libnetutils.so                 21        0       16 /system/lib/librpc.so                      21        0       16 /system/lib/libwpa_client.so               21        0       12 /system/lib/libutils.so                    22       18      404 /system/lib/libz.so                        22        3       77 /system/lib/libcutils.so                   24        5      131 /system/lib/liblog.so                      25        0       24 /system/lib/libstdc++.so                   29        1       40 /system/lib/libc.so                        30       14      444 /system/lib/libm.so                        30        2       88


 * show processes using libc, sorted by PSS : 'smem -S memdata1.tar -M libc'
 * sample output:

PID User    Command                         Swap      USS      PSS      RSS 135 0       com.android.mms                    0        8       16      264 164 0       android.process.media              0        8       16      264 181 0       com.android.voicedialer            0        8       16      260 202 0       com.google.android.apps.map        0        8       16      264 215 0       com.android.email                  0        8       16      260 228 0       com.android.calendar               0        8       16      264 241 0       com.android.alarmclock             0        8       16      264 252 0       com.rerware.android.MyBacku        0        8       16      260 263 0       com.instamapper.gpstracker         0        8       16      260 271 0       com.biggu.shopsavvy                0        8       16      260 280 0       com.acme.android.powermanag        0        8       16      260 289 0       com.android.music                  0        8       16      260 302 0       com.google.android.gm              0        8       16      264 412 0       com.biggu.shopsavvy:remote         0        8       16      264 30 1000    /system/bin/servicemanager         0       16       19      124 32 0       /system/bin/debuggerd              0       16       19      132 39 0       /system/bin/sh /runme.sh           0       16       19      132 2462 0       /system/bin/sh -                   0       16       19      136 36 1002    /system/bin/dbus-daemon --s        0       16       20      148 3780 0       ./android-smemcap                  0       16       20      144 31 0       /system/bin/vold                   0       20       25      192 37 0       /system/bin/installd               0       20       25      168 40 1008    /system/bin/akmd                   0       20       25      176 106 0       com.android.phone                  0       12       25      276 33 0       /system/bin/rild                   0       20       30      204 108 0       android.process.acore              0       16       36      312 34 0       zygote /bin/app_process -Xz        0        8       83      472 35 1013    /system/bin/mediaserver            0       44       95      344 155 0       com.google.process.gapps           0       28      187      612 66 0       system_server                      0       32      248      772

Sample Charts
Here are some examples of graphical charts produced by smem


 * product a pie chart : 'smem -S memdata1.tar --pie=command'
 * Pie chart is labeled with the command names (and pss is used as the data key)



-


 * Produce a bar chart : 'smem -S memdata1.tar --bar=command'

Providing additional system information
You can provide additional information to smem (such as the kernel image size), using the -R and -K options:

Here is an example showing the 'whole system' memory information, while specifing a kernel vmlinux image: $ smem -S ftrace-kernel.tar -w -t -K /home/tbird/work/android/kernel-msm/build/g1/vmlinux Area                          Used      Cache   Noncache firmware/hardware                0          0          0 kernel image                  4515          0       4515 kernel dynamic memory        17500       4784      12716 userspace memory             78544      28736      49808 free memory                   1988       1988          0 --                            102547      35508      67039

In this example, my kernel was a test kernel that was significantly larger than a standard production kernel.

= Using the graphical viewer = The graphical viewer provides options for zooming and panning portions of the graph, for adjusting graph display parameters, switching between views, and saving a view to an image file.

Here is a snapshot of the smem (matplotlib) chart viewer:



Here are some tips:

Sometimes it is difficult to read the chart labels. You can view the labels more easily if you pan and zoom the chart.

To zoom in on an area, click on the 'zoom to rectangle' button. Drag a rectangle in the viewing area. For a bar chart, the rectangle must reside within the graph area of the chart (above the x axis and to the right of the y axis). Once you unclick your mouse button, a zoomed image is then shown. You can pan the zoomed image by clicking the 'pan axes' button, and dragging the image with the left mouse button. (This "pan axes" button puts the chart in 'pan/zoom' mode. Click the button again to revert to regular view mode.)

You can also zoom using the right mouse button in the 'pan/zoom' mode.

You can adjust the location of the chart on the page (which is also sometimes necessary to see the full labels), by clicking on the "configure subplots" button. Experiment with changing different parameters, such as the bottom or left parameters.

Save a view to a file using the "save the figure" button. Select the image format to save in, from the drop-down list at the bottom of the dialog. Available formats are: bmp, eps, jpg, png, ps, svg.

= Interpreting the results = See the smem web site for an explanation of USS and PSS, and how these measurements should be interpreted to analyze a running system.