Talk:Kernel dynamic memory analysis

2012-10-05
Here is some feedback from testing on 2012-10-05:

These items are not in any order

Install issue
There's no 'install' mechanism for the tool, and the tool depends visualize_mem_tree.py being on the python path or in the same directory as the trace_analyze.py script. I tried hard-linking trace_analyze to a private 'bin' directory, but got a python error doing this. Using a symlink worked OK. If visualize_mem_tree.py is not used separately, it might be worth considering integrating everything into a single script. (or not - I'm not sure).

Answer
trace_analyze and visualize_mem_tree are now merged. (should we add an 'install' mechanism anyway?)

Extra rings
The chart for my kernel shows some unexpected rings. Specifically, I'm building in /home/tbird/work/mem/build/panda-torvalds My chart has an inner ring of 'build' (showing 100% of memory), a second ring of 'panda-torvalds' (showing 100% of memory), then a third ring where directories start to get broken out.

See this chart:

Is this correct?

Answer
Fixed

Overlapping annotations
The annotations showing the name and size of the ring segments overlap sometimes, making some of them difficult or impossible to read. See above chart.

Availability to show plot interactively
The tool by default produces a png in the current directory. It would be nice to be able to control the output format and location, or allow the user to directly enter matplotlibs interactive viewer. (i.e. pylab.show).

Problem with absolute paths
I tried using the tool with an absolute path, and it had problems: $ trace_analyze.py -k /home/tbird/work/mem/build/panda-torvalds/ No trace log file specified: will report on static size only Reading symbol map at /home/tbird/work/mem/build/panda-torvalds Creating tree from compiled symbols at /home/tbird/work/mem/build/panda-torvalds readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms1.o': No such file readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file ... [lots more 'No such file' errors]

Answer
Fixed

Description of how to get trace
The instructions should includes steps for how to get the trace for the dynamic analysis. It looks like a comprehensive section is started in Documentation/memory_accounting section 3.2.1, so maybe this feedback is premature. Maybe a few short steps on the wiki page for people to get their first trace would be good.

Here's what I did: $ mount -t debugfs none /debug $ cat /debug/tracing/trace >/tmp/kmem.log $ ttc cp target:/tmp/kmem.log.
 * cloned git repository from https://github.com/ezequielgarcia/kmem-probe-framework
 * built kernel
 * generated first chart:
 * $ trace_analyze.py -k ../../build/panda-torvalds -r -a static
 * $ trace_analyze.py -k ../../build/panda-torvalds -r -a static -b fs
 * check that kernel config has CONFIG_FTRACE=y
 * If not, set and rebuild kernel
 * adjust kernel command line to add tracing of kmem events
 * Add: "trace_buf_size=10000000 trace_event=kmem:kmalloc,kmem:kmem_cache_alloc,kmem:kfree,kmem:kmem_cache_free" to kernel command line
 * boot kernel
 * mount debugfs:
 * capture trace data:
 * move to host:
 * generate more charts
 * $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current ; mv linux.png linux-current.png
 * $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current -b fs ; mv fs.png fs-current.png

Note: Make sure you do a 'make clean' if you reconfigure and rebuild the kernel, before running trace_analyze.py again.

sub-sub-dirs doesn't work
I don't know if it's intended to work or not, but I tried to get a ring chart for a sub-sub-directory, and it didn't work

$ trace_analyze.py -a static -k ../../build/panda-torvalds/ -b fs/ubifs No trace log file specified: will report on static size only Reading symbol map at ../../build/panda-torvalds Creating tree from compiled symbols at ../../build/panda-torvalds/fs/ubifs [WARNING] Duplicate data entry! __func__.25429 [WARNING] Duplicate data entry! __func__.25886 ...[some lines omitted] [WARNING] Duplicate data entry! __func__.26112 Cleaning tree .. build panda-torvalds fs       ubifs tnc_misc.o         lprops.o          scan.o          ... [ some lines omitted] tnc_commit.o         io.o          dir.o          debug.o          ioctl.o          sb.o          orphan.o          log.o

[WARNING] Can't find first branch 'fs/ubifs' Creating ringchart file at fs/ubifs.png Traceback (most recent call last): File "/a/home/tbird/bin/trace_analyze.py", line 654, in    main File "/a/home/tbird/bin/trace_analyze.py", line 651, in main visualize_mem_tree(tree, opts.attr, filename) File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 148, in visualize_mem_tree rings = create_child_rings(tree, size_attr=size_attr) File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 98, in create_child_rings max_size = getattr(tree.size, size_attr) AttributeError: 'NoneType' object has no attribute 'size'