Linuxinternals-qemu-hacking

Introduction
This is a guide to quickly setting a Qemu environment, suitable for kernel testing and hacking on an x86 host. It also contains instruction on setting up PSTORE if you want to. The guide assumes that you are running an Ubuntu host.

Install libvirt / virt-manager
The easiest way to install qemu with a user-friendly front end (virt-manager) is to install virt-manager, which will pull all the dependencies including Qemu that you would need. sudo apt-get install virt-manager [Then, follow the instructions here to install the latest Ubuntu ISO https://help.ubuntu.com/community/KVM/VirtManager]

Configuration of your libvirt guest
Make sure to add bridged networking to your guest, give it enough CPUs and atleast 1GB RAM. Make sure openssh-server is installed in the guest, this will allow you to scp kernel images to your guest and test them.

Building kernel for the Qemu guest
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cd linux
 * check out Linus's kernel tree on your host:

cp /boot/config-`uname -r` .config
 * Assuming your host is running a fairly recent version of Ubuntu, run the following


 * If you're hacking on PSTORE, follow the section "Additional kernel steps for PSTORE" below and come back here.

make olddefconfig
 * Configure kernel

make bindeb-pkg
 * Build the kernel (use the bindeb-pkg as below to build convenient Ubuntu debian packages)

linux-image-4.7.0+_4.7.0+-1_amd64.deb
 * After a successful build, one directory below you'll find a linux-image .deb package (ignore the -dbg package), it should look something like

Now scp this deb pkg to your Qemu guest, ssh into your qemu guest and run: dpkg -i linux-image-4.7.0+_4.7.0+-1_amd64.deb    # or whatever the .deb filename is

Additional kernel build steps for PSTORE
CONFIG_PSTORE=y CONFIG_PSTORE_FTRACE=y CONFIG_PSTORE_PMSG=y CONFIG_PSTORE_RAM=y
 * If you're hacking on pstore, edit .config and make sure the following options are enabled (=y):

CONFIG_CMDLINE_BOOL=y CONFIG_CMDLINE="memmap=64K$0x10000000 reboot=w"
 * Also add the following to .config, this will make sure kernel reserves 64KB of RAM starting from 256MB:

wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/kernel/pstore/ramoops-force-params.patch git am ramoops-force-params.patch
 * Download the following patch and apply it, it is required for ramoops + pstore to work:

wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/kernel/pstore/0001-hack-force-devmem2-to-r-w-from-all-mem.patch git am 0001-hack-force-devmem2-to-r-w-from-all-mem.patch
 * Download the devmem2 patch, its required for devmem2 tool to read and write to all physical memory

Adding Persistent RAM support to Qemu
The following steps add support for persistent memory in Qemu. This is required to make the PSTORE feature work, as it depends on RAM persisting across a reboot. Inorder to persist RAM, Qemu will back all guest RAM with a file. This is possible after successfully following the steps outlined below:

Building Qemu from source
sudo apt-get source qemu-system-x86_64
 * First download the sources of Qemu on your Ubuntu host

wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/qemu/0001-Force-QEMU-to-persist-memory-to-mem-path-specified-f.patch git am 0001-Force-QEMU-to-persist-memory-to-mem-path-specified-f.patch
 * If you're hacking on pstore project, apply the following PSTORE patch to the source to make all guest RAM backed by a file:


 * Follow the instructions [here http://www.cyberciti.biz/faq/rebuilding-ubuntu-debian-linux-binary-package/] to rebuild the qemu package. The package name is qemu-system-x86_64 . Note that to save time, pass the option "-nc -j8" to dpkg-buildpackage. This will save time by doing incremental builds and with parallelism.

cp /usr/bin/qemu-system-x86_64 /usr/bin/qemu-system-x86_64.bak cp ./debian/qemu-system-x86/usr/bin/qemu-system-x86_64 /usr/bin
 * Instead of installing the new package deb, its faster to copy the built qemu binary to /usr/bin

Configuring Qemu to use memory backed file
Once you have created a guest in virt-manager and made sure its working fine, power the guest off.

# # # user = "root"
 * Edit /etc/libvirt/qemu.conf and change user and group to root as below:
 * 1) Some examples of valid values are:
 * 1)       user = "qemu"   # A user named "qemu"
 * 2)       user = "+0"     # Super user (uid=0)
 * 3)       user = "100"    # A user named "100" or a user with uid=100

group = "root"
 * 1) The group for QEMU processes run by the system instance. It can be
 * 2) specified in a similar way to user.

The above step is required so that qemu has permissions to open and memory map the file backing the RAM. If you have a better way, let me know ;)

Add the following to your guest XML file. It will look like /etc/libvirt/qemu/.xml. (for me, its /etc/libvirt/qemu/xenial.xml)     This makes Qemu back all RAM for this guest to/from the file /var/lib/libvirt/qemu/ramf
 * Add the memory backing file to qemu command line, for this you have to edit the libvirt configuration for the guest.

Now power up the guest, all memory should be backed by the above file, any memory writes should persist across reboots.

Verify that all RAM qemu is writing to is file backed
CONFIG_CMDLINE_BOOL=y CONFIG_CMDLINE="memmap=64K$0x20000000 reboot=w" This reserves 64KB at an offset of 256MB from start of the RAM. Rebuild your kernel, copy it to the guest and boot into it.
 * Reserve a certain area of RAM using memmap (see instructions in the "Additional kernel build steps for PSTORE" section but change the following).

sudo apt-get install devmem2
 * Install devmem2 on the guest

root@joel:~# devmem2 0x20000000 w 0xdead /dev/mem opened.
 * In the guest, write 32-bit word "0xDEAD" to the reserved memory area:

Memory mapped at address 0x7f9efae6a000. Value at address 0x10000000 (0x7f9efae6a000): 0x43474244 Written 0xDEAD; readback 0xDEAD

root@joel:~# devmem2 0x20000000 w /dev/mem opened. Memory mapped at address 0x7fd5db021000. Value at address 0x10000000 (0x7fd5db021000): 0xDEAD
 * Reboot the guest, then verify that what was written persisted:

Running Qemu on the command line
This is just a diffrent way to accomplish the same thing. I found this easier in the virtual machine I was using

Compile the kernel

 * Follow the above steps for compiling the kernel and applying the needed patches
 * Instead of creating debian packages to isntall with we'll run 'make' . You can add -j# to this too.
 * Make should have created the bzImage to use

Compile qemu
The instructions above will get you what you need. If you prefer a diffrent way or dont have Ubuntu you ca build from the source http://wiki.qemu.org/Hosts/Linux This information is pretty much from there.

git clone git://git.qemu-project.org/qemu.git cd qemu mkdir build cd build ../configure make

Apply the patch from above

cd qemu/build make make install

Running the new kernel and qemu
Im assuming the kernel was compiled for x86.

http://fs.devloop.org.uk/ I found a bunch on there. I only tried Fedora21 and Ubuntu Trusty. I don't know why the others wouldn't work. Fedora booted the quickest for me. Unzip the image you downloaded.
 * Edit the qemu config like in the above steps
 * Aquire a root file system image
 * Run qemu

qemu-system-x86_64 -nogrpahic \ -m 1024 \ -kernel path/to/bzImage \ -initrd path/to/initrd.img \ -drive file=/path/to/downloaded/filesystem,if=ide \ -append "root=/dev/sda console=ttyS0" \ -mem-path /path/to/ram/file

-kernel is the path to the kernel. This was /arch/x86_64/boot/bzImage. initrd.img is also in /arch/x86_64/boot. -drive file is the path to downloaded and unzipped drive. append is setting root to /dev/sda which is the drive added and console=ttys0 was needed to see the outout mem-path is from the options above. This is file for the memory to be written to.

initrd is optional, i found i need at least 1gb of memory if i use the initrd options. otherwise oom kills some proccess. The system boots faster without and is still functional.

For some reason root gets mounted ro. A small annoyance but shouldn't prevent any work on pstore. debugfs is mounted correctly.

Testing pstore functionality
Since the drive was ro you can't install packages. The above test with devmem2 won't work. Its probably empty. echo c > /proc/sysrq-trigger
 * ls /sys/fs/pstore
 * Trigger a panic
 * Exit qemu ( Control + x, a ) Restart the machine with the same command
 * ls /sys/fs/pstore and open any file. It should be the output from the kernel panic