Linuxinternals-qemu-hacking
Contents
Introduction
This is a guide to quickly setting a Qemu environment, suitable for kernel testing and hacking on an x86 host. It also contains instruction on setting up PSTORE if you want to. The guide assumes that you are running an Ubuntu host.
Install libvirt / virt-manager
The easiest way to install qemu with a user-friendly front end (virt-manager) is to install virt-manager, which will pull all the dependencies including Qemu that you would need.
sudo apt-get install virt-manager
[Then, follow the instructions here to install the latest Ubuntu ISO https://help.ubuntu.com/community/KVM/VirtManager]
Configuration of your libvirt guest
Make sure to add bridged networking to your guest, give it enough CPUs and atleast 1GB RAM. Make sure openssh-server is installed in the guest, this will allow you to scp kernel images to your guest and test them.
Building kernel for the Qemu guest
- check out Linus's kernel tree on your host:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cd linux
- Assuming your host is running a fairly recent version of Ubuntu, run the following
cp /boot/config-`uname -r` .config
- If you're hacking on PSTORE, follow the section "Additional kernel steps for PSTORE" below and come back here.
- Configure kernel
make olddefconfig
- Build the kernel (use the bindeb-pkg as below to build convenient Ubuntu debian packages)
make bindeb-pkg
- After a successful build, one directory below you'll find a linux-image .deb package (ignore the -dbg package), it should look something like
linux-image-4.7.0+_4.7.0+-1_amd64.deb
Now scp this deb pkg to your Qemu guest, ssh into your qemu guest and run:
dpkg -i linux-image-4.7.0+_4.7.0+-1_amd64.deb # or whatever the .deb filename is
Additional kernel build steps for PSTORE
- If you're hacking on pstore, edit .config and make sure the following options are enabled (=y):
CONFIG_PSTORE=y CONFIG_PSTORE_FTRACE=y CONFIG_PSTORE_PMSG=y CONFIG_PSTORE_RAM=y
- Also add the following to .config, this will make sure kernel reserves 64KB of RAM starting from 256MB:
CONFIG_CMDLINE_BOOL=y CONFIG_CMDLINE="memmap=64K$0x10000000 reboot=w"
- Download the following patch and apply it, it is required for ramoops + pstore to work:
wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/kernel/pstore/ramoops-force-params.patch git am ramoops-force-params.patch
- Download the devmem2 patch, its required for devmem2 tool to read and write to all physical memory
wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/kernel/pstore/0001-hack-force-devmem2-to-r-w-from-all-mem.patch git am 0001-hack-force-devmem2-to-r-w-from-all-mem.patch
Adding Persistent RAM support to Qemu
The following steps add support for persistent memory in Qemu. This is required to make the PSTORE feature work, as it depends on RAM persisting across a reboot. Inorder to persist RAM, Qemu will back all guest RAM with a file. This is possible after successfully following the steps outlined below:
Building Qemu from source
- First download the sources of Qemu on your Ubuntu host
sudo apt-get source qemu-system-x86_64
- If you're hacking on pstore project, apply the following PSTORE patch to the source to make all guest RAM backed by a file:
wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/qemu/0001-Force-QEMU-to-persist-memory-to-mem-path-specified-f.patch git am 0001-Force-QEMU-to-persist-memory-to-mem-path-specified-f.patch
- Follow the instructions [here http://www.cyberciti.biz/faq/rebuilding-ubuntu-debian-linux-binary-package/] to rebuild the qemu package. The package name is qemu-system-x86_64. Note that to save time, pass the option "-nc -j8" to dpkg-buildpackage. This will save time by doing incremental builds and with parallelism.
- Instead of installing the new package deb, its faster to copy the built qemu binary to /usr/bin
cp /usr/bin/qemu-system-x86_64 /usr/bin/qemu-system-x86_64.bak cp ./debian/qemu-system-x86/usr/bin/qemu-system-x86_64 /usr/bin
Configuring Qemu to use memory backed file
Once you have created a guest in virt-manager and made sure its working fine, power the guest off.
- Edit /etc/libvirt/qemu.conf and change user and group to root as below:
# # Some examples of valid values are: # # user = "qemu" # A user named "qemu" # user = "+0" # Super user (uid=0) # user = "100" # A user named "100" or a user with uid=100 # user = "root"
# The group for QEMU processes run by the system instance. It can be # specified in a similar way to user. group = "root"
The above step is required so that qemu has permissions to open and memory map the file backing the RAM. If you have a better way, let me know ;)
- Add the memory backing file to qemu command line, for this you have to edit the libvirt configuration for the guest.
Add the following to your guest XML file. It will look like /etc/libvirt/qemu/<guest-name>.xml. (for me, its /etc/libvirt/qemu/xenial.xml)
<qemu:commandline> <qemu:arg value='-mem-path'/> <qemu:arg value='/var/lib/libvirt/qemu/ramf'/> </qemu:commandline>
This makes Qemu back all RAM for this guest to/from the file /var/lib/libvirt/qemu/ramf
Now power up the guest, all memory should be backed by the above file, any memory writes should persist across reboots.
Verify that all RAM qemu is writing to is file backed
- Reserve a certain area of RAM using memmap (see instructions in the "Additional kernel build steps for PSTORE" section but change the following).
CONFIG_CMDLINE_BOOL=y CONFIG_CMDLINE="memmap=64K$0x20000000 reboot=w"
This reserves 64KB at an offset of 256MB from start of the RAM. Rebuild your kernel, copy it to the guest and boot into it.
- Install devmem2 on the guest
sudo apt-get install devmem2
- In the guest, write 32-bit word "0xDEAD" to the reserved memory area:
root@joel:~# devmem2 0x20000000 w 0xdead /dev/mem opened. Memory mapped at address 0x7f9efae6a000. Value at address 0x10000000 (0x7f9efae6a000): 0x43474244 Written 0xDEAD; readback 0xDEAD
- Reboot the guest, then verify that what was written persisted:
root@joel:~# devmem2 0x20000000 w /dev/mem opened. Memory mapped at address 0x7fd5db021000. Value at address 0x10000000 (0x7fd5db021000): 0xDEAD
Running Qemu on the command line
This is just a diffrent way to accomplish the same thing. I found this easier in the virtual machine I was using
Compile the kernel
- Follow the above steps for compiling the kernel and applying the needed patches
- Instead of creating debian packages to isntall with we'll run 'make' . You can add -j# to this too.
- Make should have created the bzImage to use
Compile qemu
The instructions above will get you what you need. If you prefer a diffrent way or dont have Ubuntu you ca build from the source http://wiki.qemu.org/Hosts/Linux This information is pretty much from there.
git clone git://git.qemu-project.org/qemu.git cd qemu mkdir build cd build ../configure make
Apply the patch from above
cd qemu/build make make install
Running the new kernel and qemu
Im assuming the kernel was compiled for x86.
- Edit the qemu config like in the above steps
- Aquire a root file system image
http://fs.devloop.org.uk/ I found a bunch on there. I only tried Fedora21 and Ubuntu Trusty. I don't know why the others wouldn't work. Fedora booted the quickest for me. Unzip the image you downloaded.
- Run qemu
qemu-system-x86_64 -nogrpahic \ -m 1024 \ -kernel path/to/bzImage \ -initrd path/to/initrd.img \ -drive file=/path/to/downloaded/filesystem,if=ide \ -append "root=/dev/sda console=ttyS0" \ -mem-path /path/to/ram/file
-kernel is the path to the kernel. This was /arch/x86_64/boot/bzImage . initrd.img is also in /arch/x86_64/boot . -drive file is the path to downloaded and unzipped drive. append is setting root to /dev/sda which is the drive added and console=ttys0 was needed to see the outout mem-path is from the options above. This is file for the memory to be written to.
initrd is optional, i found i need at least 1gb of memory if i use the initrd options. otherwise oom kills some proccess. The system boots faster without and is still functional.
For some reason root gets mounted ro. A small annoyance but shouldn't prevent any work on pstore. debugfs is mounted correctly.
Testing pstore functionality
Since the drive was ro you can't install packages. The above test with devmem2 won't work.
- ls /sys/fs/pstore
Its probably empty.
- Trigger a panic
echo c > /proc/sysrq-trigger
- Exit qemu ( Control + x , a ) Restart the machine with the same command
- ls /sys/fs/pstore and open any file. It should be the output from the kernel panic