Linuxinternals-qemu-hacking

From eLinux.org
Jump to: navigation, search

Introduction

This is a guide to quickly setting a Qemu environment, suitable for kernel testing and hacking on an x86 host. It also contains instruction on setting up PSTORE if you want to. The guide assumes that you are running an Ubuntu host.

Install libvirt / virt-manager

The easiest way to install qemu with a user-friendly front end (virt-manager) is to install virt-manager, which will pull all the dependencies including Qemu that you would need.

sudo apt-get install virt-manager

[Then, follow the instructions here to install the latest Ubuntu ISO https://help.ubuntu.com/community/KVM/VirtManager]

Configuration of your libvirt guest

Make sure to add bridged networking to your guest, give it enough CPUs and atleast 1GB RAM. Make sure openssh-server is installed in the guest, this will allow you to scp kernel images to your guest and test them.

Building kernel for the Qemu guest

  • check out Linus's kernel tree on your host:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
  • Assuming your host is running a fairly recent version of Ubuntu, run the following
cp /boot/config-`uname -r` .config
  • If you're hacking on PSTORE, follow the section "Additional kernel steps for PSTORE" below and come back here.
  • Configure kernel
make olddefconfig
  • Build the kernel (use the bindeb-pkg as below to build convenient Ubuntu debian packages)
make bindeb-pkg
  • After a successful build, one directory below you'll find a linux-image .deb package (ignore the -dbg package), it should look something like
linux-image-4.7.0+_4.7.0+-1_amd64.deb

Now scp this deb pkg to your Qemu guest, ssh into your qemu guest and run:

dpkg -i  linux-image-4.7.0+_4.7.0+-1_amd64.deb    # or whatever the .deb filename is

Additional kernel build steps for PSTORE

  • If you're hacking on pstore, edit .config and make sure the following options are enabled (=y):
CONFIG_PSTORE=y
CONFIG_PSTORE_FTRACE=y
CONFIG_PSTORE_PMSG=y
CONFIG_PSTORE_RAM=y
  • Also add the following to .config, this will make sure kernel reserves 64KB of RAM starting from 256MB:
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="memmap=64K$0x10000000 reboot=w"
  • Download the following patch and apply it, it is required for ramoops + pstore to work:
wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/kernel/pstore/ramoops-force-params.patch
git am ramoops-force-params.patch
  • Download the devmem2 patch, its required for devmem2 tool to read and write to all physical memory
wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/kernel/pstore/0001-hack-force-devmem2-to-r-w-from-all-mem.patch
git am 0001-hack-force-devmem2-to-r-w-from-all-mem.patch

Adding Persistent RAM support to Qemu

The following steps add support for persistent memory in Qemu. This is required to make the PSTORE feature work, as it depends on RAM persisting across a reboot. Inorder to persist RAM, Qemu will back all guest RAM with a file. This is possible after successfully following the steps outlined below:

Building Qemu from source

  • First download the sources of Qemu on your Ubuntu host
sudo apt-get source qemu-system-x86_64
  • If you're hacking on pstore project, apply the following PSTORE patch to the source to make all guest RAM backed by a file:
wget https://raw.githubusercontent.com/joelagnel/joel-snips/master/patches/qemu/0001-Force-QEMU-to-persist-memory-to-mem-path-specified-f.patch
git am 0001-Force-QEMU-to-persist-memory-to-mem-path-specified-f.patch
  • Instead of installing the new package deb, its faster to copy the built qemu binary to /usr/bin
cp /usr/bin/qemu-system-x86_64 /usr/bin/qemu-system-x86_64.bak
cp ./debian/qemu-system-x86/usr/bin/qemu-system-x86_64 /usr/bin

Configuring Qemu to use memory backed file

Once you have created a guest in virt-manager and made sure its working fine, power the guest off.

  • Edit /etc/libvirt/qemu.conf and change user and group to root as below:
#
# Some examples of valid values are:
#
#       user = "qemu"   # A user named "qemu"
#       user = "+0"     # Super user (uid=0)
#       user = "100"    # A user named "100" or a user with uid=100
#
user = "root"
# The group for QEMU processes run by the system instance. It can be
# specified in a similar way to user.
group = "root"

The above step is required so that qemu has permissions to open and memory map the file backing the RAM. If you have a better way, let me know ;)

  • Add the memory backing file to qemu command line, for this you have to edit the libvirt configuration for the guest.

Add the following to your guest XML file. It will look like /etc/libvirt/qemu/<guest-name>.xml. (for me, its /etc/libvirt/qemu/xenial.xml)

 <qemu:commandline>
   <qemu:arg value='-mem-path'/>
   <qemu:arg value='/var/lib/libvirt/qemu/ramf'/>
 </qemu:commandline>

This makes Qemu back all RAM for this guest to/from the file /var/lib/libvirt/qemu/ramf

Now power up the guest, all memory should be backed by the above file, any memory writes should persist across reboots.

Verify that all RAM qemu is writing to is file backed

  • Reserve a certain area of RAM using memmap (see instructions in the "Additional kernel build steps for PSTORE" section but change the following).
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="memmap=64K$0x20000000 reboot=w"

This reserves 64KB at an offset of 256MB from start of the RAM. Rebuild your kernel, copy it to the guest and boot into it.

  • Install devmem2 on the guest
sudo apt-get install devmem2
  • In the guest, write 32-bit word "0xDEAD" to the reserved memory area:
root@joel:~# devmem2 0x20000000 w 0xdead
/dev/mem opened.

Memory mapped at address 0x7f9efae6a000.
Value at address 0x10000000 (0x7f9efae6a000): 0x43474244
Written 0xDEAD; readback 0xDEAD
  • Reboot the guest, then verify that what was written persisted:
root@joel:~# devmem2 0x20000000 w
/dev/mem opened.
Memory mapped at address 0x7fd5db021000.
Value at address 0x10000000 (0x7fd5db021000): 0xDEAD

Running Qemu on the command line

This is just a diffrent way to accomplish the same thing. I found this easier in the virtual machine I was using

Compile the kernel

  • Follow the above steps for compiling the kernel and applying the needed patches
  • Instead of creating debian packages to isntall with we'll run 'make' . You can add -j# to this too.
  • Make should have created the bzImage to use

Compile qemu

The instructions above will get you what you need. If you prefer a diffrent way or dont have Ubuntu you ca build from the source http://wiki.qemu.org/Hosts/Linux This information is pretty much from there.

git clone git://git.qemu-project.org/qemu.git
cd qemu
mkdir build
cd build
../configure
make

Apply the patch from above

cd qemu/build
make
make install

Running the new kernel and qemu

Im assuming the kernel was compiled for x86.

  • Edit the qemu config like in the above steps
  • Aquire a root file system image

http://fs.devloop.org.uk/ I found a bunch on there. I only tried Fedora21 and Ubuntu Trusty. I don't know why the others wouldn't work. Fedora booted the quickest for me. Unzip the image you downloaded.

  • Run qemu
qemu-system-x86_64 -nogrpahic \
	-m 1024 \ 
	-kernel path/to/bzImage \
	-initrd path/to/initrd.img \
	-drive file=/path/to/downloaded/filesystem,if=ide \
	-append "root=/dev/sda console=ttyS0" \
	-mem-path /path/to/ram/file

-kernel is the path to the kernel. This was /arch/x86_64/boot/bzImage . initrd.img is also in /arch/x86_64/boot . -drive file is the path to downloaded and unzipped drive. append is setting root to /dev/sda which is the drive added and console=ttys0 was needed to see the outout mem-path is from the options above. This is file for the memory to be written to.

initrd is optional, i found i need at least 1gb of memory if i use the initrd options. otherwise oom kills some proccess. The system boots faster without and is still functional.

For some reason root gets mounted ro. A small annoyance but shouldn't prevent any work on pstore. debugfs is mounted correctly.

Testing pstore functionality

Since the drive was ro you can't install packages. The above test with devmem2 won't work.

  • ls /sys/fs/pstore

Its probably empty.

  • Trigger a panic
echo c > /proc/sysrq-trigger
  • Exit qemu ( Control + x , a ) Restart the machine with the same command
  • ls /sys/fs/pstore and open any file. It should be the output from the kernel panic