R-Car/IO-Virtualization

= Generic introduction =

This document is about describing the different methods how to attach IO devices to a virtual machine provided by QEMU+KVM.

For the basic setup, Geert Uytterhoeven did an introduction how to setup QEMU+KVM on R-Car Gen3.

Please note also that this document is only about QEMU+KVM. Other hypervisors would need a seperate document.

= Virtualization of IO devices =

In general, there are three types of passing IO devices.

Emulation
Also called 'full virtualization'. This means a IO device on the guest is emulated completely by software. It aims to fully mimic the original hardware behaviour. The guest OS can use the same hardware driver as it would for the real hardware. To feed the virtual hardware with real data from the outside world, standard Linux interfaces are used, usually by connecting to files or device nodes on the host. Because emulation is pure software, this offers highest encapsulation at the price of performance.

Paravirtualization
Paravirtualization means a device will be emulated but it will not match a real hardware. It is a special &quot;theoretical&quot; device designed for better latency and throughput. The downside is that this device needs its own driver in the guest OS and, thus, is aware that it is running virtualized.

virtio is a standard to ease the implementation of paravirtualized devices. It also defines an API which different hypervisors can implement. So, the Linux virtio-net driver of the guest will work with KVM and Xen out of the box. Note that virtio devices in the guest show up as regular PCI devices which the driver can bind to.

For a more detailed introduction to virtio, read this article. The specification is here.

An example how to setup a serial connection with virtio can be found here. This can also be used as a simple data transfer mechanism between host and guest.

Passthrough
Passthrough means that a device from the host will be presented to the guest transparently. This allows for near native performance, but breaks the encapsulation which is bad for live mitigation of the guest.

Also, for devices which need to map memory or use DMA, address translation will be needed. This requires an IOMMU. Interrupts need to be remapped. Linux provides the VFIO subsystem to ease the above tasks. More information can be found in the Linux kernel tree in the file.

Usually, this way you can pass a device to only one guest. To overcome this limitation, your device needs to be SR-IOV (Single Root I/O Virtualization) capable. Using a PCI example, this means, the device can create multiple virtualized PCI devices which can then be mapped each to a guest.

A more detailed introduction to passthrough and SR-IOV can be found here.

Here is a description how to SR-IOV it with network cards.

= Usage of the different types of virtualized IO devices =

In this section, the qemu command line will be shortened to an easily understandable minimum. The base part is left out and always the same. The base part depends a little on your rootfs creation. However, it will look similar to the one used by the author:

qemu-system-x86_64 -enable-kvm -kernel bzImage -hda qemu-rootfs-x86_64.img \ -append 'root=/dev/sda1 console=ttyS0' -nographic ... Setting up an additional device is done with the '-device' switch

qemu... -device pci-serial This will create the device within the guest. But it will be of not much use, because the data is nowhere to be put. We need to create a backend for this frontend. Keep in mind that you will nearly always need some kind of backend-frontend mapping when you add an IO device to the guest:

qemu... -chardev tty,path=/dev/ttyS0,id=hostserial \ -device pci-serial,chardev=hostserial This connects the emulated pci-serial device with ttyS0 on the host.

Now, let's create a loopback with a passedthrough USB-to-serial converter:

qemu... -chardev tty,path=/dev/ttyS0,id=hostserial \ -device pci-serial,chardev=hostserial      \ -usb -device usb-host,hostbus=&lt;x&gt;,hostaddr=&lt;y&gt; Of course, you need to find out the bus/addr pair beforehand with e.g. . Now, this USB device should be recognized by the guest kernel and create a ´ttyUSB0´.

If you plug a null-modem cable between the USB-to-serial adapter and the native serial port, you have a loopback consisting of an emulated device and a passthrough device.

For a more comprehensive understanding of QEMU command line parameters, this talk is recommended.

= Extending the different types of virtualized IO devices =

Emulation
In the case you need to add another emulated device, try to implement a generic PCI or USB based one with an already existing Linux kernel driver. Because we want different &quot;generic&quot; guests for encapsulation, generic devices are good. The direct emulation of R-Car hardware is usually not needed.

If you need to extend an emulated device or add a new one to QEMU, you need to look here:

Paravirtualization
Be aware that QEMU maintainers only accept new virtio drivers if the benefit in terms of throughput or latency can be justified. The price of having specialized virtio drivers in the guest kernel is considered high, so this is usually only done as an exception.

If you need to extend a paravirtualized device or add a new one to QEMU, you need to look here:

Passthrough
If you need to extend a passthrough device or add a new one to QEMU, you need to look here:

Additionally, if VFIO is desired because you need memory translation or have interrupts remapped, look here: