R-Car/Virtualization/VFIO

Device Pass-Through Using VFIO

= Platform Device Pass-Through Prototype =

This is a proof-of-concept showing how to provide guest access to an R-Car GPIO controller block on the Renesas Salvator-X and Salvator-XS boards.

Host Side

 * Build and boot a host kernel using:
 * Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git
 * Branch: "topic/rcar3-virt-gpio-passthrough-v2"
 * Config: "renesas_defconfig"

$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 * Configure workarounds for missing functionality:

$ echo e6055400.gpio > /sys/bus/platform/drivers/gpio_rcar/unbind gpio gpiochip6: REMOVING GPIOCHIP WITH GPIOS STILL REQUESTED
 * Unbind GPIO6 from the "gpio-rcar" driver:

$ echo vfio-platform > /sys/bus/platform/devices/e6055400.gpio/driver_override $ echo e6055400.gpio > /sys/bus/platform/drivers/vfio-platform/bind iommu: Adding device e6055400.gpio to group 0 vfio-platform e6055400.gpio: Adding kernel taint for vfio-noiommu group on device
 * Bind GPIO6 to the "vfio-platform" driver, for pass-through to guests:

Guest Side

 * Build a guest kernel using:
 * Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git
 * Branch: "topic/rcar3-virt-gpio-passthrough-v2"
 * Config: "virt_defconfig"


 * Build QEMU using:
 * Repository: https://github.com/geertu/qemu.git
 * Branch: "topic/rcar3-virt-gpio-passthrough-v2"

I.e.: -device vfio-platform,host=$1a,manufacturer=$2,model=$3 -device vfio-platform,host=e6055400.gpio,manufacturer=renesas,model=rcar-gen3-gpio -device vfio-platform,sysfsdev=$1b,manufacturer=$2,model=$3 -device vfio-platform,sysfsdev=/sys/bus/platform/devices/e6055400.gpio,manufacturer=renesas,model=rcar-gen3-gpio
 * When starting QEMU, specify pass-through of the GPIO6 platform device. The device specifier consists of three parts:
 * $1a
 * the device's node name, or
 * $1b
 * the full path to the device's node in sysfs,
 * $2
 * the manufacturer part of the device's compatible value,
 * $3
 * the model part of the device's compatible value.

Full command line: $ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=e6055400.gpio,manufacturer=renesas,model=rcar-gen3-gpio

Guest kernel output: Booting Linux on physical CPU 0x0000000000 [0x411fd073] Linux version 4.16.0-arm64-virt-00010-g8f416425ed2e81bb ... ... gpio_rcar c000000.e6055400.gpio: driving 32 GPIOs ...

Host kernel output: vfio-platform e6055400.gpio: reset vfio-platform e6055400.gpio: vfio-noiommu device opened by user (qemu-system-aar:3003)

--- /sys/firmware/devicetree/base.orig +++ /sys/firmware/devicetree/base @@ -124,6 +124,14 @@               compatible = "qemu,platform", "simple-bus"; interrupt-parent = ; ranges = <0x0 0x0 0xc000000 0x2000000>; + +              e6055400.gpio@0 { +                      #gpio-cells = ; +                      compatible = "renesas,rcar-gen3-gpio"; +                      gpio-controller; +                      interrupts = ; +                      reg = ; +              };        };
 * Analysis of "/sys/firmware/devicetree/base" using "dtx_diff" shows that "e6055400.gpio@0" has been added as a subnode to the existing "platform@c000000" node:

pmu {

echo $(($base + $i)) > /sys/class/gpio/export echo low > /sys/class/gpio/gpio$(($base + $i))/direction done
 * Export GPIOs used for LEDs (LED4/5/6), and turn all LEDs off:
 * 1) base=$(cat /sys/class/gpio/gpiochip*/base)
 * 2) for i in 11 12 13; do

echo high > /sys/class/gpio/gpio$(($base + $i))/direction sleep 1 echo low > /sys/class/gpio/gpio$(($base + $i))/direction done
 * Cycle through all LEDs, letting them blink once:
 * 1) for i in 11 12 13; do

5:         0     GIC-0 144 Level     c000000.e6055400.gpio 42:         0  c000000.e6055400.gpio  11 Edge      gpiolib
 * Change the first GPIO from output to input, and configure edge detection to enable its interrupt:
 * 1) echo in > /sys/class/gpio/gpio$(($base + 11))/direction
 * 2) echo both > /sys/class/gpio/gpio$(($base + 11))/edge
 * 3) grep gpio /proc/interrupts

71298 42:          0  c000000.e6055400.gpio  11 Edge      gpiolib 388 42:          1  c000000.e6055400.gpio  11 Edge      gpiolib 247 42:          2  c000000.e6055400.gpio  11 Edge      gpiolib 139 42:          3  c000000.e6055400.gpio  11 Edge      gpiolib 586 42:          4  c000000.e6055400.gpio  11 Edge      gpiolib ^C
 * Monitor the GPIO interrupt (press button SW20 to trigger):
 * 1) while /bin/true; do grep gpiolib /proc/interrupts; done | uniq -c


 * Shut the guest down:
 * 1) poweroff

Guest kernel output: reboot: Power down

Host kernel output: vfio-platform e6055400.gpio: reset

The GPIO module has been reset, turning on all three LEDs.

Future Work

 * IOMMU groups

= Serial Pass through = Creating a serial pass-through will allow you to test VFIO platform pass-through using a serial loopback (USB-A -> USB Micro-B) between a host and guest using the R-Car platform.

Host Side
git://git.kernel.org/pub/scm/linux/kernel/git/kbingham/rcar.git virt/serial-passthrough
 * Utilise a host kernel configured as stated in the section Host Side above
 * To work around the lack of dynamic clock handling within QEmu, a [HACK] patch is provided to hardcode the SCIF clocks:

$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 * Configure workarounds for no IOMMU support

$ echo e6e68000.serial > /sys/bus/platform/drivers/sh-sci/unbind
 * Unbind the SCIF1 from the "sh-sci" (rcar-gen3-scif compatible) driver:

$ echo vfio-platform > /sys/bus/platform/devices/e6e68000.serial/driver_override $ echo e6e68000.serial > /sys/bus/platform/drivers/vfio-platform/bind
 * Rebind the SCIF1 to the vfio-platform driver:

Guest Side
Build QEMU using: Repository: https://github.com/kbingham/qemu.git Branch: "rcar3/serial/passthrough"

Launch with the Serial passthrough: -device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \

Full Q-Emu command: qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \ -append "loglevel=7 root=/dev/nfs nfsroot=192.168.0.39:/path/to/nfsroot/,nolock,v3 rw ip=dhcp"

Ensure that SLiRP is configured in your QEmu build to support networking, and NFS roots.

Serial loopback test
Dependencies: https://github.com/geertu/sertest * Compile and install the sertest utility in both the host, and guest root filesystem.

We can utilise a USB-A -> USB Micro-B cable as a serial-loopback to test the SCIF device passthrough.


 * Connect a USB cable between the Debug Serial 1 port, and a free available USB2 Type A socket.
 * Launch a guest with SCIF pass through as above in Guest Side

$ sertest --slave -s 9600 /dev/ttySC0
 * 'In' the *guest*:

$ sertest --master -s 9600 /dev/ttyUSB0
 * 'On' the *host*: