R-Car/Virtualization/VFIO

Device Pass-Through Using VFIO

= GPIO Pass-Through Prototype =

This is a proof-of-concept showing how to provide guest access to an R-Car GPIO controller block on the Renesas Salvator-X and Salvator-XS boards.

Host Side

 * Build and boot a host kernel using:
 * Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git
 * Branch: "topic/rcar3-virt-gpio-passthrough-v3"
 * Config: "renesas_defconfig"

$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 * Configure workarounds for missing functionality:

$ echo e6055400.gpio > /sys/bus/platform/drivers/gpio_rcar/unbind gpio gpiochip6: REMOVING GPIOCHIP WITH GPIOS STILL REQUESTED
 * Unbind GPIO6 from the "gpio-rcar" driver:

$ echo vfio-platform > /sys/bus/platform/devices/e6055400.gpio/driver_override $ echo e6055400.gpio > /sys/bus/platform/drivers/vfio-platform/bind iommu: Adding device e6055400.gpio to group 0 vfio-platform e6055400.gpio: Adding kernel taint for vfio-noiommu group on device
 * Bind GPIO6 to the "vfio-platform" driver, for pass-through to guests:

Guest Side

 * Build a guest kernel using:
 * Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git
 * Branch: "topic/rcar3-virt-gpio-passthrough-v3"
 * Config: "virt_defconfig"


 * Build QEMU using:
 * Repository: https://github.com/geertu/qemu.git
 * Branch: "topic/rcar3-virt-gpio-passthrough-v3"

I.e.: -device vfio-platform,host=e6055400.gpio -device vfio-platform,sysfsdev=/sys/bus/platform/devices/e6055400.gpio
 * When starting QEMU, specify pass-through of the GPIO6 platform device. The device specifier consists of either the device's node name, or the full path to the device's node in sysfs.

Full command line: $ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=e6055400.gpio

Guest kernel output: Booting Linux on physical CPU 0x0000000000 [0x411fd073] Linux version 4.18.0-rc6-arm64-virt-00012-g33dccc007797d60a ... ... gpio_rcar c000000.e6055400.gpio: driving 32 GPIOs ...

Host kernel output: vfio-platform e6055400.gpio: reset vfio-platform e6055400.gpio: vfio-noiommu device opened by user (qemu-system-aar:3003)

--- /sys/firmware/devicetree/base.orig +++ /sys/firmware/devicetree/base @@ -126,6 +126,14 @@               compatible = "qemu,platform", "simple-bus"; interrupt-parent = ; ranges = <0x0 0x0 0xc000000 0x2000000>; + +              e6055400.gpio@0 { +                      #gpio-cells = ; +                      #interrupt-cells = ; +                      compatible = "renesas,gpio-r8a7795", "renesas,rcar-gen3-gpio"; +                      gpio-controller; +                      interrupt-controller; +                      interrupts = ; +                      reg = ; +              };        };
 * Analysis of "/sys/firmware/devicetree/base" using "dtx_diff" shows that "e6055400.gpio@0" has been added as a subnode to the existing "platform@c000000" node:

pmu {

echo $(($base + $i)) > /sys/class/gpio/export echo low > /sys/class/gpio/gpio$(($base + $i))/direction done
 * Export GPIOs used for LEDs (LED4/5/6), and turn all LEDs off:
 * 1) base=$(cat /sys/class/gpio/gpiochip*/base)
 * 2) for i in 11 12 13; do

echo high > /sys/class/gpio/gpio$(($base + $i))/direction sleep 1 echo low > /sys/class/gpio/gpio$(($base + $i))/direction done
 * Cycle through all LEDs, letting them blink once:
 * 1) for i in 11 12 13; do

5:         0     GIC-0 144 Level     c000000.e6055400.gpio 42:         0  c000000.e6055400.gpio  11 Edge      gpiolib
 * Change the first GPIO from output to input, and configure edge detection to enable its interrupt:
 * 1) echo in > /sys/class/gpio/gpio$(($base + 11))/direction
 * 2) echo both > /sys/class/gpio/gpio$(($base + 11))/edge
 * 3) grep gpio /proc/interrupts

71298 42:          0  c000000.e6055400.gpio  11 Edge      gpiolib 388 42:          1  c000000.e6055400.gpio  11 Edge      gpiolib 247 42:          2  c000000.e6055400.gpio  11 Edge      gpiolib 139 42:          3  c000000.e6055400.gpio  11 Edge      gpiolib 586 42:          4  c000000.e6055400.gpio  11 Edge      gpiolib ^C
 * Monitor the GPIO interrupt (press button SW20 to trigger):
 * 1) while /bin/true; do grep gpiolib /proc/interrupts; done | uniq -c


 * Shut the guest down:
 * 1) poweroff

Guest kernel output: reboot: Power down

Host kernel output: vfio-platform e6055400.gpio: reset

The GPIO module has been reset, turning on all three LEDs.

= Serial Pass through = Creating a serial pass-through will allow you to test VFIO platform pass-through using a serial loopback (USB-A -> USB Micro-B) between a host and guest using the R-Car platform.

Host Side
git://git.kernel.org/pub/scm/linux/kernel/git/kbingham/rcar.git virt/serial-passthrough
 * Utilise a host kernel configured as stated in the section Host Side above
 * To work around the lack of dynamic clock handling within QEmu, a [HACK] patch is provided to hardcode the SCIF clocks:

$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 * Configure workarounds for no IOMMU support

$ echo e6e68000.serial > /sys/bus/platform/drivers/sh-sci/unbind
 * Unbind the SCIF1 from the "sh-sci" (rcar-gen3-scif compatible) driver:

$ echo vfio-platform > /sys/bus/platform/devices/e6e68000.serial/driver_override $ echo e6e68000.serial > /sys/bus/platform/drivers/vfio-platform/bind
 * Rebind the SCIF1 to the vfio-platform driver:

Guest Side
Build QEMU using: Repository: https://github.com/kbingham/qemu.git Branch: "rcar3/serial/passthrough"

Launch with the Serial passthrough: -device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \

Full Q-Emu command: qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \ -append "loglevel=7 root=/dev/nfs nfsroot=192.168.0.39:/path/to/nfsroot/,nolock,v3 rw ip=dhcp"

Ensure that SLiRP is configured in your QEmu build to support networking, and NFS roots.

Serial loopback test
Dependencies: https://github.com/geertu/sertest * Compile and install the sertest utility in both the host, and guest root filesystem.

We can utilise a USB-A -> USB Micro-B cable as a serial-loopback to test the SCIF device passthrough.


 * Connect a USB cable between the Debug Serial 1 port, and a free available USB2 Type A socket.
 * Launch a guest with SCIF pass through as above in Guest Side

$ sertest --slave -s 9600 /dev/ttySC0
 * 'In' the *guest*:

$ sertest --master -s 9600 /dev/ttyUSB0
 * 'On' the *host*:

= SATA Pass-Through =

This shows how to provide guest access to an R-Car SATA controller block on the Renesas Salvator-X and Salvator-XS boards.

Host Side

 * Connect a SATA device to the SATA port on Salvator-X(S) (R-Car H3 only!).


 * Build and boot a host kernel according to section Host Side above.

$ echo ee300000.sata > /sys/bus/platform/drivers/sata_rcar/unbind ata1.00: disabled sd 0:0:0:0: [sda] Synchronizing SCSI cache sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00 sd 0:0:0:0: [sda] Stopping disk sd 0:0:0:0: [sda] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00
 * Unbind SATA from the "sata_rcar" driver (actual SATA device output may vary):

$ echo vfio-platform > /sys/bus/platform/devices/ee300000.sata/driver_override $ echo ee300000.sata > /sys/bus/platform/drivers/vfio-platform/bind
 * Bind SATA to the "vfio-platform" driver, for pass-through to guests:

Guest Side

 * Build a guest kernel and QEMU according to section Guest Side above.

$ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=ee300000.sata
 * Launch QEMU:

Guest kernel output (actual SATA device output may vary): Booting Linux on physical CPU 0x0000000000 [0x411fd073] Linux version 4.18.0-rc6-arm64-virt-00012-g33dccc007797d60a ... ... scsi host0: sata_rcar ata1: SATA max UDMA/133 irq 5 ... ata1: link resume succeeded after 1 retries ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata1.00: ATA-7: Maxtor 6L160M0, BANC1G10, max UDMA/133 ata1.00: 320173056 sectors, multi 0: LBA48 NCQ (not used) ata1.00: configured for UDMA/133 scsi 0:0:0:0: Direct-Access    ATA      Maxtor 6L160M0   1G10 PQ: 0 ANSI: 5 sd 0:0:0:0: [sda] 320173056 512-byte logical blocks: (164 GB/153 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sd 0:0:0:0: [sda] Attached SCSI disk ...

Host kernel output: vfio-platform ee300000.sata: Using IPMMU context 1 vfio-platform ee300000.sata: reset

--- /sys/firmware/devicetree/base.orig +++ /sys/firmware/devicetree/base @@ -126,6 +126,12 @@               compatible = "qemu,platform", "simple-bus"; interrupt-parent = ; ranges = <0x0 0x0 0xc000000 0x2000000>; +  +               ee300000.sata@0 { +                      compatible = "renesas,sata-r8a7795", "renesas,rcar-gen3-sata"; +                      interrupts = ; +                      reg = ; +              };        };
 * Analysis of "/sys/firmware/devicetree/base" using "dtx_diff" shows that "ee300000.sata@0" has been added as a subnode to the existing "platform@c000000" node:

pmu {


 * Depending on the SATA device, you can mount a partition.


 * Shut the guest down:
 * 1) poweroff

Guest kernel output: reboot: Power down

Host kernel output: vfio-platform ee300000.sata: reset vfio-platform ee300000.sata: Reusing IPMMU context 0