Tests:I2C-fault-injection

This document describes how to test the I2C fault injection feature. The Renesas Salvator-X (R-Car M3-W) and Lager (R-Car H2) boards are used.

Setup

Kernel Version and Configuration

Support for this feature is upstream since v4.16. An update is scheduled for v4.19. Please note that in this update, the file incomplete_transfer has been renamed to incomplete_address_phase to distinguish it better from another fault injector.

The original patches are available in a topic branches:

git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux.git renesas/topic/i2c-fault-injection

git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux.git renesas/i2c/recovery/write-byte-fix

The following kernel options need to be set in addition to the defconfig:

CONFIG_I2C_GPIO=y
CONFIG_I2C_GPIO_FAULT_INJECTOR=y

Hardware Environment

Salvator-X/r8a7796 (R-Car M3-W SoC)

On the EXIO_B connector, you need to connect the pins:

57 <-> 73 (for SCL)
54 <-> 74 (for SDA)

This will hook two GPIO to the I2C2 bus. The above branch will update the DTS to enable the GPIO as a bit-banged I2C bus.

Lager/r8a7790 (R-Car H2 SoC)

You need to connect the pins:

EXIO_C 78 <-> EXIO_A 58 (for SCL)
EXIO_C 80 <-> EXIO_A 60 (for SDA)

This will connect the I2C1 and the I2C2 bus. We will use the I2C demultiplexer at runtime to switch I2C1 to GPIO:

# echo 2 > /sys/devices/platform/i2c-11/current_master

Software Environment

Only a busybox with standard commands like cd, cat, echo is needed.

Testing

On both boards, you should go to the i2c fault injector directory and see the following files:

# cd /sys/kernel/debug/i2c-fault-injector/i2c-8/
# ls -1
incomplete_address_phase
incomplete_write_byte
scl
sda

Please refer to the documentation file in Documentation/i2c/gpio-fault-injection about the meaning of the files.

Incomplete transfer

An incomplete transfer is used to make a device keep SDA low. Here the first example of stopping the transfer during acknowledgment of the address of the client.

Salvator-X/r8a7796 (R-Car M3-W SoC)

Use these commands:

# cd /sys/kernel/debug/i2c-fault-injector/i2c-8/
# echo 0x10 > incomplete_address_phase

With a scope, you can see the following happening:

You can see that SDA is kept low. After 45ms, SCL is toggled again, so the device releases SDA again. This is done by the external LTC4313 chip which monitors the bus and tries to reactivate it. So, the bus stalled condition was met which made the LTC4313 reactivate the bus.

Lager/r8a7790 (R-Car H2 SoC)

Use these commands:

# echo 2 > /sys/devices/platform/i2c-11/current_master
# cd /sys/kernel/debug/i2c-fault-injector/i2c-8/
# echo 0x5a > incomplete_address_phase

With a scope, you can see the following happening:

You can see that SDA is kept low. After 30ms, SDA raises without SCL being toggled. This is because the DA9xxx chips detect the timeout on SCL and release SDA on their own. So, the bus stalled condition was met which made the DA9xxx reactivate the bus.

Checking another device:

# echo 0x12 > incomplete_address_phase

With a scope, you can see the following happening:

You can see that SDA is kept low forever. The audio codec does not have any timeout detection and there is no external chip to reactivate the bus. It is now the I2C bus masters turn to detect this condition and to recover the bus by toggling SCL.

A similar situation gets created by the "incomplete_write_byte" fault injector. However, it is more delicate, from its documentation:

       The injector will again stop at one ACK phase, so the device will keep SDA low
       because it acknowledges data. However, there are two differences compared to
       'incomplete_address_phase':
       
       a) the message sent out will be a write message
       b) after the address byte, a 0x00 byte will be transferred. Then, stop at ACK.
       
       This is a highly delicate state, the device is set up to write any data to
       register 0x00 (if it has registers) when further clock pulses happen on SCL.
       This is why bus recovery (up to 9 clock pulses) must either check SDA or send
       additional STOP conditions to ensure the bus has been released. Otherwise
       random data will be written to a device!

Here is detailed scope of it to visualize what it does.

Pin SDA low

Use these commands on the Salvator-X:

# cd /sys/kernel/debug/i2c-fault-injector/i2c-8/
# echo 0 > sda

With a scope, you can see the following happening:

Now, SDA is pinned low by us until we release it. You can see the LTC4313 trying to recover the bus, but to no avail. The bus can only be released with

# echo 1 > sda

On the Lager board, nothing special will happen because there is no LTC4313. SDA just stays low until it gets manually released again.

Pin SCL low

Use these commands on the Salvator-X or Lager:

# cd /sys/kernel/debug/i2c-fault-injector/i2c-8/
# echo 0 > scl

With a scope, you can see the following happening:

Now, SCL is pinned low by us until we release it. No communication is possible anymore. The bus can only be released with:

# echo 1 > scl

Conclusion

It could be demonstrated that the incomplete transfer fault injectors create a stalled bus. It could be observed that some devices detect this condition and try to recover the bus. Cases which create a completely stalled bus could also be created (e.g. by the audio codec on Lager). This can serve as a test case for the bus recovery feature of the Linux I2C core. Also, the features of pinning SCL and SDA permanently low could be demonstrated. This can be used to make I2C drivers handle this abnormal situation gracefully.