RZ-G/RZG2 pcie ep

= 🐧 Renesas RZ/G2 PCIe EP =

Introduction
The PCIe controller IP in RZ/G2 is capable of operating either in Root Complex mode (host) or Endpoint mode (device). When operating in endpoint mode, the controller can be configured to be used as any function depending on the use case. ("Test endpoint" is the only PCIe EP function supported in Linux kernel right now). The endpoint driver supports both MSI and legacy interrupt modes.

Source Code
Driver and DT (PCIe EP nodes in SoC DTSI) additions have been merged into upstream and are part of the Linux 5.10 release.

PCIe EP driver - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pcie-rcar-ep.c?h=v5.10

DT Documentation - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml?h=v5.10

🛠️ Hardware Modifications and Cable Requirements for RZ/G2E RevA/B

 * Remove U11 (9fgv0441) from the board
 * Use a wire to connect the pads from pin 13 (DIF0) to pin 18 (DIF1).
 * Use a wire to connect the pads from pin 14 (DIF0#) to pin 19 (DIF1#).



📄 Host PC Tools

 * Terminal software, such as TeraTerm or HyperTerminal on Windows, to see the output from and interact with the U-Boot boot loader and the Linux Kernel.

📄 Preconditions

 * Serial terminal configured for 115200 baud, 8-bit data, no parity, 1-bit stop, no flow control.
 * Hardware modifications required for the board to work as endpoint as per the requirements above.
 * pcitest utility should be part Rootfs/NFS for board working as RC.

Linux Driver Configuration for EP
The following config options has to be enabled in order to configure the PCI controller to be used as a “Endpoint Test” function driver.

CONFIG_PCI_ENDPOINT=y CONFIG_PCI_EPF_TEST=y CONFIG_PCIE_RCAR_EP=y

📄 PCIe EP DTS configuration
By default, the DTS configures the PCIe controller to be used in root complex mode. In order to use it in endpoint mode, the following changes has to be made in dts file.

RZ/G2H HiHope board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts index 812995939841..9ceb7bbad727 100644 --- a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts +++ b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts @@ -18,3 +18,11 @@ &sata { status = "okay"; }; + +&pciec0 { +      status = "disabled"; +}; + +&pciec0_ep { +      status = "okay"; +};

RZ/G2M HiHope board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts index a5ca86196a7b..67802520fb0d 100644 --- a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts +++ b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts @@ -19,3 +19,11 @@ &pciec1 { status = "okay"; }; + +&pciec0 { +      status = "disabled"; +}; + +&pciec0_ep { +      status = "okay"; +};

RZ/G2N HiHope board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts index 60d7c8adea02..f7ed164a0e34 100644 --- a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts +++ b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts @@ -19,3 +19,11 @@ &sata { status = "okay"; }; + +&pciec0 { +      status = "disabled"; +}; + +&pciec0_ep { +      status = "okay"; +};

RZ/G2E EK874 board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/cat875.dtsi b/arch/arm64/boot/dts/renesas/cat875.dtsi index 801ea54b027c..b4edfcaf323e 100644 --- a/arch/arm64/boot/dts/renesas/cat875.dtsi +++ b/arch/arm64/boot/dts/renesas/cat875.dtsi @@ -41,6 +41,10 @@ }; &pciec0 { +      status = "disabled"; +}; + +&pciec0_ep { status = "okay"; };

PCIe RC DTS configuration
By default PCIe controller is enabled in root complex mode in the DTS.

PCIe EP Setup
To find the list of endpoint controller devices in the system: fe000000.pcie_ep
 * 1) ls /sys/class/pci_epc/

To find the list of endpoint function drivers in the system: pci_epf_test
 * 1) ls /sys/bus/pci-epf/drivers

Using the pci-epf-test function driver
The pci-epf-test function driver can be used to test the endpoint functionality of the PCIe controller. The tests that are currently supported


 * BAR tests
 * Interrupt tests (legacy)
 * Read tests
 * Write tests
 * Copy tests

Creating pci-epf-test device
PCIe endpoint function device can be created using configfs. To create pci-epf-test device, the following commands can be used The “mkdir functions/pci_epf_test/func1” above creates the pci-epf-test function device.
 * 1) mount -t configfs none /sys/kernel/config
 * 2) cd /sys/kernel/config/pci_ep
 * 3) mkdir functions/pci_epf_test/func1

The PCIe endpoint framework populates the directory with configurable fields. baseclass_code   function       revid      vendorid cache_line_size  interrupt_pin  subclass_code deviceid         peripheral     subsys_id epc              progif_code    subsys_vendor_id
 * 1) ls functions/pci_epf_test/func1

The driver populates these entries with default values when the device is bound to the driver. The pci-epf-test driver populates vendorid with 0xffff and interrupt_pin with 0x0001

0xffff 0x0001
 * 1) cat vendorid
 * 1) cat interrupt_pin

Creating pci-epf-test device
The user can configure the pci-epf-test device using the configfs. In order to change the vendorid and device-id, the following command can be used.

Configure Renesas as the vendor.
 * 1) echo 0x1912 > functions/pci_epf_test/func1/vendorid

Configure the device ID.
 * 1) echo 0x0025 > functions/pci_epf_test/func1/deviceid # For RZ/G2H
 * 2) echo 0x0028 > functions/pci_epf_test/func1/deviceid # For RZ/G2M
 * 3) echo 0x002b > functions/pci_epf_test/func1/deviceid # For RZ/G2N
 * 4) echo 0x002d > functions/pci_epf_test/func1/deviceid # For RZ/G2E

Configure MSI Interrupt
Set the MSI interrupts supported, msi vectors supported are 1/2/4/8/16/32, so the value to echo should be one of them. Note: If the root device isn't capable of handling the vectors then the endpoint device (/dev/pci-endpoint-test.0) wont be created on the host. On G2x platforms max value should be 16.
 * 1) echo 1 > functions/pci_epf_test/func1/msi_interrupts

Binding pci-epf-test device to a EP controller
In order for the endpoint function device to be useful, it has to be bound to a PCIe endpoint controller driver. Use the configfs to bind the function device to one of the controller driver present in the system.
 * 1) ln -s functions/pci_epf_test/func1/ controllers/fe000000.pcie_ep/

Starting the EP device
In order for the EP device to be ready to establish the link, the following command should be given. Once the above step is completed, the PCIe endpoint is ready to establish a link with the host (RC).
 * 1) echo 1 > controllers/fe000000.pcie_ep/start

PCIe RC
The PCIe EP device must be powered-on and configured before the PCIe HOST device. This restriction is because the PCIe HOST driver doesn’t support hot plug.

Endpoint Test function device
pci_endpoint_test driver creates the Endpoint Test function device (/dev/pci-endpoint-test.0) which will be used by the following pcitest utility. pci_endpoint_test can either be built-in to the kernel or built as a module.
 * 1) insmod pci_endpoint_test.ko

Verification of PCIe EP on RC using lspci
root@salvator-x:~# lspci -vv 00:00.0 PCI bridge: Renesas Technology Corp. Device 002d (prog-if 00 [Normal decode]) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+ Address: 00000000b4924000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd- AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- AtomicOpsCtl: ReqEn- EgressBlck- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis L[ 140.002782] random: crng init done [ 140.006278] random: 7 urandom warning(s) missed due to ratelimiting evel: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Kernel driver in use: pcieport lspci: Unable to load libkmod resources: error -12

01:00.0 Unassigned class [ff00]: Renesas Technology Corp. Device 002d Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 132 Region 0: Memory at fe200200 (64-bit, non-prefetchable) [size=128] Region 2: Memory at fe200000 (64-bit, non-prefetchable) [size=256] Region 4: Memory at fe200100 (64-bit, non-prefetchable) [size=256] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Kernel driver in use: pci-endpoint-test

Building pcitest
Note: The pcitest utility tool is run on PCIe RC and hence needs to be in root file system of PCIe RC device.
 * 1) cd 
 * 2) make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- -C tools/pci/
 * 3) cp pcitest /usr/sbin/

pcitest options
$ pcitest -h usage: pcitest [options] Options: -D               PCI endpoint test device {default: /dev/pci-endpoint-test.0} -b           BAR test (bar number between 0..5) -m           MSI test (msi number between 1..32) -x          MSI-X test (msix number between 1..2048) -i          Set IRQ type (0 - Legacy, 1 - MSI, 2 - MSI-X) -I                     Get current IRQ type configured -l                     Legacy IRQ test -r                     Read buffer test -w                     Write buffer test -c                     Copy buffer test -s              Size of buffer {default: 100KB} -h                     Print this help message

BAR test
$ pcitest -b 0 BAR0:          OKAY $ pcitest -b 2 BAR2:          OKAY $ pcitest -b 4 BAR4:          OKAY

💡 Note: BAR's 0/2/4 are 64-bit hence BAR's 1/3/5 are not usable

Legacy interrupt test
$ pcitest -i 0 SET IRQ TYPE TO LEGACY:        OKAY $ pcitest -l LEGACY IRQ:    OKAY

MSI interrupt test
$ pcitest -i 1 SET IRQ TYPE TO MSI:        OKAY

$ pcitest -m x (Note: value of x can range from 1 to value echoed into msi_interrupts)

Example to test MSI1: $ pcitest -i 1 SET IRQ TYPE TO MSI:        OKAY $ pcitest -m 1 MSI1:    OKAY

READ test
$ pcitest -r 1024 READ ( 102400 bytes):          OKAY

WRITE test
$ pcitest -w 1024 WRITE ( 102400 bytes):         OKAY

COPY test
$ pcitest -c 1024 COPY ( 102400 bytes):          OKAY

💡 Note: Before running READ/WRITE/COPY test cases make sure the interrupt mode is either set to Legacy/MSI

💡 Known Issues

 * On RZ/G2 platform when using as RC the read/write/copy tests fail due to TLB issues. (Refer https://www.spinics.net/lists/linux-pci/msg92385.html for more details). To get around with this issue use the attached patch gfp-dma.patch for upstream kernel.
 * For RZ/G2E RevA/B boards hardware modification is required along with different cable as described above.