RZ-G/RZG2 pcie ep

From eLinux.org
< RZ-G
Revision as of 08:57, 12 January 2021 by Patersonc (talk | contribs)
(diff) โ† Older revision | Latest revision (diff) | Newer revision โ†’ (diff)
Jump to: navigation, search

๐Ÿง Renesas RZ/G2 PCIe EP

Introduction

The PCIe controller IP in RZ/G2 is capable of operating either in Root Complex mode (host) or Endpoint mode (device). When operating in endpoint mode, the controller can be configured to be used as any function depending on the use case. ("Test endpoint" is the only PCIe EP function supported in Linux kernel right now). The endpoint driver supports both MSI and legacy interrupt modes.

Source Code

Mainline Linux

Driver and DT (PCIe EP nodes in SoC DTSI) additions have been merged into upstream and are part of the Linux v5.10 release.

PCIe EP driver - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pcie-rcar-ep.c?h=v5.10

DT Documentation - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml?h=v5.10

CIP SLTS Linux v4.19

Driver and DT (PCIe EP nodes in SoC DTSI) additions have been merged into the CIP SLTS Linux v4.19 Kernel as part of the v4.19.157-cip38 release.

PCIe EP driver - https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/tree/drivers/pci/controller/pcie-rcar-ep.c?h=linux-4.19.y-cip

DT Documentation - https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/tree/Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml?h=linux-4.19.y-cip

๐Ÿ› ๏ธ HW Requirements

๐Ÿ› ๏ธ Cable Requirements for RZ/G2{HMN} Rev.4.0

Cable Requirements for RZ/G2E RevA/B

๐Ÿ› ๏ธ Hardware Modifications and Cable Requirements for RZ/G2E RevA/B

  • Remove U11 (9fgv0441) from the board
  • Use a wire to connect the pads from pin 13 (DIF0) to pin 18 (DIF1).
  • Use a wire to connect the pads from pin 14 (DIF0#) to pin 19 (DIF1#).
Hardware Modifications for RZ/G2E RevA/B




Cable Requirements for RZ/G2E RevA/B


Other Requirements

๐Ÿ“„ Host PC Tools

  • Terminal software, such as TeraTerm or HyperTerminal on Windows, to see the output from and interact with the U-Boot boot loader and the Linux Kernel.

๐Ÿ“„ Preconditions

  • Serial terminal configured for 115200 baud, 8-bit data, no parity, 1-bit stop, no flow control.
  • Hardware modifications required for the board to work as endpoint as per the requirements above.
  • pcitest utility should be part Rootfs/NFS for board working as RC.
PCIe connection


๐Ÿ“„ Configuration

Linux Driver Configuration for EP

The following config options has to be enabled in order to configure the PCI controller to be used as a โ€œEndpoint Testโ€ function driver.

CONFIG_PCI_ENDPOINT=y
CONFIG_PCI_EPF_TEST=y
CONFIG_PCIE_RCAR_EP=y

๐Ÿ“„ PCIe EP DTS configuration

By default, the DTS configures the PCIe controller to be used in root complex mode. In order to use it in endpoint mode, the following changes has to be made in dts file.

RZ/G2H HiHope board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts
index 812995939841..9ceb7bbad727 100644
--- a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts
+++ b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts
@@ -18,3 +18,11 @@
 &sata {
        status = "okay";
 };
+
+&pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
+       status = "okay";
+};

RZ/G2M HiHope board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts
index a5ca86196a7b..67802520fb0d 100644
--- a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts
+++ b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts
@@ -19,3 +19,11 @@
 &pciec1 {
        status = "okay";
 };
+
+&pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
+       status = "okay";
+};

RZ/G2N HiHope board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts
index 60d7c8adea02..f7ed164a0e34 100644
--- a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts
+++ b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts
@@ -19,3 +19,11 @@
 &sata {
        status = "okay";
 };
+
+&pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
+       status = "okay";
+};

RZ/G2E EK874 board in EP mode:

diff --git a/arch/arm64/boot/dts/renesas/cat875.dtsi b/arch/arm64/boot/dts/renesas/cat875.dtsi
index 801ea54b027c..b4edfcaf323e 100644
--- a/arch/arm64/boot/dts/renesas/cat875.dtsi
+++ b/arch/arm64/boot/dts/renesas/cat875.dtsi
@@ -41,6 +41,10 @@
 };
 
 &pciec0 {
+       status = "disabled";
+};
+
+&pciec0_ep {
        status = "okay";
 };

PCIe RC DTS configuration

By default PCIe controller is enabled in root complex mode in the DTS.

PCIe EP Setup

To find the list of endpoint controller devices in the system:

$ ls /sys/class/pci_epc/
  fe000000.pcie_ep # On upstream kernel
  fe000000.pcie-ep # On CIP SLTS Linux v4.19

To find the list of endpoint function drivers in the system:

# ls /sys/bus/pci-epf/drivers
  pci_epf_test

Using the pci-epf-test function driver

The pci-epf-test function driver can be used to test the endpoint functionality of the PCIe controller. The tests that are currently supported

  • BAR tests
  • Interrupt tests (legacy)
  • Read tests
  • Write tests
  • Copy tests

Creating pci-epf-test device

PCIe endpoint function device can be created using configfs. To create pci-epf-test device, the following commands can be used

# mount -t configfs none /sys/kernel/config
# cd /sys/kernel/config/pci_ep
# mkdir functions/pci_epf_test/func1

The โ€œmkdir functions/pci_epf_test/func1โ€ above creates the pci-epf-test function device.

The PCIe endpoint framework populates the directory with configurable fields.

# ls functions/pci_epf_test/func1
  baseclass_code    function       revid      vendorid
  cache_line_size   interrupt_pin  subclass_code
  deviceid          peripheral     subsys_id
  epc               progif_code    subsys_vendor_id

The driver populates these entries with default values when the device is bound to the driver. The pci-epf-test driver populates vendorid with 0xffff and interrupt_pin with 0x0001

# cat vendorid
  0xffff
# cat interrupt_pin
  0x0001

Creating pci-epf-test device

The user can configure the pci-epf-test device using the configfs. In order to change the vendorid and device-id, the following command can be used.

Configure Renesas as the vendor.

# echo 0x1912 > functions/pci_epf_test/func1/vendorid

Configure the device ID.

# echo 0x0025 > functions/pci_epf_test/func1/deviceid # For RZ/G2H
# echo 0x0028 > functions/pci_epf_test/func1/deviceid # For RZ/G2M
# echo 0x002b > functions/pci_epf_test/func1/deviceid # For RZ/G2N
# echo 0x002d > functions/pci_epf_test/func1/deviceid # For RZ/G2E

Configure MSI Interrupt

Set the MSI interrupts supported, msi vectors supported are 1/2/4/8/16/32, so the value to echo should be one of them.

# echo 1 > functions/pci_epf_test/func1/msi_interrupts

Note: If the root device isn't capable of handling the vectors then the endpoint device (/dev/pci-endpoint-test.0) wont be created on the host. On G2x platforms max value should be 16.

Binding pci-epf-test device to a EP controller

In order for the endpoint function device to be useful, it has to be bound to a PCIe endpoint controller driver. Use the configfs to bind the function device to one of the controller driver present in the system.

# ln -s functions/pci_epf_test/func1/ controllers/fe000000.pcie_ep/ # On upstream kernel
# ln -s functions/pci_epf_test/func1/ controllers/fe000000.pcie-ep/ # On CIP SLTS Linux v4.19

Starting the EP device

In order for the EP device to be ready to establish the link, the following command should be given.

# echo 1 > controllers/fe000000.pcie_ep/start # On upstream kernel
# echo 1 > controllers/fe000000.pcie-ep/start # On CIP SLTS Linux v4.19

Once the above step is completed, the PCIe endpoint is ready to establish a link with the host (RC).

PCIe RC

The PCIe EP device must be powered-on and configured before the PCIe HOST device. This restriction is because the PCIe HOST driver doesnโ€™t support hot plug.

Endpoint Test function device

pci_endpoint_test driver creates the Endpoint Test function device (/dev/pci-endpoint-test.0) which will be used by the following pcitest utility. pci_endpoint_test can either be built-in to the kernel or built as a module.

# insmod pci_endpoint_test.ko

Verification of PCIe EP on RC using lspci

root@salvator-x:~# lspci -vv
00:00.0 PCI bridge: Renesas Technology Corp. Device 002d (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin ? routed to IRQ 141
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: None
Memory behind bridge: fe200000-fe2fffff [size=1M]
Prefetchable memory behind bridge: None
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000b4924000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis L[ 140.002782] random: crng init done
[ 140.006278] random: 7 urandom warning(s) missed due to ratelimiting
evel: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Unassigned class [ff00]: Renesas Technology Corp. Device 002d
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 132
Region 0: Memory at fe200200 (64-bit, non-prefetchable) [size=128]
Region 2: Memory at fe200000 (64-bit, non-prefetchable) [size=256]
Region 4: Memory at fe200100 (64-bit, non-prefetchable) [size=256]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Kernel driver in use: pci-endpoint-test

pcitest

Building pcitest

# cd <kernel-dir>
# make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- -C tools/pci/
# cp pcitest <rootfs>/usr/sbin/

Note: The pcitest utility tool is run on PCIe RC and hence needs to be in root file system of PCIe RC device.

pcitest options

$ pcitest  -h
usage: pcitest [options]
Options:
        -D <dev>                PCI endpoint test device {default: /dev/pci-endpoint-test.0}
        -b <bar num>            BAR test (bar number between 0..5)
        -m <msi num>            MSI test (msi number between 1..32)
        -x <msix num>           MSI-X test (msix number between 1..2048)
        -i <irq type>           Set IRQ type (0 - Legacy, 1 - MSI, 2 - MSI-X)
        -I                      Get current IRQ type configured
        -l                      Legacy IRQ test
        -r                      Read buffer test
        -w                      Write buffer test
        -c                      Copy buffer test
        -s <size>               Size of buffer {default: 100KB}
        -h                      Print this help message

BAR test

$ pcitest -b 0
BAR0:           OKAY
$ pcitest -b 2
BAR2:           OKAY
$ pcitest -b 4
BAR4:           OKAY

๐Ÿ’ก Note: BAR's 0/2/4 are 64-bit hence BAR's 1/3/5 are not usable

Legacy interrupt test

$ pcitest -i 0
SET IRQ TYPE TO LEGACY:         OKAY
$ pcitest -l
LEGACY IRQ:     OKAY

MSI interrupt test

$ pcitest -i 1
SET IRQ TYPE TO MSI:         OKAY

$ pcitest -m x (Note: value of x can range from 1 to value echoed into msi_interrupts)

Example to test MSI1:
$ pcitest -i 1
SET IRQ TYPE TO MSI:         OKAY
$ pcitest -m 1
MSI1:     OKAY

READ test

$ pcitest -r 1024
READ ( 102400 bytes):           OKAY

WRITE test

$ pcitest -w 1024
WRITE ( 102400 bytes):          OKAY

COPY test

$ pcitest -c 1024
COPY ( 102400 bytes):           OKAY

๐Ÿ’ก Note: Before running READ/WRITE/COPY test cases make sure the interrupt mode is either set to Legacy/MSI

๐Ÿ’ก Known Issues

  • On RZ/G2 platform when using as RC the read/write/copy tests fail due to TLB issues. (Refer https://www.spinics.net/lists/linux-pci/msg92385.html for more details). To get around with this issue use the attached patch gfp-dma.patch for upstream kernel. 
  • For RZ/G2E RevA/B boards hardware modification is required along with different cable as described above.