Difference between revisions of "RZ-G/RZG2 pcie ep"
Prabhakarlad (talk | contribs) m (→💡 Known Issues) |
Prabhakarlad (talk | contribs) m (→COPY test) |
||
Line 416: | Line 416: | ||
</pre> | </pre> | ||
− | '''💡 Note:''' Before running READ/WRITE/COPY test cases make sure | + | '''💡 Note:''' Before running READ/WRITE/COPY test cases make sure the interrupt mode is either set to Legacy/MSI |
== 💡 Known Issues == | == 💡 Known Issues == | ||
* On RZ/G2 platform when using as RC the read/write/copy tests fail due to TLB issues. (Refer https://www.spinics.net/lists/linux-pci/msg92385.html for more details). To get around with this issue use the attached patch [https://elinux.org/images/e/e8/0001-misc-pci_endpoint_test-HACK-Use-GFP_DMA-flag-for-rea.patch gfp-dma.patch] for upstream kernel. | * On RZ/G2 platform when using as RC the read/write/copy tests fail due to TLB issues. (Refer https://www.spinics.net/lists/linux-pci/msg92385.html for more details). To get around with this issue use the attached patch [https://elinux.org/images/e/e8/0001-misc-pci_endpoint_test-HACK-Use-GFP_DMA-flag-for-rea.patch gfp-dma.patch] for upstream kernel. | ||
* For RZ/G2E RevA/B boards hardware modification is required along with different cable as described above. | * For RZ/G2E RevA/B boards hardware modification is required along with different cable as described above. |
Revision as of 15:20, 22 December 2020
🐧 Renesas RZ/G2 PCIe EP
Introduction
The PCIe controller IP in RZ/G2 is capable of operating either in Root Complex mode (host) or Endpoint mode (device). When operating in endpoint mode, the controller can be configured to be used as any function depending on the use case. ("Test endpoint" is the only PCIe EP function supported in Linux kernel right now). The endpoint driver supports both MSI and legacy interrupt modes.
Source Code
Driver and DT (PCIe EP nodes in SoC DTSI) additions have been merged into upstream and are part of the Linux 5.10 release.
PCIe EP driver - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pcie-rcar-ep.c?h=v5.10
DT Documentation - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml?h=v5.10
🛠️ Cable Requirements for RZ/G2{HMN} Rev.4.0
🛠️ Hardware Modifications and Cable Requirements for RZ/G2E RevA/B
- Remove U11 (9fgv0441) from the board
- Use a wire to connect the pads from pin 13 (DIF0) to pin 18 (DIF1).
- Use a wire to connect the pads from pin 14 (DIF0#) to pin 19 (DIF1#).
📄 Host PC Tools
- Terminal software, such as TeraTerm or HyperTerminal on Windows, to see the output from and interact with the U-Boot boot loader and the Linux Kernel.
📄 Preconditions
- Serial terminal configured for 115200 baud, 8-bit data, no parity, 1-bit stop, no flow control.
- Hardware modifications required for the board to work as endpoint as per the requirements above.
- pcitest utility should be part Rootfs/NFS for board working as RC.
📄 Linux Configuration
Driver Configuration for EP
The following config options has to be enabled in order to configure the PCI controller to be used as a “Endpoint Test” function driver.
CONFIG_PCI_ENDPOINT=y CONFIG_PCI_EPF_TEST=y CONFIG_PCIE_RCAR_EP=y
📄 PCIe EP DTS configuration
By default, the DTS configures the PCIe controller to be used in root complex mode. In order to use it in endpoint mode, the following changes has to be made in dts file.
RZ/G2H HiHope board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts index 812995939841..9ceb7bbad727 100644 --- a/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts +++ b/arch/arm64/boot/dts/renesas/r8a774e1-hihope-rzg2h-ex.dts @@ -18,3 +18,11 @@ &sata { status = "okay"; }; + +&pciec0 { + status = "disabled"; +}; + +&pciec0_ep { + status = "okay"; +};
RZ/G2M HiHope board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts index a5ca86196a7b..67802520fb0d 100644 --- a/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts +++ b/arch/arm64/boot/dts/renesas/r8a774a1-hihope-rzg2m-ex.dts @@ -19,3 +19,11 @@ &pciec1 { status = "okay"; }; + +&pciec0 { + status = "disabled"; +}; + +&pciec0_ep { + status = "okay"; +};
RZ/G2N HiHope board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts index 60d7c8adea02..f7ed164a0e34 100644 --- a/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts +++ b/arch/arm64/boot/dts/renesas/r8a774b1-hihope-rzg2n-ex.dts @@ -19,3 +19,11 @@ &sata { status = "okay"; }; + +&pciec0 { + status = "disabled"; +}; + +&pciec0_ep { + status = "okay"; +};
RZ/G2E EK874 board in EP mode:
diff --git a/arch/arm64/boot/dts/renesas/cat875.dtsi b/arch/arm64/boot/dts/renesas/cat875.dtsi index 801ea54b027c..b4edfcaf323e 100644 --- a/arch/arm64/boot/dts/renesas/cat875.dtsi +++ b/arch/arm64/boot/dts/renesas/cat875.dtsi @@ -41,6 +41,10 @@ }; &pciec0 { + status = "disabled"; +}; + +&pciec0_ep { status = "okay"; };
PCIe RC DTS configuration
By default PCIe controller is enabled in root complex mode in the DTS.
PCIe EP Setup
To find the list of endpoint controller devices in the system:
# ls /sys/class/pci_epc/ fe000000.pcie_ep
To find the list of endpoint function drivers in the system:
# ls /sys/bus/pci-epf/drivers pci_epf_test
Using the pci-epf-test function driver
The pci-epf-test function driver can be used to test the endpoint functionality of the PCIe controller. The tests that are currently supported
- BAR tests
- Interrupt tests (legacy)
- Read tests
- Write tests
- Copy tests
Creating pci-epf-test device
PCIe endpoint function device can be created using configfs. To create pci-epf-test device, the following commands can be used
# mount -t configfs none /sys/kernel/config # cd /sys/kernel/config/pci_ep # mkdir functions/pci_epf_test/func1
The “mkdir functions/pci_epf_test/func1” above creates the pci-epf-test function device.
The PCIe endpoint framework populates the directory with configurable fields.
# ls functions/pci_epf_test/func1 baseclass_code function revid vendorid cache_line_size interrupt_pin subclass_code deviceid peripheral subsys_id epc progif_code subsys_vendor_id
The driver populates these entries with default values when the device is bound to the driver. The pci-epf-test driver populates vendorid with 0xffff and interrupt_pin with 0x0001
# cat vendorid 0xffff # cat interrupt_pin 0x0001
Creating pci-epf-test device
The user can configure the pci-epf-test device using the configfs. In order to change the vendorid and device-id, the following command can be used.
Configure Renesas as the vendor.
# echo 0x1912 > functions/pci_epf_test/func1/vendorid
Configure the device ID.
# echo 0x0025 > functions/pci_epf_test/func1/deviceid # For RZ/G2H # echo 0x0028 > functions/pci_epf_test/func1/deviceid # For RZ/G2M # echo 0x002b > functions/pci_epf_test/func1/deviceid # For RZ/G2N # echo 0x002d > functions/pci_epf_test/func1/deviceid # For RZ/G2E
Configure MSI Interrupt
Set the MSI interrupts supported, msi vectors supported are 1/2/4/8/16/32, so the value to echo should be one of them.
# echo 1 > functions/pci_epf_test/func1/msi_interrupts
Note: If the root device isn't capable of handling the vectors then the endpoint device (/dev/pci-endpoint-test.0) wont be created on the host. On G2x platforms max value should be 16.
Binding pci-epf-test device to a EP controller
In order for the endpoint function device to be useful, it has to be bound to a PCIe endpoint controller driver. Use the configfs to bind the function device to one of the controller driver present in the system.
# ln -s functions/pci_epf_test/func1/ controllers/fe000000.pcie_ep/
Starting the EP device
In order for the EP device to be ready to establish the link, the following command should be given.
# echo 1 > controllers/fe000000.pcie_ep/start
Once the above step is completed, the PCIe endpoint is ready to establish a link with the host (RC).
PCIe RC Setup
The PCIe EP device must be powered-on and configured before the PCIe HOST device. This restriction is because the PCIe HOST driver doesn’t have hot plug support.
lspci output
root@salvator-x:~# lspci -vv 00:00.0 PCI bridge: Renesas Technology Corp. Device 002d (prog-if 00 [Normal decode]) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin ? routed to IRQ 141 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: None Memory behind bridge: fe200000-fe2fffff [size=1M] Prefetchable memory behind bridge: None Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+ Address: 00000000b4924000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd- AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- AtomicOpsCtl: ReqEn- EgressBlck- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis L[ 140.002782] random: crng init done [ 140.006278] random: 7 urandom warning(s) missed due to ratelimiting evel: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Kernel driver in use: pcieport lspci: Unable to load libkmod resources: error -12 01:00.0 Unassigned class [ff00]: Renesas Technology Corp. Device 002d Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 132 Region 0: Memory at fe200200 (64-bit, non-prefetchable) [size=128] Region 2: Memory at fe200000 (64-bit, non-prefetchable) [size=256] Region 4: Memory at fe200100 (64-bit, non-prefetchable) [size=256] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Kernel driver in use: pci-endpoint-test
Using the Endpoint Test function device
pci_endpoint_test driver creates the Endpoint Test function device (/dev/pci-endpoint-test.0) which will be used by the following pcitest utility. pci_endpoint_test can either be built-in to the kernel or built as a module.
# insmod pci_endpoint_test.ko
Building pcitest
# cd <kernel-dir> # make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- -C tools/pci/ # cp pcitest <rootfs>/usr/sbin/
Note: The pcitest utility tool is run on PCIe RC and hence needs to be in root file system of PCIe RC device.
pcitest options
$ pcitest -h usage: pcitest [options] Options: -D <dev> PCI endpoint test device {default: /dev/pci-endpoint-test.0} -b <bar num> BAR test (bar number between 0..5) -m <msi num> MSI test (msi number between 1..32) -x <msix num> MSI-X test (msix number between 1..2048) -i <irq type> Set IRQ type (0 - Legacy, 1 - MSI, 2 - MSI-X) -I Get current IRQ type configured -l Legacy IRQ test -r Read buffer test -w Write buffer test -c Copy buffer test -s <size> Size of buffer {default: 100KB} -h Print this help message
BAR test
$ pcitest -b 0 BAR0: OKAY $ pcitest -b 2 BAR2: OKAY $ pcitest -b 4 BAR4: OKAY
💡 Note: BAR's 0/2/4 are 64-bit hence BAR's 1/3/5 are not usable
Legacy interrupt test
$ pcitest -i 0 SET IRQ TYPE TO LEGACY: OKAY $ pcitest -l LEGACY IRQ: OKAY
MSI interrupt test
$ pcitest -i 1 SET IRQ TYPE TO MSI: OKAY $ pcitest -m x (Note: value of x can range from 1 to value echoed into msi_interrupts) Example to test MSI1: $ pcitest -i 1 SET IRQ TYPE TO MSI: OKAY $ pcitest -m 1 MSI1: OKAY
READ test
$ pcitest -r 1024 READ ( 102400 bytes): OKAY
WRITE test
$ pcitest -w 1024 WRITE ( 102400 bytes): OKAY
COPY test
$ pcitest -c 1024 COPY ( 102400 bytes): OKAY
💡 Note: Before running READ/WRITE/COPY test cases make sure the interrupt mode is either set to Legacy/MSI
💡 Known Issues
- On RZ/G2 platform when using as RC the read/write/copy tests fail due to TLB issues. (Refer https://www.spinics.net/lists/linux-pci/msg92385.html for more details). To get around with this issue use the attached patch gfp-dma.patch for upstream kernel.
- For RZ/G2E RevA/B boards hardware modification is required along with different cable as described above.